DETAILED ACTION
In response to communications filed 22 August 2025, this is the first Office action on the merits. Claims 21-40 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 21-40 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-19 of U.S. Patent No. 12,493,650 B2. Although the claims at issue are not identical, they are not patentably distinct from each as shown in the following table.
Instant Application
US 12,493,650 B2
21. A method of analyzing video files in a content search system, comprising:
applying classifications to video files using an artificial intelligence (Al) model, the Al model trained according to training data comprising images separated into object of interest classes or foreign object classes corresponding to occlusion of an object of interest, the classifications comprising one or more objects or events;
extracting, using natural language processing, one or more entities from a natural language search query received, in a natural language format, via a user interface, the one or more entities comprising one or more objects or events indicated by the natural language search query;
searching the video files using the classifications applied by the Al model and the one or more entities extracted from the natural language search query; and
presenting one or more of the video files identified as results of the natural language search query via the user interface.
1. A method for classifying and searching video files in a building security system, the method comprising:
applying classifications to video files using an artificial intelligence (AI) model, the AI model trained according to training data comprising images separated into object of interest classes and foreign object classes corresponding to occlusion of an object of interest, the classifications comprising one or more objects or events recognized in the video files by the Al model;
extracting one or more entities from a search query received, in a natural language format, via a user interface, the one or more entities comprising one or more objects or events indicated by the search query;
searching the video files using the classifications applied by the AI model and the one or more entities extracted from the search query;
determining an intent of the search query;
searching the video files further using the intent to identify one or more video files based on the one or more video files having a relevancy score above a threshold, the relevancy score of each of the one or more video files based on how well the one or more video files match the intent and the one or more entities; and
presenting the one or more of the video files identified as results of the search query as playable videos via the user interface.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 21-29 and 32-40 are rejected under 35 U.S.C. 103 as being unpatentable over Bender et al. (US 11,341,186 B2) in view of Kulandai Samy et al. (US 2023/0076241 A1) and Janakiraman et al. (US 2023/0205817 A1).
Regarding claim 21, Bender teaches a method of analyzing video files in a content search system, comprising:
applying classifications to video files using an artificial intelligence (Al) model, the classifications comprising one or more objects or events (see Bender12:28-41, “extract . . . objects, entities, actors, locations” applies classifications to the “videos,” where 12:55-13:15 teaches that the classifications are applied using “various machine learning and deep learning” Al models);
extracting one or more entities from a search query received, via a user interface, the one or more entities comprising one or more objects or events indicated by the search query (see Bender 15:41-60, “identifies relevant entities from the provided search input”);
searching the video files using the classifications applied by the Al model and the one or more entities extracted from the search query (see Bender 15:41-60, “utilizes the identified relevant entities to search an indexed repository for video results”); and
presenting one or more of the video files identified as results of the search query via the user interface (see Bender 15:65-16:18, “provides the ranked results to the user responsive to the query . . . specific and relevant video fragments (shots) in the response . . . plays only the relevant fragments, while playing the video to play from specific start and end times”).
Bender does not explicitly teach the Al model trained according to training data comprising images separated into object of interest classes or foreign object classes corresponding to occlusion of an object of interest.
However, Kulandai Samy teaches the Al model trained according to training data comprising images separated into object of interest classes or foreign object classes corresponding to occlusion of an object of interest (see Kulandai Samy [0033] and [0057], “threshold number of images in the training dataset include an occluded view of a person”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to train the AI model, as taught by Kulandai Samy, in combination with the techniques taught by Bender, because “Inclusion of enough occlusion data for training will improve model accuracy in real time scenes such as in retail shops, supermarket, coffee shop, restaurant and office, where ROI boundaries are occluded most of the time” (see Kulandai Samy [0033]).
Bender as modified does not explicitly teach extracting the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format.
However, Janakiraman teaches extracting the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format (see Janakiraman [0057], “search query 34 may be entered using natural human language”; “convert the natural human language into a computer search language”; and “one or more entities . . . within the search query 34 to be extracted”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to process a natural language search query, as taught by Janakiraman, in combination with the techniques taught by Bender as modified, because “In some cases, the video cognitive services 29 may utilize the machine learning application 59 to be able to better understand the user’s intent for the search based on the user’s context” (see Janakiraman [0057]).
Regarding claim 32, Bender teaches a system of video file analysis in a content search system, comprising:
one or more processing circuits coupled with memory to (see Bender 25:54-26:12):
apply classifications to video files using an artificial intelligence (AI) model the classifications comprising one or more objects or events recognized in the video files by the AI model (see Bender12:28-41, “extract . . . objects, entities, actors, locations” applies classifications to the “videos,” where 12:55-13:15 teaches that the classifications are applied using “various machine learning and deep learning” Al models);
extract one or more entities from a search query received, via a user interface, the entities comprising one or more objects or events indicated by the search query (see Bender 15:41-60, “identifies relevant entities from the provided search input”);
search the video files using the classifications applied by the AI model and the one or more entities extracted from the search query (see Bender 15:41-60, “utilizes the identified relevant entities to search an indexed repository for video results”); and
present one or more of the video files identified as results of the search query via the user interface (see Bender 15:65-16:18, “provides the ranked results to the user responsive to the query . . . specific and relevant video fragments (shots) in the response . . . plays only the relevant fragments, while playing the video to play from specific start and end times”).
Bender does not explicitly teach the AI model trained according to training data comprising images separated into object of interest classes or foreign object classes corresponding to occlusion of an object of interest.
However, Kulandai Samy teaches the AI model trained according to training data comprising images separated into object of interest classes or foreign object classes corresponding to occlusion of an object of interest (see Kulandai Samy [0033] and [0057], “threshold number of images in the training dataset include an occluded view of a person”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to train the AI model, as taught by Kulandai Samy, in combination with the techniques taught by Bender, because “Inclusion of enough occlusion data for training will improve model accuracy in real time scenes such as in retail shops, supermarket, coffee shop, restaurant and office, where ROI boundaries are occluded most of the time” (see Kulandai Samy [0033]).
Bender as modified does not explicitly teach to extract the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format.
However, Janakiraman teaches to extract the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format (see Janakiraman [0057], “search query 34 may be entered using natural human language”; “convert the natural human language into a computer search language”; and “one or more entities . . . within the search query 34 to be extracted”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to process a natural language search query, as taught by Janakiraman, in combination with the techniques taught by Bender as modified, because “In some cases, the video cognitive services 29 may utilize the machine learning application 59 to be able to better understand the user’s intent for the search based on the user’s context” (see Janakiraman [0057]).
Regarding claim 40, Bender teaches a non-transitory system of video file analysis in a content search system, comprising:
one or more processing circuits coupled with memory to store instructions that, when executed by the one or more processors (see Bender 25:54-26:12),
cause the one or more processors to:
apply classifications to video files using an artificial intelligence (AI) model, the classifications comprising one or more objects or events recognized in the video files by the AI model (see Bender12:28-41, “extract . . . objects, entities, actors, locations” applies classifications to the “videos,” where 12:55-13:15 teaches that the classifications are applied using “various machine learning and deep learning” Al models);
extract one or more entities from a search query received, via a user interface, the entities comprising one or more objects or events indicated by the search query (see Bender 15:41-60, “identifies relevant entities from the provided search input”);
search the video files using the classifications applied by the AI model and the one or more entities extracted from the search query (see Bender 15:41-60, “utilizes the identified relevant entities to search an indexed repository for video results”); and
present one or more of the video files identified as results of the search query via the user interface (see Bender 15:65-16:18, “provides the ranked results to the user responsive to the query . . . specific and relevant video fragments (shots) in the response . . . plays only the relevant fragments, while playing the video to play from specific start and end times”).
Bender as modified does not explicitly teach extracting the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format.
However, Janakiraman teaches extracting the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format (see Janakiraman [0057], “search query 34 may be entered using natural human language”; “convert the natural human language into a computer search language”; and “one or more entities . . . within the search query 34 to be extracted”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to process a natural language search query, as taught by Janakiraman, in combination with the techniques taught by Bender as modified, because “In some cases, the video cognitive services 29 may utilize the machine learning application 59 to be able to better understand the user’s intent for the search based on the user’s context” (see Janakiraman [0057]).
Regarding claim 22, Bender as modified teaches comprising:
tagging at least one of the video files with a semantic tag (see Bender 10:10-36, “automatic tagging of entities”);
wherein the AI model comprises at least one of a foundation AI model, a generative AI model, or a large language model (see Bender 12:55-13:15).
Regarding claim 23, Bender as modified teaches comprising:
searching, by at least one cloud server, the video files using the classifications applied by the AI model and the one or more entities extracted from the natural language search query (see Bender 20:22-37, “cloud computing node” and 15:41-60, “utilizes the identified relevant entities to search an indexed repository for video results”);
the natural language search query including freeform text, image, voice, or verbal inputs provided by a user via the user interface (see Bender 15:41-60, “query . . . text, voice,” and Janakiraman [0057], “natural human language”).
Regarding claims 24 and 35, Bender as modified teaches comprising: extracting two or more entities from the natural language search query (see Bender15:41-60, “relevant entities”).
Bender as modified does not explicitly teach
determining an intended relationship between the two or more entities based on information linking the two or more entities in the natural language search query; and
one or more of the video files classified as having the two or more entities linked by the intended relationship.
However, Janakiraman teaches
determining an intended relationship between the two or more entities based on information linking the two or more entities in the natural language search query (see Janakiraman [0057], “find a man wearing a red shirt, carrying a briefcase” and “entities . . . extracted, such as a primary object (e.g., a water bottle, an airport, etc.) and/or event of interest to the user”);
one or more of the video files classified as having the two or more entities linked by the intended relationship (see Janakiraman [0072], “describe the object attributes within the video frames” and “object association with other detected objects”).
Bender teaches in 15:41-60 to process queries with two or more entities and an
intended relationship by “finding a video clip with a particular family member riding a horse at
a particular geographic location.” It would have been obvious to one of ordinary skill in the art
before the effective filing date of the claimed invention to determine an intended relationship in a query and classify one or more of the video files, as further taught by Janakiraman, with the techniques taught by Bender as modified, because “In some cases, the video cognitive services 29 may utilize the machine learning application 59 to be able to better understand the user’s intent for the search based on the user’s context” (see Janakiraman [0057]).
Bender as modified teaches using the intended relationship in combination with the two or more entities to identify one or more of the video files classified as having the two or more entities linked by the intended relationship (see Bender 15:41-60 and Janakiraman [0057] and [0072], where searching video files with two or more entities, as taught by Bender, uses the intended relationship in combination with the two or more entities taught by Janakiraman)
Regarding claims 25 and 36, Bender as modified does not explicitly teach comprising:
adding supplemental annotations to the video files using the Al model, the supplemental annotations marking an area or location within a video frame of the video files at which a particular object or event is depicted in the video frame;
presenting the supplemental annotations overlaid with the video frame via the user interface.
However, Kulandai Samy teaches
adding supplemental annotations to the video files using the Al model, the supplemental annotations marking an area or location within a video frame of the video files at which a particular object or event is depicted in the video frame (see Kulandai Samy [0043], “generate an ROI boundary around the object”);
presenting the supplemental annotations overlaid with the video frame via the user interface (see Kulandai Samy [0020] and Fig. 1, element 104, “ROI Boundary”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to add and present supplemental annotations, as further taught by Kulandai Samy, in combination with the techniques taught by Bender as modified, because “Some surveillance and retail analytics use-cases use models for the detection of a region of interest (ROI) that bounds one or more objects, such as persons, vehicles, or any other object configured to be detected, in live camera videos” (see Kulandai Samy [0003]).
Regarding claims 26 and 37, Bender as modified teaches comprising:
processing a timeseries of video frames of a video file recorded over a time period using the Al model to identify an event that begins at a start time during the time period and ends at an end time during the time period (see Bender 11:59-12:8, “links start and stop times (encapsulating a fragment with the entities) within the one or more videos to the entities,” where a “fragment” is a time series of video frames and 13:38-43 teaches the identified entity may be an “event”); and
applying a classification to the video file that identifies the event, the start time of the event, and the end time of the event (see Bender 11:59-12:8 and 15:6-17).
Regarding claims 27 and 38, Bender as modified teaches wherein the video files are recorded by one or more cameras and the classifications are applied to the video files during a first time period to generate a database of pre-classified video files (see Bender 11:58-12:8, the first time period includes when the “videos” are recorded up to when the classifications are applied after the “videos are uploaded,” where 15:6-17 teaches a “search index” database of pre-classified video files);
wherein the natural language search query is received via the user interface during a second time period after the first time period (see Bender 15:41-60, “searching the previously uploaded content”); and
searching the database of the pre-classified video files using the one or more entities extracted from the natural language search query after the video files are classified (see Bender 15:41-60, “search an indexed repository for video results”).
Regarding claim 28, Bender as modified does not explicitly teach wherein the natural language search query is received via the user interface and the one or more entities are extracted from the natural language search query during a first time period to generate a stored rule based on the natural language search query;
wherein the video files comprise live video streams received from one or more cameras and the classifications are applied to the live video streams during a second time period after the first time period; and
searching the live video streams using the stored rule to determine whether the one or more entities extracted from the natural language search query are depicted in the live video streams.
However, Janakiraman teaches wherein the natural language search query is received via the user interface and the one or more entities are extracted from the natural language search query during a first time period to generate a stored rule based on the natural language search query (see Janakiraman [0057], a “search query” is received and used to generate a stored rule that is applied to the “video data streams” as taught in [0059]);
wherein the video files comprise live video streams received from one or more cameras and the classifications are applied to the live video streams during a second time period after the first time period (see Janakiraman [0057] and [0040], “processing live or substantially live video feeds,” wherein the classifications are applied to the live video streams during a second time period after the first time period when new “live” data is received after the search query); and
searching the live video streams using the stored rule to determine whether the one or more entities extracted from the natural language search query are depicted in the live video streams (see Janakiraman [0059], “identify one or more matching objects and/or events within the plurality of video data streams and/or the metadata 18 that match the search query”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to process live or substantially live video feeds, as taught by Janakiraman, in combination with the techniques taught by Bender, because it “would be desirable is a more efficient way of capturing, organizing and/or processing video content to help identify one or more events in the captured video” (see Janakiraman [0004]).
Regarding claims 29 and 39, Bender as modified teaches comprising:
cutting the video files to create one or more snippets of the video files based on an output of the Al model indicating one or more times at which the one or more entities extracted from the natural language search query appear in the video files (see Bender12:9-27, “converts one or more videos into temporal shots . . . fragments, segments”); and
presenting the one or more snippets of the video files as the results of the natural language search query via the user interface (see Bender 15:65-16:18, “plays only the relevant fragments”).
Regarding claim 33, Bender as modified teaches wherein the AI model comprises at least one of a foundation AI model, a generative AI model, or a large language model (see Bender 12:55-13:15).
Regarding claim 34, Bender as modified teaches comprising: the natural language search query including freeform text or verbal inputs (see Bender 15:41-60, “query . . . text, voice,” and Janakiraman [0057], “natural human language”).
Claim 31 is rejected under 35 U.S.C. 103 as being unpatentable over Bender et al. (US 11,341,186 B2) in view of Kulandai Samy et al. (US 2023/0076241 A1) and Janakiraman et al. (US 2023/0205817 A1) as applied to claim 21 above, and further in view of Zadeh et al. (US 11,468,677 B2).
Regarding claim 31, Bender as modified teaches comprising: determining a relevance score or ranking for each of the video files using the classifications applied by the AI model and the one or more entities extracted from the natural language search query (see Bender 15:61-64, “ranks the search results”).
Bender as modified does not explicitly teach presenting the relevance score or ranking for each of the video files presented as results of the natural language search query via the user interface.
However, Zadeh teaches presenting the relevance score or ranking for each of the video files presented as results of the natural language search query via the user interface (see Zadeh 11:35-43, “presents the confidence score”).
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to present the relevance score or ranking, as taught by Zadeh, in
combination with the techniques taught by Bender as modified , because the “user interface element 310a provides a interface module 150 [that] dynamically updates the user interface 200 by updating the user interface elements 209 according to the threshold confidence score input by the user” (see Zadeh 11:44-62).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kristopher Andersen whose telephone number is (571)270-5743. The examiner can normally be reached 8:30 AM-5:00 PM ET, Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann Lo can be reached at (571) 272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kristopher Andersen/Primary Examiner, Art Unit 2159
Prosecution Insights