DETAILED ACTION
In response to communications filed 24 June 2026, claims 21, 26, 32, 37, and 40 are amended and claim 41 is added per applicant’s request. Claims 21-41 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claim 30 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: Bender et al. (US 11,341,186 B2); Kulandai Samy et al. (US 2023/0076241 A1); Janakiraman et al. (US 2023/0205817 A1); and Zadeh et al. (US 11,468,677 B2) are the closest prior art on record as shown in the instant rejection.
Janakiraman is the closest prior art on record to the specific features of claim 30, where in paragraph [0050] Janakiraman teaches to search videos for entities extracted from a natural language query. However, Janakiram does not explicitly teach to determine an “intent” of the natural language query and to “identify the one more video files based on the one or more video files having a relevancy score above a threshold, the relevancy score of each of the one or more video files based on how well the one or more video files match the intent and the one or more entities,” as recited. These features, when considered as a whole in combination with claim 21, therefore amount to allowable subject matter.
Response to Arguments
Applicant’s arguments, see section “Nonstatutory Double Patenting,” filed 24 June 2026, with respect to claims 21-40 have been fully considered and are persuasive. The rejection of claims 21-40 has been withdrawn.
Applicant’s arguments, see section “Rejections Under 35 U.S.C. § 103,” filed 24 June 2026, with respect to claims 21, 32, and 40 have been fully considered but are not persuasive.
On page 10, applicant argues that Bender does not teach claims 21, 32, and 40 as amended, because Bender merely teaches analyzes audio and does not teach analyzing video content.
Bender states “the program code can utilize an existing cognitive agent to identify entities (individuals, events, actors, objects, locations, etc.) in the audio (e.g., speech).” Bender, 13 :38-43. However, Bender is just stating that the program code extracts text from an audio track that identifies that something was spoken and does not teach or suggest analyzing video content of frames.
However, applicant’s arguments are not persuasive. First, the audio analysis highlighted by applicant is an analysis audio from the video fragments (“a timeseries of video frames”) as taught by Bender in 13:16-29, because they program code “extracts entities from the audio that accompanies the images (e.g. , frames) comprising the shots or fragments” (emphasis added). Accordingly, the AI model audio analysis acknowledged by applicant teaches to analyze video content by analyzing its accompanying audio. In addition, Bender teaches in 12:9-41 an analysis of the video content itself to “segment the one or more videos into fragments that can be linked to entities” that are extracted “from both the images and the audio associated with the fragments.” Bender therefore teaches the claim language at issue by classifying objects or events from “fragments,” i.e., a timeseries of video frames of a video file,” by segmenting the videos, extracting entities, and linking “start and stop times . . . within the one ore more videos to the entities”; see Bender 11:59-12:8.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 21-29 and 32-41 are rejected under 35 U.S.C. 103 as being unpatentable over Bender et al. (US 11,341,186 B2) in view of Kulandai Samy et al. (US 2023/0076241 A1) and Janakiraman et al. (US 2023/0205817 A1).
Regarding claim 21, Bender teaches a method of analyzing video files in a content search system, comprising:
applying classifications to video files using an artificial intelligence (Al) model (see Bender12:28-41, “extract . . . objects, entities, actors, locations” applies classifications to the “videos,” where 12:55-13:15 teaches that the classifications are applied using “various machine learning and deep learning” Al models),
the classifications comprising one or more objects or events identified from a timeseries of video frames of a video file of the video files that begins at a start time and ends at an end time, wherein a classification of the classifications identifies an event of the one or more objects or events, the start time of the event, and the end time of the event (see Bender 11:59-12:8, “links start and stop times (encapsulating a fragment with the entities) within the one or more videos to the entities,” where a “fragment” is a time series of video frames and 13:38-43 teaches the identified entity may be an “event”);
extracting one or more entities from a search query received, via a user interface, the one or more entities comprising one or more objects or events indicated by the search query (see Bender 15:41-60, “identifies relevant entities from the provided search input”);
searching the video files using the classifications applied by the Al model and the one or more entities extracted from the search query (see Bender 15:41-60, “utilizes the identified relevant entities to search an indexed repository for video results”); and
presenting one or more of the video files identified as results of the search query via the user interface (see Bender 15:65-16:18, “provides the ranked results to the user responsive to the query . . . specific and relevant video fragments (shots) in the response . . . plays only the relevant fragments, while playing the video to play from specific start and end times”).
Bender does not explicitly teach the Al model trained according to training data comprising images separated into object of interest classes or foreign object classes corresponding to occlusion of an object of interest.
However, Kulandai Samy teaches the Al model trained according to training data comprising images separated into object of interest classes or foreign object classes corresponding to occlusion of an object of interest (see Kulandai Samy [0033] and [0057], “threshold number of images in the training dataset include an occluded view of a person”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to train the AI model, as taught by Kulandai Samy, in combination with the techniques taught by Bender, because “Inclusion of enough occlusion data for training will improve model accuracy in real time scenes such as in retail shops, supermarket, coffee shop, restaurant and office, where ROI boundaries are occluded most of the time” (see Kulandai Samy [0033]).
Bender as modified does not explicitly teach extracting the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format.
However, Janakiraman teaches extracting the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format (Janakiraman and “one or more entities . . . within the search query 34 to be extracted”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to process a natural language search query, as taught by Janakiraman, in combination with the techniques taught by Bender as modified, because “In some cases, the video cognitive services 29 may utilize the machine learning application 59 to be able to better understand the user’s intent for the search based on the user’s context” (see Janakiraman [0057]).
Regarding claim 32, Bender teaches a system of video file analysis in a content search system, comprising:
one or more processing circuits coupled with memory to (see Bender 25:54-26:12):
apply classifications to video files using an artificial intelligence (AI) model (see Bender12:28-41, “extract . . . objects, entities, actors, locations” applies classifications to the “videos,” where 12:55-13:15 teaches that the classifications are applied using “various machine learning and deep learning” Al models)
the classifications comprising one or more objects or events recognized from a timeseries of video frames of a video file of the video files that begins at a start time and ends at an end time, wherein a classification of the classifications identifies an event of the one or more objects or events, the start time of the event, and the end time of the event (see Bender 11:59-12:8, “links start and stop times (encapsulating a fragment with the entities) within the one or more videos to the entities,” where a “fragment” is a time series of video frames and 13:38-43 teaches the identified entity may be an “event”);
extract one or more entities from a search query received, via a user interface, the entities comprising one or more objects or events indicated by the search query (see Bender 15:41-60, “identifies relevant entities from the provided search input”);
search the video files using the classifications applied by the AI model and the one or more entities extracted from the search query (see Bender 15:41-60, “utilizes the identified relevant entities to search an indexed repository for video results”); and
present one or more of the video files identified as results of the search query via the user interface (see Bender 15:65-16:18, “provides the ranked results to the user responsive to the query . . . specific and relevant video fragments (shots) in the response . . . plays only the relevant fragments, while playing the video to play from specific start and end times”).
Bender does not explicitly teach the AI model trained according to training data comprising images separated into object of interest classes or foreign object classes corresponding to occlusion of an object of interest.
However, Kulandai Samy teaches the AI model trained according to training data comprising images separated into object of interest classes or foreign object classes corresponding to occlusion of an object of interest (see Kulandai Samy [0033] and [0057], “threshold number of images in the training dataset include an occluded view of a person”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to train the AI model, as taught by Kulandai Samy, in combination with the techniques taught by Bender, because “Inclusion of enough occlusion data for training will improve model accuracy in real time scenes such as in retail shops, supermarket, coffee shop, restaurant and office, where ROI boundaries are occluded most of the time” (see Kulandai Samy [0033]).
Bender as modified does not explicitly teach to extract the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format.
However, Janakiraman teaches to extract the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format (see Janakiraman [0057], “search query 34 may be entered using natural human language”; “convert the natural human language into a computer search language”; and “one or more entities . . . within the search query 34 to be extracted”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to process a natural language search query, as taught by Janakiraman, in combination with the techniques taught by Bender as modified, because “In some cases, the video cognitive services 29 may utilize the machine learning application 59 to be able to better understand the user’s intent for the search based on the user’s context” (see Janakiraman [0057]).
Regarding claim 40, Bender teaches a non-transitory system of video file analysis in a content search system, comprising:
one or more processing circuits coupled with memory to store instructions that, when executed by the one or more processors (see Bender 25:54-26:12),
cause the one or more processors to:
apply classifications to video files using an artificial intelligence (AI) model (see Bender12:28-41, “extract . . . objects, entities, actors, locations” applies classifications to the “videos,” where 12:55-13:15 teaches that the classifications are applied using “various machine learning and deep learning” Al models),
the classifications comprising one or more objects or events recognized in the video files by the AI model the one or more objects or events recognized from a timeseries of video frames of a video file of the video files that begins at a start time and ends at an end time, wherein a classification of the classifications identifies an event of the one or more objects or events, the start time of the event, and the end time of the event (see Bender 11:59-12:8, “links start and stop times (encapsulating a fragment with the entities) within the one or more videos to the entities,” where a “fragment” is a time series of video frames and 13:38-43 teaches the identified entity may be an “event”);
extract one or more entities from a search query received, via a user interface, the entities comprising one or more objects or events indicated by the search query (see Bender 15:41-60, “identifies relevant entities from the provided search input”);
search the video files using the classifications applied by the AI model and the one or more entities extracted from the search query (see Bender 15:41-60, “utilizes the identified relevant entities to search an indexed repository for video results”); and
present one or more of the video files identified as results of the search query via the user interface (see Bender 15:65-16:18, “provides the ranked results to the user responsive to the query . . . specific and relevant video fragments (shots) in the response . . . plays only the relevant fragments, while playing the video to play from specific start and end times”).
Bender as modified does not explicitly teach extracting the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format.
However, Janakiraman teaches extracting the one or more entities using natural language processing, wherein the search query is a natural language search query received in a natural language format (see Janakiraman [0057], “search query 34 may be entered using natural human language”; “convert the natural human language into a computer search language”; and “one or more entities . . . within the search query 34 to be extracted”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to process a natural language search query, as taught by Janakiraman, in combination with the techniques taught by Bender as modified, because “In some cases, the video cognitive services 29 may utilize the machine learning application 59 to be able to better understand the user’s intent for the search based on the user’s context” (see Janakiraman [0057]).
Regarding claim 22, Bender as modified teaches comprising:
tagging at least one of the video files with a semantic tag (see Bender 10:10-36, “automatic tagging of entities”);
wherein the AI model comprises at least one of a foundation AI model, a generative AI model, or a large language model (see Bender 12:55-13:15).
Regarding claim 23, Bender as modified teaches comprising:
searching, by at least one cloud server, the video files using the classifications applied by the AI model and the one or more entities extracted from the natural language search query (see Bender 20:22-37, “cloud computing node” and 15:41-60, “utilizes the identified relevant entities to search an indexed repository for video results”);
the natural language search query including freeform text, image, voice, or verbal inputs provided by a user via the user interface (see Bender 15:41-60, “query . . . text, voice,” and Janakiraman [0057], “natural human language”).
Regarding claims 24 and 35, Bender as modified teaches comprising: extracting two or more entities from the natural language search query (see Bender15:41-60, “relevant entities”).
Bender as modified does not explicitly teach
determining an intended relationship between the two or more entities based on information linking the two or more entities in the natural language search query; and
one or more of the video files classified as having the two or more entities linked by the intended relationship.
However, Janakiraman teaches
determining an intended relationship between the two or more entities based on information linking the two or more entities in the natural language search query (see Janakiraman [0057], “find a man wearing a red shirt, carrying a briefcase” and “entities . . . extracted, such as a primary object (e.g., a water bottle, an airport, etc.) and/or event of interest to the user”);
one or more of the video files classified as having the two or more entities linked by the intended relationship (see Janakiraman [0072], “describe the object attributes within the video frames” and “object association with other detected objects”).
Bender teaches in 15:41-60 to process queries with two or more entities and an
intended relationship by “finding a video clip with a particular family member riding a horse at
a particular geographic location.” It would have been obvious to one of ordinary skill in the art
before the effective filing date of the claimed invention to determine an intended relationship in a query and classify one or more of the video files, as further taught by Janakiraman, with the techniques taught by Bender as modified, because “In some cases, the video cognitive services 29 may utilize the machine learning application 59 to be able to better understand the user’s intent for the search based on the user’s context” (see Janakiraman [0057]).
Bender as modified teaches using the intended relationship in combination with the two or more entities to identify one or more of the video files classified as having the two or more entities linked by the intended relationship (see Bender 15:41-60 and Janakiraman [0057] and [0072], where searching video files with two or more entities, as taught by Bender, uses the intended relationship in combination with the two or more entities taught by Janakiraman)
Regarding claims 25 and 36, Bender as modified does not explicitly teach comprising:
adding supplemental annotations to the video files using the Al model, the supplemental annotations marking an area or location within a video frame of the video files at which a particular object or event is depicted in the video frame;
presenting the supplemental annotations overlaid with the video frame via the user interface.
However, Kulandai Samy teaches
adding supplemental annotations to the video files using the Al model, the supplemental annotations marking an area or location within a video frame of the video files at which a particular object or event is depicted in the video frame (see Kulandai Samy [0043], “generate an ROI boundary around the object”);
presenting the supplemental annotations overlaid with the video frame via the user interface (see Kulandai Samy [0020] and Fig. 1, element 104, “ROI Boundary”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to add and present supplemental annotations, as further taught by Kulandai Samy, in combination with the techniques taught by Bender as modified, because “Some surveillance and retail analytics use-cases use models for the detection of a region of interest (ROI) that bounds one or more objects, such as persons, vehicles, or any other object configured to be detected, in live camera videos” (see Kulandai Samy [0003]).
Regarding claims 26 and 37, Bender as modified teaches comprising:
processing the timeseries of video frames of the video file recorded over a time period using the Al model to identify the event that begins at the start time during the time period and ends at the end time during the time period (see Bender 11:59-12:8 and 13:38-43); and
applying the classification to the video file that identifies the event, the start time of the event, and the end time of the event (see Bender 11:59-12:8 and 15:6-17).
Regarding claims 27 and 38, Bender as modified teaches wherein the video files are recorded by one or more cameras and the classifications are applied to the video files during a first time period to generate a database of pre-classified video files (see Bender 11:58-12:8, the first time period includes when the “videos” are recorded up to when the classifications are applied after the “videos are uploaded,” where 15:6-17 teaches a “search index” database of pre-classified video files);
wherein the natural language search query is received via the user interface during a second time period after the first time period (see Bender 15:41-60, “searching the previously uploaded content”); and
searching the database of the pre-classified video files using the one or more entities extracted from the natural language search query after the video files are classified (see Bender 15:41-60, “search an indexed repository for video results”).
Regarding claim 28, Bender as modified does not explicitly teach wherein the natural language search query is received via the user interface and the one or more entities are extracted from the natural language search query during a first time period to generate a stored rule based on the natural language search query;
wherein the video files comprise live video streams received from one or more cameras and the classifications are applied to the live video streams during a second time period after the first time period; and
searching the live video streams using the stored rule to determine whether the one or more entities extracted from the natural language search query are depicted in the live video streams.
However, Janakiraman teaches wherein the natural language search query is received via the user interface and the one or more entities are extracted from the natural language search query during a first time period to generate a stored rule based on the natural language search query (see Janakiraman [0057], a “search query” is received and used to generate a stored rule that is applied to the “video data streams” as taught in [0059]);
wherein the video files comprise live video streams received from one or more cameras and the classifications are applied to the live video streams during a second time period after the first time period (see Janakiraman [0057] and [0040], “processing live or substantially live video feeds,” wherein the classifications are applied to the live video streams during a second time period after the first time period when new “live” data is received after the search query); and
searching the live video streams using the stored rule to determine whether the one or more entities extracted from the natural language search query are depicted in the live video streams (see Janakiraman [0059], “identify one or more matching objects and/or events within the plurality of video data streams and/or the metadata 18 that match the search query”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to process live or substantially live video feeds, as taught by Janakiraman, in combination with the techniques taught by Bender, because it “would be desirable is a more efficient way of capturing, organizing and/or processing video content to help identify one or more events in the captured video” (see Janakiraman [0004]).
Regarding claims 29 and 39, Bender as modified teaches comprising:
cutting the video files to create one or more snippets of the video files based on an output of the Al model indicating one or more times at which the one or more entities extracted from the natural language search query appear in the video files (see Bender12:9-27, “converts one or more videos into temporal shots . . . fragments, segments”); and
presenting the one or more snippets of the video files as the results of the natural language search query via the user interface (see Bender 15:65-16:18, “plays only the relevant fragments”).
Regarding claim 33, Bender as modified teaches wherein the AI model comprises at least one of a foundation AI model, a generative AI model, or a large language model (see Bender 12:55-13:15).
Regarding claim 34, Bender as modified teaches comprising: the natural language search query including freeform text or verbal inputs (see Bender 15:41-60, “query . . . text, voice,” and Janakiraman [0057], “natural human language”).
Regarding claim 41, Bender as modified teaches comprising: presenting a video file of the one or more video files identified as results of the natural language search query via the user interface as a playable video (see Bender 15:65-16:18).
Bender as modified does not explicitly teach the playable video having a supplemental annotation created from the video file overlaid with a video frame at a location of the one or more objects or events.
However, Kulandai Samy teaches the playable video having a supplemental annotation created from the video file overlaid with a video frame at a location of the one or more objects or events (see Kulandai Samy [0020] and [0043], “generate an ROI boundary around the object,” and Fig. 1, element 104, “ROI Boundary”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention include a supplemental annotation, as taught by Kulandai Samy, with the techniques taught by Bender as modified, because “Some surveillance and retail analytics use-cases use models for the detection of a region of interest (ROI) that bounds one or more objects, such as persons, vehicles, or any other object configured to be detected, in live camera videos” (see Kulandai Samy [0003]).
Claim 31 is rejected under 35 U.S.C. 103 as being unpatentable over Bender et al. (US 11,341,186 B2) in view of Kulandai Samy et al. (US 2023/0076241 A1) and Janakiraman et al. (US 2023/0205817 A1) as applied to claim 21 above, and further in view of Zadeh et al. (US 11,468,677 B2).
Regarding claim 31, Bender as modified teaches comprising: determining a relevance score or ranking for each of the video files using the classifications applied by the AI model and the one or more entities extracted from the natural language search query (see Bender 15:61-64, “ranks the search results”).
Bender as modified does not explicitly teach presenting the relevance score or ranking for each of the video files presented as results of the natural language search query via the user interface.
However, Zadeh teaches presenting the relevance score or ranking for each of the video files presented as results of the natural language search query via the user interface (see Zadeh 11:35-43, “presents the confidence score”).
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to present the relevance score or ranking, as taught by Zadeh, in
combination with the techniques taught by Bender as modified , because the “user interface element 310a provides a interface module 150 [that] dynamically updates the user interface 200 by updating the user interface elements 209 according to the threshold confidence score input by the user” (see Zadeh 11:44-62).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kristopher Andersen whose telephone number is (571)270-5743. The examiner can normally be reached 8:30 AM-5:00 PM ET, Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann Lo can be reached at (571) 272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kristopher Andersen/Primary Examiner, Art Unit 2159
Prosecution Insights