Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 7/7/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1023 have been considered but are moot because the new ground of rejection, which were necessitated by amendment and no longer rely on Kim: as amended, the claims are rejected over Nicastri in view of Lev-Tov and Qasemi, with previously cited Nicastri teaching the automatic, request triggered selection and chronological arrangement. See discussions below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-7, 10, 12-15 and 17-21 are rejected under 35 U.S.C. 103 as being unpatentable over Nicastri et al. (US 2021/0312188) in view of Lev-Tov et al. (US 2018/0232451) further in view of Qasemi et al. (US 2024/0029422).
Regarding claims 1 and 14, Nicastri teaches an apparatus comprising:
an image processing system configured to receive images from an image capture device (Fig. 1, cameras 14 supplies images received by system 10); and
an event summarization system coupled to the image processing system and configured (Fig. 1, system 10 and paragraph 34 summarizes video using processor 30 and exporter 48) to:
receive a request to create an event summarization, the request including details associated with the event summarization (Figs. 4-20, search query received through UI to perform a search and output a compiled composite video; paragraph 40, block 64 teaches receiving characteristics 38 of the designated individual; paragraphs 44-46 teaches characteristics, cameras and a search time window input via tabs 94);
in response to receiving the request (Fig. 3 and paragraph 40: on receipt of the characteristics at block 64, processor 30 performs the batch search (block 66), extraction (block 68), association (block 70), and export (blocks 72-74) of method 62. Nicastri’s claim 19 also recites automatically extracting and associating clips and exporting them for time-ordered playback as one sequence following the search):
identify a plurality of images relevant to the event summarization based on the structured data and the details associated with the event summarization (paragraphs 43, 49, 53-54 teaches identifying moments within the video captured by cameras based on search. The search algorithm is utilized to generate comparisons between the search and video data to determine which ones match the search. A plurality of matching video frames 34 is identified across multiple video tracks 82a-d, Paragraph 43, Figs. 5-18. Video frames 34 are “individual snapshots,” i.e., images (paragraph 38));
automatically selecting, by an event summarization system, a plurality of the identified images relevant to the event summarization; arrange the selected images (Fig. 21 and paragraphs 70-72 teaches selecting various video clips 40a, 40b, etc.. Nicastri teaches that processor 30 extracts the video clips containing the matching frames (paragraph 40, block 68), and that “for each video frame including at least one video frame that includes an individual with characteristics matching the one or more received characteristics ... the one or more video clips are automatically extracted from their respective video stream without user input" (claim 4, see also claim 16, “without further user input,” and claim 19, “automatically extracting one or more video clips”). Nicastri further teaches that the identifying, extracting, compiling, and organizing of the relevant clips may be performed manually, automatically, or a combination of the two (paragraph 63). Process the 30 itself determines clip boundaries by collapsing sidings that are close in time into a single clip based on a threshold(paragraphs 49, 54 and 58), and “automatically determines whether to include a non detection ... based on the period's duration" (paragraph 68). The result is a plurality of selected clips 40a-e (paragraph 70 and Figs. 20-2)); and
automatically arranging, by the event summarization system, the selected images in a chronological or topical video narrative that satisfies the specific temporal constraint and the specific object category (Nicastri teaches that “the extracted video clips 40a-e are combined into a single composite video 122 … such that when subsequently retrieved and viewed, video clips 40a-e are played in time order” (paragraph 70, 4 and 30), “compiled into a single composite video with the video clips being arranged by time”; paragraph 34, clock 50 “to help sort video clips 40 in order of time”; claims 7, 9 and 19). The compiling and organizing are expressly disclosed as performable automatically (paragraph 63 and claim 19). The resulting chronological narrative satisfies a temporal constraint, because processor 30 “limits its search to videos recorded only during the specific search time window” and limit export to " only those video clips falling within the specified export time window” (paragraph 46). It also satisfies an object category, because each extracted clip contains frames showing the designated subject (paragraph 40, block 68), and the designated subject may be a non-human object such as a bag or weapon (paragraphs 77 and 80). Nicastri receives the time window and the subject through UI fields rather than from LLM-extracted structured data; that difference is addressed by Qasemi below); and
create a video summary representing the event summarization, the video summary including the arrangement of the selected images (Fig. 21 and paragraphs 70-72 teaches selecting various video clips 40a, 40b, etc. are compiled into a summarized composite video 122/ composite file 134 for the user based on the search. Further, while Nicastri generates a video summary, does not explicitly teach arranging in in chronological or topical video narrative that satisfies a specific temporal constraint and specific object category, but is addressed below).
Although Nicastri discloses an embodiment in which personnel positions start and stop bars by hand to define clip boundaries (paragraphs 64067), a reference is relied upon for all that it would reasonably suggest, including alternative embodiments (MPEP 2123). Nicastri expressly discloses the alternative in which the extraction, compiling, and time ordering performed automatically without user input (paragraphs 63, 68 6 and claims 4, 16 and 19).
However, although Nicastri teaches the claimed as discussed above, fails to teach, but Lev-Tov teaches the claimed:
pre-process, by the image processing system, a plurality of images into image feature vectors (paragraph 52 teaches the claimed wherein images are processed to generate feature vectors. See also paragraph 51, image vectors added to the multimodal space and the DNN trained “prior to user interaction”);
store, by the image processing system, the image feature vectors in a common embedding space (paragraph 52 teaches storing the image vectors in an image database to create an image feature (IFV) space; see also paragraph 53, index database 518 “stores image vectors for N images … in the multimodal space,” and paragraph 40, image and text vectors embedded in the same multimodal space 330);
identify, by a large language model (LLM), structured data from the received request (paragraphs 4-6, 20-21 and 52 teaches an LLM using a query to generated structured data based on the request, however fails to teach the temporal constraint and specific object category as required by the amendment, but is addressed below);
identify at least one image relevant to the event summarization based on the structured data and the details associated with the event summarization (paragraphs 4-6, 20-23 and 38-40 teaches an LLM using a query to generated structured data based on the request);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to incorporate the teachings of Lev-Tov into they system of Nicastri because said incorporation allows for the benefit of providing an alternative or improved upon method of searching for video/image content by using text based searches which additionally allows for language based searches (paragraphs 3-8).
However, while Lev-Tov and Nicastri teaches the claimed as discussed above Qasemi et al. (US 2024/0029422 A1) teaches the newly added limitation of:
“in response to receiving the request: identifying, by a large language model (LLM), structured data from the received request, the structured data including a specific temporal constraint and a specific object category (Qasemi teaches receiving natural language input seeing an object or event (paragraph 51, step 512) and, in response, “relies on the natural language model infused with the human-augmented knowledge graph to derive and output fine-grained classifications corresponding to the desired object or event” (Paragraph 51 and step 514). Qasemi’s natural language model is a pretrained transformer language model, e.g., RoBERTa (paragraphs 41 and 50). For example, for the input “show me a scene with a garbage truck in the left lane from February 22nd of last year” (paragraph 51), the model resolves a specific object category (garbage truck, a fine grain vehicle type determined “via its node in the knowledge graph” (paragraph 51) and a specific temporal constraint (February 22nd of last year), against a knowledge graph “built on the time scale (e.g. time stamped)” (paragraph 48; see also paragraph 34). Similarly, for “show me suspicious activity last Tuesday,” the system resolves the request into structured node characteristics including “a certain time of night (e.g., between midnight and 4:00am)” (paragraph 53). The knowledge-graph node attributes produced from the request (object type, lane, time) are structured data under the broadest reasonable interpretation. Qasemi then retrieves and outputs “a video segment” matching those constraints (paragraphs 43, 51 and claim 1).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify the existing combination to Receive the request as a natural language input and to use Qasemi's language model to extract the temporal constraint and object category from that request, so that the extracted constraints populate Nicastri’s search time window (paragraph 46) and designated subject (paragraphs 77 and 80), and so that Nicastri’s automatic search, selection, and time ordered compilation (paragraphs 40, 63, 70 and claims 4 and 19) are carried out in response to that single request. Qasemi expressly identifies the benefit: it is “difficult for a user to navigate a knowledge graph manually or with desired rules to seek a desired scene. Therefore, natural language may be incorporated” (paragraph 43), and the approach serves to “reduce the need for humans to manually review through video feeds” (paragraph 53). Nicastri likewise seeks to spare surveillance personnel from having to watch an excessive amount of extraneous recorded material” (paragraph 70). Both references are directed to retrieving relevant segments from stored surveillance video, and the modification uses each element for established function with predictable results. KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 417 (2007); MPEP 2143 (I)(D).
Additionally, incorporating the natural language semantic retrieval techniques of Qasemi into the proposed combination also allows for the benefit of improving the accuracy of video retrievals and improve user searching and review of large image/video datasets by allowing semantic event retrieval using temporal/object constraints (Qasemi: paragraphs [0037], [0039], [0051], and [0052]).
Claim 1’s methodology is implemented by the apparatus of claim 14.
Regarding claims 2 and 17, Nicastri teaches the claimed wherein the request to create an event summarization includes at least one of an object to include in the event summarization, an activity to include in the event summarization, a time period associated with the event summarization, or a time limit for the event summarization (Examiner notes the alternative language stated above and while all are not required to meet the claimed limitations, Figs. 5-20, and more specifically paragraphs 31, 44-45 search input includes objects, time range and activities. Figs. 4-20, search query received through UI to perform a search and output a compiled composite video; paragraph 40, block 64 teaches receiving characteristics 38 of the designated individual; paragraphs 44-46 teaches characteristics, cameras and a search time window input via tabs 94).
Regarding claims 3 and 18, Nicastri teaches the claimed wherein arranging the selected images to be included in the event summarization is based on at least one of a chronological order, a particular topic, or a specific theme (Nicastri teaches that “the extracted video clips 40a-e are combined into a single composite video 122 … such that when subsequently retrieved and viewed, video clips 40a-e are played in time order” (paragraph 70, 4 and 30), “compiled into a single composite video with the video clips being arranged by time”; paragraph 34, clock 50 “to help sort video clips 40 in order of time”; claims 7, 9 and 19). The compiling and organizing are expressly disclosed as performable automatically (paragraph 63 and claim 19). The resulting chronological narrative satisfies a temporal constraint, because processor 30 “limits its search to videos recorded only during the specific search time window” and limit export to " only those video clips falling within the specified export time window” (paragraph 46). It also satisfies an object category, because each extracted clip contains frames showing the designated subject (paragraph 40, block 68), and the designated subject may be a non-human object such as a bag or weapon (paragraphs 77 and 80).).
Regarding claim 4, Nicastri teaches the claimed wherein a first portion of the selected images are associated with a first camera and a second portion of the selected images are associated with a second camera, and wherein at least one image from the first portion and at least one image from the second portion are included in the video summary (See Fig. 21 wherein multiple camera outputs are used and selected portions are taken from various camera angles and compiled).
Regarding claims 6 and 19, Nicastri teaches the claimed further comprising:
determining an event summarization time limit associated with the video summary (paragraph 30 and 46);
determining an image time limit for each image in the video summary (paragraph 30 and 46 searches for images/videos around the search time range); and
identifying specific selected images to include in the video summary based on the event summarization time limit and the image time limit (paragraph 30 and 46 based on search time range “from those video streams, a collection of video clips contain the relevant video frame may be identified”).
Regarding claims 7 and 15, Nicastri teaches the claimed wherein the request to create an event summarization is a natural language request (paragraph 44).
Regarding claim 10, Nicastri teaches the claimed further comprising adjusting an amount of time a particular image is displayed in the video summary based on an importance associated with the particular image (Fig. 21, the length of the image/video displayed in the summary is based on how long the search query results in the finding of such events/person/objects/activity, etc. and is therefore adjusted in accordance).
Regarding claim 12, Nicastri teaches the claimed wherein creating the video summary representing the event summarization is performed by an image processing system (Fig. 1 and 21, system performs the video summarization/compilation).
Regarding claims 13 and 20, Nicastri teaches the claimed further comprising communicating the video summary to at least one system associated with the request to create the event summarization (Fig. 1 exported to user’s device requesting the video search).
Regarding claim 21, Lev-Tov and Qasemi teaches the claimed determining, by the LLM, an intent of the received request; and generating, by the LLM, queries for searching relevant images (Lev-Tov: paragraphs 4-6, 20-21 and 52 teaches an LLM using a query to generated structured data based on the request. paragraphs 4-6, 20-23 and 38-40 teaches an LLM using a query to generated structured data based on the request. Qasemi teaches receiving natural language input seeing an object or event (paragraph 51, step 512) and, in response, “relies on the natural language model infused with the human-augmented knowledge graph to derive and output fine-grained classifications corresponding to the desired object or event” (Paragraph 51 and step 514). Qasemi’s natural language model is a pretrained transformer language model, e.g., RoBERTa (paragraphs 41 and 50). For example, for the input “show me a scene with a garbage truck in the left lane from February 22nd of last year” (paragraph 51), the model resolves a specific object category (garbage truck, a fine grain vehicle type determined “via its node in the knowledge graph” (paragraph 51) and a specific temporal constraint (February 22nd of last year), against a knowledge graph “built on the time scale (e.g. time stamped)” (paragraph 48; see also paragraph 34). Similarly, for “show me suspicious activity last Tuesday,” the system resolves the request into structured node characteristics including “a certain time of night (e.g., between midnight and 4:00am)” (paragraph 53). The knowledge-graph node attributes produced from the request (object type, lane, time) are structured data under the broadest reasonable interpretation. Qasemi then retrieves and outputs “a video segment” matching those constraints (paragraphs 43, 51 and claim 1). The prior motivation as discussed above is incorporated herein.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Nicastri et al. (US 2021/0312188) in view of Lev-Tov et al. (US 2018/0232451) in view of Qasemi et al. (US 2024/0029422) and further in view of Gong et al. (US 2014/0365506).
Regarding claim 5, Nicastri, Lev-Tov and Qasemi teaches the claimed as discussed in claim 1 above, however fails to, but Gong teaches the claimed further comprising:
identifying at least one missing event summarization detail in the request to create an event summarization (paragraphs 47 and 56 teaches a video searching tool that identifies missing detections in the search); and
requesting clarification of the at least one missing event summarization detail (paragraphs 47 and 56 teaches a video searching tool that identifies missing detections in the search and allows an iterative process of asking a user to refine its search to refine the results).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to incorporate the teachings of Gong into the system of Nicastri, Lev-Tov and Qasemi because such an incorporation allows for the benefit of improving the system by improving the accuracy of a video query search and filter down the results to a “correct” or intended video results (paragraph 47).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Nicastri et al. (US 2021/0312188) in view of Lev-Tov et al. (US 2018/0232451) in view of Qasemi et al. (US 2024/0029422) and further in view of Chen et al. (US 2014/0365506).
Regarding claim 11, Nicastri, Lev-Tov and Qasemi teaches the claimed as discussed in claim 1 above, however fails to, but Chen teaches wherein arranging the selected images to be included in the event summarization includes:
adding a first group of images associated with a first topic to the beginning of the event summarization (paragraphs 188-192); and
adding a second group of images associated with a second topic to the end of the event summarization (paragraphs 188-192 teaches wherein search results for a video query is organized in a specific order based on scores, which are based on query inputs/text).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to incorporate the teachings of Chen into the system of Nicastri, Lev-Tov and Qasemi because said incorporation allows for the benefit of improving the quality of the video summary sequence (paragraph 188).
Claims 8-9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Nicastri et al. (US 2021/0312188) in view of Lev-Tov et al. (US 2018/0232451) in view of Qasemi et al. (US 2024/0029422) and further in view of Harary et al. (US 2025/0005727).
Regarding claim 8, Nicastri, Lev-Tov and Qasemi teaches the claimed as discussed in claim 1 above, however, fails to teach, but Harary teaches the claimed further comprising:
encoding the natural language request using a text embedding model (paragraph 39-40);
creating features associated with the encoded natural language request (paragraph 40-44);
identifying features associated with the identified images (paragraph 40-44);
comparing the identified features of the identified images to the features associated with the encoded natural language request (paragraph 40-44, checking for distances between query and normal text embeddings); and
identifying relevant identified images based on the comparison (paragraph 40-44).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to incorporate the teachings of Harary to be incorporated into the alphanumeric search ability of Lev-Tov and Qasemi because such an incorporation would allow for the benefit of improving the accuracy of text/natural language searches (paragraphs 3-6, 17 and 56).
Regarding claim 9, Harary teaches the claimed wherein the features associated with the encoded natural language request and the features of the identified images are stored in a common embedding space (paragraph 40-44). The prior motivation as discussed above is incorporated herein.
Regarding claim 16, Nicastri teaches the claimed as discussed in claim 14 above, however, fails to teach, but Harary teaches the claimed further comprising a multimodal embedding system including:
a text embedding model configured to encode the natural language request and create features associated with the encoded natural language request (paragraph 39-40); and
an image embedding model configured to: encode the identified images (paragraph 39 and 42 teaches image encoder);
create features associated with the encoded identified images (paragraphs 39 and 42);
compare the identified image features with the natural language request features (paragraphs 40-44); and
identify relevant identified images based on the comparison (paragraph 40-44).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to incorporate the teachings of Harary to be incorporated into the alphanumeric search ability of Nicastri, Lev-Tov and Qasemi because such an incorporation would allow for the benefit of improving the accuracy of text/natural language searches (paragraphs 3-6, 17 and 56).
Claims 22-23 are rejected under 35 U.S.C. 103 as being unpatentable over Nicastri et al. (US 2021/0312188) in view of Lev-Tov et al. (US 2018/0232451) in view of Qasemi et al. (US 2024/0029422) and further in view of Xie et al. (US 2009/0024598).
Regarding claim 22, Nicastri and Lev-Tov teaches the claimed as discussed in claim 1 above, however, fails to teach, but Xie teaches the claimed further comprising updating the LLM based on the structured data (paragraphs 12 and 17 teaches a language model being updated/learning new structured data based on initial input).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to incorporate the teachings of Xie into the proposed combinations of Nicastri, Lev-Tov and Qasemi because said incorporation allows for the benefit of improving the language model’s accuracy and give better results to the user (paragraph 6 of Xie).
Regarding claim 23, Xie teaches the claimed wherein updating the LLM includes continually updating LLM training and evaluation data based on user feedback to improve subsequent event summarizations (paragraphs 12 and 17 teaches a language model being updated/learning new structured data based on initial input).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to incorporate the teachings of Xie into the proposed combinations of Nicastri, Lev-Tov and Qasemi because said incorporation allows for the benefit of improving the language model’s accuracy and give better results to the user (paragraph 6 of Xie).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GELEK W TOPGYAL whose telephone number is (571)272-8891. The examiner can normally be reached M-F (9:30-6 PST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Vaughn can be reached at 571-272-3922. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GELEK W TOPGYAL/ Primary Examiner, Art Unit 2481
