DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Amendment
The amendment filed Aug 05, 2026 has been entered. Applicant’s amendments
to the Drawing and Claims have overcome all objections.
Response to Arguments
Applicant’s arguments filed Aug 05, 2026 have been fully considered but they are not persuasive.
Applicant argues that none of the cited references (Wang (US 20240404283 A1), Zhao (ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond." arXiv preprint arXiv:2305.17098 (2023)), Zhao2 ("Learning Video Representations from Large Language Models." arXiv preprint arXiv:2212.04501 (2022)), Hassan (Synthesizing Physical Character-Scene Interactions." arXiv preprint arXiv:2302.00883 (2023); IDS REF), and Wu ("Synthesizing Physical Character-Scene Interactions." arXiv preprint arXiv:2302.00883 (2023)), individually or in any combination, discloses or suggest identifying a split point in a video feed based on a user input, selecting frames at and after that split point as the subset to be altered, and producing an altered video feed that comprises the original video before the split point appended with generated alternate frames after the split point as recited claims 1, 8, and 15.
Examiner replies that Applicant’s argument with respect to claims 1, 8, and 15 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Instead, Harney (US 12626421 B1) is used to teach identifying a split point in a video feed based on a user input, selecting frames at and after that split point as the subset to be altered, and producing an altered video feed that comprises the original video before the split point appended with generated alternate frames after the split point.
Conclusion: The rejections set forth in the previous Office Action are shown to
have been proper, and the claims are rejected below. New citations and parenthetical
remarks can be considered new grounds of rejection, such new grounds of rejection are
necessitated by Applicant’s amendments to the claims. Therefore, the
present Office Action is made final.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 8, and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20240404283 A1) (hereinafter Wang) in view of Zhao et al. (ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond." arXiv preprint arXiv:2305.17098 (2023)) (hereinafter Zhao) and further in view of Harney (US 12626421 B1) (hereinafter Harney).
Regarding claim 1, Wang discloses a computer-implemented method comprising: (claim 1, " A method comprising", para. [0101], "Embodiments of the present disclosure may comprise or utilize a special purpose or general-purpose computer including computer hardware,")
receiving an input; (claim 1, "receiving a video input and a text transcription of the video input"; para. [0022], "In some embodiments, the input 102 may include a user query 102D. The user query 102D is an additional text input but is distinguished from text 102B that transcribes the video 102A. An example user query 102D may be a user prompt asking about content of the video 102A. For example, a user query 102D may be a prompt asking if the video 102A includes/describes a particular concept, when a particular concept is described in the video 120A (e.g., at what time stamp), and the like. When the input 102 includes a user query 102D, the multimodal video summarization system 100 is considered a “query focused multimodal video summarization system,” because the summarized video 120A and summarized text 120B will be biased towards answering the user query 102D. That is, the summarized video 120A and summarized text 120B are dependent on the user query 102D in a query focused multimodal video summarization system.")
receiving a video feed comprising a plurality of frames; (para. [0020] The video 102A can include any digital visual media including a plurality of frames which, when played, includes a moving visual representation of a story and/or an event; claim 1, “receiving a video input and a text transcription of the video input, wherein the video input includes a plurality of frames"”
selecting a subset of the plurality of frames based on the input, (FIG. 5; claim 1, "determining, by a multimodal summarization model, a subset of key frames of the plurality of frames and a subset of key sentences of the plurality of sentences; and providing a summary of the video input and a summary of the text transcription based on the subset of key frames and the subset of key sentences."; para. [0022], “In some embodiments, the input 102 may include a user query 102D. The user query 102D is an additional text input but is distinguished from text 102B that transcribes the video 102A. An example user query 102D may be a user prompt asking about content of the video 102A. For example, a user query 102D may be a prompt asking if the video 102A includes/describes a particular concept, when a particular concept is described in the video 120A (e.g., at what time stamp), and the like. When the input 102 includes a user query 102D, the multimodal video summarization system 100 is considered a “query focused multimodal video summarization system,” because the summarized video 120A and summarized text 120B will be biased towards answering the user query 102D. That is, the summarized video 120A and summarized text 120B are dependent on the user query 102D in a query focused multimodal video summarization system.")
PNG
media_image1.png
625
805
media_image1.png
Greyscale
generating at least one alternate frame based on at least one original frame of the selected subset; (see paragraph 53 and FIG.5)
displaying the altered video feed on a display. (see FIG. 1 and para. [0033], "At numeral 6, the output 120 may be displayed to a user via a computing device,")
Wang does not explicitly disclose replacing the at least one original frame in the selected subset with the at least one alternate frame to generate an altered video feed.
However, Zhao more explicitly teaches a computer implemented method comprising: receiving an input (text); receiving a video feed comprising a plurality of frames (see figures 1, 2, 4); selecting subset of the plurality of frames based on an input (see figures 1, 2, 3, 4); generating at least one alternate frame based on at least one original image frame of the selected subset (text input determines the output image-section);
replacing the at least one original frame in the selected subset with the at least one alternate frame to generate an altered video feed, (see Figure 1;Figure 2; Figure 3; Figure 4; Examiner’s note: based on the text prompt, frames are selected. A video is generated as output in Figure 1 and ‘Edited Video” in Figure 2)
PNG
media_image2.png
588
585
media_image2.png
Greyscale
PNG
media_image3.png
624
694
media_image3.png
Greyscale
PNG
media_image4.png
607
835
media_image4.png
Greyscale
As both Wang and Zhao are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include replacing the at least one original frame in the selected subset with the at least one alternate frame to generate an altered video feed, in the context of video generation, by Wang according to the teaching of Zhao in order to generate a video from textual prompt, thereby reducing manual labor (1 Introduction of Zhao) and improve temporal consistency in text driven video editing methods (6 Conclusion of Zhao).
The combination of Wang and Zhao fails to disclose wherein selecting the subset comprises identifying a split point in the video feed based on the input, and wherein the subset comprises frames at and after the split point and wherein the altered video feed comprises a first portion of the video feed before the split point appended with the at least one alternate frame after the split point.
Harney teaches wherein selecting the subset comprises identifying a split point in the video feed based on the input, and wherein the subset comprises frames at and after the split point and wherein the altered video feed comprises a first portion of the video feed before the split point appended with the at least one alternate frame after the split point. (Col 5 Lines 25-45, The system provides a call-to-action in the video production workspace to enable a user to initiate automated creation of scenes for a branching narrative in the video being created within the video production workspace (step 220). For example, the system may add a “plus sign” button to each scene added to the workspace. When a user clicks on the plus sign, a window pops up in which a user can enter a request for a branch at this point in the video, where the user can provide natural language instructions/requests for the branch creation. In certain embodiments, the user is able to speak a natural language request for the branching scenes. The user may add assets to a scene before requesting that the video branch from that scene, or the user may create a blank scene and have the system create both the parent and child branch scenes. The user may also enter a branch creation request without creating any scenes and have the system generate an entire narrative and all the scenes for the video based on some user-defined instruction (“create an interactive branching video with questions and answer choices for what to do in case of a house fire”); Col 3 Lines 5-29, In one embodiment, a method for automatically and interactively creating branching video scenes comprises the following steps: providing a call to action in the video production workspace to enable a user to initiate automated creation of scenes for a branching narrative in a video being created within the video production workspace; in response to receiving user input to create scenes for a branching narrative in a video being created in the video production workspace, performing the following: identifying a current state of the video in the video production workspace, including identifying all the assets in the current state of the video; obtaining metadata related to assets in the current state of the video; inputting said metadata into a generative AI model configured to generate a branching narrative for the video and output data describing a plurality of branching scenes for the branching narrative; and rendering the branching scenes in the video production workspace in accordance with the data outputted by the generative AI model, wherein the branching scenes are incorporated into a video timeline such that child branching scenes follow a parent branch scene.)
As Wang, Zhao, and Harney are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include wherein selecting the subset comprises identifying a split point in the video feed based on the input, and wherein the subset comprises frames at and after the split point and wherein the altered video feed comprises a first portion of the video feed before the split point appended with the at least one alternate frame after the split point, in the context of video generation, by the combination of Wang and Zhao according to the teaching of Harney in order to significantly reduce time and effort required when automating and streamlining the management and production of branching videos (Col 2 Lines 9-12 of Harney).
Regarding claim 8, claim 8 has similar limitations as of claim 1, except claim 8 is an alternative replay generation system, comprising: an electronic device comprising a processor, the processor configured to cause the electronic device to (Wang, FIG. 5; para. [0090], “The one or more servers can provide the key frames and key text to the client device for display to the user.”; para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”). Therefore, the claim is rejected under the same rationale as claim 1.
Regarding claim 15, claim 15 has similar limitations as of claim 1 and 8, except claim 15 has a non-transitory computer-readable medium comprising program code, that when executed by at least one processor of an electronic device, causes the electronic device to: (Wang, para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”; para. [0101], “ In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory, etc.), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.”; para [0103], “Non-transitory computer-readable storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.”) Therefore, the claim is rejected under the same rationale as claim 1 and 8.
Regarding claim 16, Wang discloses the non-transitory computer-readable medium of Claim 15, wherein the program code, that when executed by the at least one processor, causes the electronic device to generate the at least one alternate frame based on the at least one original frame of the subset, comprises program code, that when executed by the at least one processor, causes the electronic device to: (Wang, para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”; para. [0101], “ In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory, etc.), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.”; para [0103], “Non-transitory computer-readable storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.”)
receive a requested alteration prompt from a prompt parser module based on the input and (Wang, para. [0022], “In some embodiments, the input 102 may include a user query 102D. The user query 102D is an additional text input but is distinguished from text 102B that transcribes the video 102A. An example user query 102D may be a user prompt asking about content of the video 102A. For example, a user query 102D may be a prompt asking if the video 102A includes/describes a particular concept, when a particular concept is described in the video 120A (e.g., at what time stamp), and the like. When the input 102 includes a user query 102D, the multimodal video summarization system 100 is considered a “query focused multimodal video summarization system,” because the summarized video 120A and summarized text 120B will be biased towards answering the user query 102D. That is, the summarized video 120A and summarized text 120B are dependent on the user query 102D in a query focused multimodal video summarization system.")
Wang does not explicitly disclose generate the at least one alternate frame of a target object and other elements of a scene in the selected subset to match the requested alteration prompt.
However, Zhao more explicitly disclose generate the at least one alternate frame of a target object and other elements of a scene in the selected subset to match the requested alteration prompt. (Figure 1; Figure 4)
As Wang, Zhao, and Harney are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include generating the at least one alternate frame of a target object and other elements of a scene in the selected subset to match the requested alteration prompt, in the context of video generation, by Wang according to the teaching of Zhao in order to generate a video from textual prompt, thereby reducing manual labor (1 Introduction of Zhao) and improve temporal consistency in text driven video editing methods (6 Conclusion of Zhao)
Claims 2, 9, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20240404283 A1) (hereinafter Wang) in view of Zhao et al. (ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond." arXiv preprint arXiv:2305.17098 (2023)) (hereinafter Zhao) and further in view of Harney (US 12626421 B1) (hereinafter Harney) and Zhao et el. ("Learning Video Representations from Large Language Models." arXiv preprint arXiv:2212.04501 (2022)) (hereinafter Zhao2)
Regarding claim 2, the combination of Wang, Zhao, and Harney discloses the method of Claim 1, (Wang, para. [0035], “"In other embodiments, the multimodal summarization system 100 executes one or more machine learning models (such as any language model) to generate a refined/comprehensive text summary. For example, as opposed to concatenating sentences/phrases of text received as summarized text 120B, a language model (not shown) receives the summarized text 120B as a prompt and refines the summarized text 120B by adding punctuation, transitional language, removing redundant words, and the like."; para. [0100], “In some embodiments, key text sentences are concatenated to provide the summary of the text transcription. In other embodiments, one or more machine learning models (such as any language model) are executed to generate a refined/comprehensive text summary. For example, as opposed to concatenating sentences/phrases of text received, a language model receives the identified key sentences as a prompt and creates a refined summary of the key sentences by adding punctuation, transitional language, removing redundant words, and the like.")
The combination of Wang, Zhao, and Harney does not explicitly disclose wherein receiving a video feed comprises using a multimodal language model to generate a frame-by-frame narration and
However, Zhao2 more explicitly teaches wherein receiving a video feed comprises using a multimodal language model to generate a frame-by-frame narration and (Zhao2, Figure 2; 1. Introduction, “Our method, called LAVILA: Language-model augmented Video-Language pre-training, leverages pre trained LLMs, e.g. GPT-2 [50], which encode within their weights a treasure trove of factual knowledge and conversational ability. As shown in Figure 2, we repurpose these LLMs to be “visually-conditioned narrators”, and finetune on all accessible paired video-text clips. Once trained, we use the model to densely annotate thousands of hours of videos by generating rich textual descriptions”)
PNG
media_image5.png
685
653
media_image5.png
Greyscale
As Wang, Zhao, Zhao2, and Harney are from the same field of endeavor, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to include wherein receiving a video feed comprises using a multimodal language model to generate a frame-by-frame narration, in the context of video generation, by the combination of Wang, Zhao, and Harney according to the teaching of Zhao2 in order to generate text that is well-aligned with the visual input (1. Introduction of Zhao2).
Regarding claims 9 and 17, claims 9 and 17 have similar limitations of claim 2 except claim 9 has alternative replay generation system, the processor, electronic device (Wang, FIG. 5; para. [0090], “The one or more servers can provide the key frames and key text to the client device for display to the user.”; para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”), and claim 17 has non-transitory computer-readable medium, the program, at least one processor, and electronic device (Wang, para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”; para. [0101], “ In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory, etc.), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.”; para [0103], “Non-transitory computer-readable storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.”) Therefore, claims 9 and 17 are rejected under the same rationale as claim 2.
Claims 3, 10, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20240404283 A1) (hereinafter Wang) in view of Zhao et al. (ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond." arXiv preprint arXiv:2305.17098 (2023)) (hereinafter Zhao) and further in view of Harney (US 12626421 B1) (hereinafter Harney) and Hassan et el. ("Synthesizing Physical Character-Scene Interactions." arXiv preprint arXiv:2302.00883 (2023); IDS REF) (hereinafter Hassan).
Regarding claim 3, the combination of Wang, Zhao, and Harney discloses the method of Claim 1, (Zhao, Figure 1; Figure 4)
The combination of Wang, Zhao, and Harney does not disclose the method of Claim 1, further comprising inputting the altered video feed into an adversarial learning architecture configured to determine a plausibility of
However, Hassan more explicitly teaches the method of Claim 1, further comprising inputting the altered video feed into an adversarial learning architecture configured to determine a plausibility of (page. 1, “These scene interactions are learned using an adversarial discriminator that evaluates the realism of a motion within the context of a scene.”)
As Wang, Zhao, Harney, and Hassan are from the same field of endeavor, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to include inputting the altered video feed into an adversarial learning architecture configured to determine a plausibility, in the context of video generation, by the combination of Wang, Zhao, and Harney according to the teaching of Hassan in order to synthesize interaction between virtual characteristics and their surroundings (page 1 of Hassan).
Regarding claims 10 and 18, claims 10 and 18 have similar limitations of claim 3 except claim 10 has alternative replay generation system, the processor, electronic device (Wang, FIG. 5; para. [0090], “The one or more servers can provide the key frames and key text to the client device for display to the user.”; para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”) and claim 18 has non-transitory computer-readable medium, the program, at least one processor, and electronic device (Wang, para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”; para. [0101], “ In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory, etc.), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.”; para [0103], “Non-transitory computer-readable storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.”) Therefore, claims 10 and 18 are rejected under the same rationale as claim 3.
Claims 4, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20240404283 A1) (hereinafter Wang) in view of Zhao et al. (ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond." arXiv preprint arXiv:2305.17098 (2023)) (hereinafter Zhao) and further in view of Harney (US 12626421 B1) (hereinafter Harney) and Wu et el. ("Synthesizing Physical Character-Scene Interactions." arXiv preprint arXiv:2302.00883 (2023)) (hereinafter Wu).
Regarding claim 4, the combination of Wang, Zhao, and Harney discloses the method of Claim 1, wherein generating the at least one alternate frame based on the at least one original frame of the subset comprises: (see supra rejection of claim 1)
The combination of Wang, Zhao, and Harney does not disclose analyzing a movement of a target object in a scene of the at least one original frame;
predicting a future trajectory of the target object; and
generating other elements of the scene based on the future trajectory of the target object and previously-generated alternate frames of the at least one alternate frame.
However, Wu more explicitly teaches analyzing a movement of a target object in a scene of the at least one original frame; (Figure 2; 3.1 Moving object detection; "we train a moving object detection network to identify moving objects and static scenes. Based on Cityscapes-Motion dataset and KITTI-Motion dataset [37] that provide annotations of moving areas, we build an encoder-decoder architecture to detect moving regions, with ResNet50 as the backbone [14]."
predicting a future trajectory of the target object; and (trajectory prediction in Figure 2)
generating other elements of the scene based on the future trajectory of the target object and previously-generated alternate frames of the at least one alternate frame. (Composition in Figure 2)
PNG
media_image6.png
533
1103
media_image6.png
Greyscale
As Wang, Zhao, Harney, and Wu are from the same field of endeavor, it would
have been obvious to one of the ordinary skill in the art before the effective filing date of
the claimed invention to include disclose analyzing a movement of a target object in a scene of the at least one original frame; predicting a future trajectory of the target object; and generating other elements of the scene based on the future trajectory of the target object and previously-generated alternate frames of the at least one alternate frame, in the context of video generation, by the combination of Wang, Harney, and Zhao according to the teaching of Wu in order to predict video frames based on a sequence of continuous video frames in the past (Abstract of Wu).
Regarding claim 11, claim 11 has similar limitations as of claim 4, except claim 11 has an alternative replay generation system, comprising: an electronic device comprising a processor, the processor configured to cause the electronic device to (Wang, FIG. 5; para. [0090], “The one or more servers can provide the key frames and key text to the client device for display to the user.”; para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”). Therefore, the claim is rejected under the same rationale as claim 4.
Claims 5-7 and 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20240404283 A1) (hereinafter Wang) in view of Zhao et al. (ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond." arXiv preprint arXiv:2305.17098 (2023)) (hereinafter Zhao) and further in view of Harney (US 12626421 B1) (hereinafter Harney) and Eggert et el. (US 20110129119 A1) (hereinafter Eggert).
Regarding claim 5, the combination of Wang, Zhao, and Harney discloses the method of Claim 1, wherein receiving the video feed comprising the plurality of frames comprises: (see supra rejection of claim 1)
selecting a subset of frames from the plurality of frames using a large language model; (Wang, FIG. 5; para. [0022], “In some embodiments, the input 102 may include a user query 102D. The user query 102D is an additional text input but is distinguished from text 102B that transcribes the video 102A. An example user query 102D may be a user prompt asking about content of the video 102A. For example, a user query 102D may be a prompt asking if the video 102A includes/describes a particular concept, when a particular concept is described in the video 120A (e.g., at what time stamp), and the like. When the input 102 includes a user query 102D, the multimodal video summarization system 100 is considered a “query focused multimodal video summarization system,” because the summarized video 120A and summarized text 120B will be biased towards answering the user query 102D. That is, the summarized video 120A and summarized text 120B are dependent on the user query 102D in a query focused multimodal video summarization system."; para. [0035], "n some embodiments, the score predictor 110 concatenates the key text sentences to generate the summarized text 120B. In other embodiments, the multimodal summarization system 100 executes one or more machine learning models (such as any language model) to generate a refined/comprehensive text summary. For example, as opposed to concatenating sentences/phrases of text received as summarized text 120B, a language model (not shown) receives the summarized text 120B as a prompt and refines the summarized text 120B by adding punctuation, transitional language, removing redundant words, and the like."; para. [0051], "The video frame selector 406 identifies the k-most relevant frames associated with the video input based on identifying the top k-segments with the highest relevance score. As a result, the video frame selector 406 identifies the subset of most relevant frames to the video input as summarized video 120A of output 120.")
identifying a status of a target object in each frame of the selected subsets of frames; and (Zhao, Figure 1; Figure 2)
PNG
media_image7.png
771
827
media_image7.png
Greyscale
(FIG. 5; para. [0022], “In some embodiments, the input 102 may include a user query 102D. The user query 102D is an additional text input but is distinguished from text 102B that transcribes the video 102A. An example user query 102D may be a user prompt asking about content of the video 102A. For example, a user query 102D may be a prompt asking if the video 102A includes/describes a particular concept, when a particular concept is described in the video 120A (e.g., at what time stamp), and the like. When the input 102 includes a user query 102D, the multimodal video summarization system 100 is considered a “query focused multimodal video summarization system,” because the summarized video 120A and summarized text 120B will be biased towards answering the user query 102D. That is, the summarized video 120A and summarized text 120B are dependent on the user query 102D in a query focused multimodal video summarization system.”; para. [0035], "n some embodiments, the score predictor 110 concatenates the key text sentences to generate the summarized text 120B. In other embodiments, the multimodal summarization system 100 executes one or more machine learning models (such as any language model) to generate a refined/comprehensive text summary. For example, as opposed to concatenating sentences/phrases of text received as summarized text 120B, a language model (not shown) receives the summarized text 120B as a prompt and refines the summarized text 120B by adding punctuation, transitional language, removing redundant words, and the like."; para. [0051], "The video frame selector 406 identifies the k-most relevant frames associated with the video input based on identifying the top k-segments with the highest relevance score. As a result, the video frame selector 406 identifies the subset of most relevant frames to the video input as summarized video 120A of output 120."; para. [0037], "the feature extractor 104 extracts features of input 102 (including video 102A and text 102B) and creates multimodality embeddings (e.g., video embedding 214A and text embedding 214B)"
The combination of Wang, Zhao, and Harney does not disclose generating a multi-object tracking model,
However, Eggert more explicitly teaches generating a multi-object tracking model, (claim 1, “A method for tracking objects based on sensory data (3) supplied from streaming sensors such as e.g. video camera(s) (30, 31), the method comprising the following steps: processing the sensory input data (3) using one or more tracker models (9), each tracker model comprising a tracker prediction and a measurement process, deciding (16) whether the sensory data (3) contain interesting parts not yet covered by a tracker model (9), and in the positive case, initializing new tracker objects,; para [0002], “Generally a tracking system in the present invention processes sensory data and generates control signals in order to make a sensor track one or more targets in the input field of the sensor(s), or
indicates the region of one or more tracked targets in the input field of the sensor(s).")
As Wang, Zhao, Harney, and Eggert are from the same field of endeavor, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to include generating a multi-object tracking model, in the context of video generation, by the combination of Wang, Zhao, and Harney according to the teaching of Eggert in order to make a sensor track one or more targets in the input field of the sensor(s) (para. [0003] of Eggert).
Regarding claim 6, the combination of Wang, Zhao, and Harney discloses the method of Claim 5, wherein selecting a subset of frames of the plurality of frames based on the input (see supra rejection of claim 5) comprises selecting a subset of frames of the plurality of frames of the video feed that matches a requested subset of frames of the input. (Wang, FIG5; para. [0022], "In some embodiments, the input 102 may include a user query 102D. The user query 102D is an additional text input but is distinguished from text 102B that transcribes the video 102A. An example user query 102D may be a user prompt asking about content of the video 102A. For example, a user query 102D may be a prompt asking if the video 102A includes/describes a particular concept, when a particular concept is described in the video 120A (e.g., at what time stamp), and the like. When the input 102 includes a user query 102D, the multimodal video summarization system 100 is considered a “query focused multimodal video summarization system,” because the summarized video 120A and summarized text 120B will be biased towards answering the user query 102D. That is, the summarized video 120A and summarized text 120B are dependent on the user query 102D in a query focused multimodal video summarization system.”)
Regarding claim 7, the combination of Wang, Zhao, and Harney discloses the method of Claim 6, wherein generating at least one alternate frame based on at least one original frame of the selected subset (see supra rejection of claim 6) comprises:
Wang discloses receiving a requested alteration prompt from the prompt parser module based on the input and (Wang, para. [0022], “In some embodiments, the input 102 may include a user query 102D. The user query 102D is an additional text input but is distinguished from text 102B that transcribes the video 102A. An example user query 102D may be a user prompt asking about content of the video 102A. For example, a user query 102D may be a prompt asking if the video 102A includes/describes a particular concept, when a particular concept is described in the video 120A (e.g., at what time stamp), and the like. When the input 102 includes a user query 102D, the multimodal video summarization system 100 is considered a “query focused multimodal video summarization system,” because the summarized video 120A and summarized text 120B will be biased towards answering the user query 102D. That is, the summarized video 120A and summarized text 120B are dependent on the user query 102D in a query focused multimodal video summarization system.")
Wang does not disclose generating the at least one alternate frame of a target object and other elements of a scene in the selected subset to match the requested alteration prompt.
Zhao more explicitly teaches generating the at least one alternate frame of a target object and other elements of a scene in the selected subset to match the requested alteration prompt. (Zhao, Figure 1; Figure 4)
As Wang, Zhao, Harney are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include generating the at least one alternate frame of a target object and other elements of a scene in the selected subset to match the requested alteration prompt, in the context of video generation, by Wang according to the teaching of Zhao in order to generate a video from textual prompt, thereby reducing manual labor (1 Introduction of Zhao) and improve temporal consistency in text driven video editing methods (6 Conclusion of Zhao).
Regarding claims 12 and 13, claims 12 and 13 have similar limitations of claim 5 except claims 12 and 13 have alternative replay generation system, the processor, electronic device (Wang, FIG. 5; para. [0090], “The one or more servers can provide the key frames and key text to the client device for display to the user.”; para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”). Therefore, claims 12 and 13 are rejected under the same rationale as claim 5.
Regarding claim 14, claim 14 has similar limitations as of claim 7 except the claim 14 is an alternative replay generation system claim and has the processor when causing the electronic device, and the electronic device (Wang, FIG. 5; para. [0090], “The one or more servers can provide the key frames and key text to the client device for display to the user.”; para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”) Therefore, the claim is rejected under the same rationale as claim 7.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20240404283 A1) (hereinafter Wang) in view of Zhao et al. (ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond." arXiv preprint arXiv:2305.17098 (2023)) (hereinafter Zhao) and further in view of Harney (US 12626421 B1) (hereinafter Harney), Hassan et el. ("Synthesizing Physical Character-Scene Interactions." arXiv preprint arXiv:2302.00883 (2023); IDS REF) (hereinafter Hassan), and Wu et el. ("Synthesizing Physical Character-Scene Interactions." arXiv preprint arXiv:2302.00883 (2023)) (hereinafter Wu).
Regarding claim 19, the combination of Wang, Zhao, Harney, and Hassan discloses the non-transitory computer-readable medium of Claim 18, wherein the program code, that when executed by the at least one processor, causes the electronic device to generate the at least one alternate frame based on the at least one original frame of the subset, comprises program code, that when executed by the at least one processor, causes the electronic device to: (see supra rejection of claim 18; Wang, para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”; para. [0101], “ In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory, etc.), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.”; para [0103], “Non-transitory computer-readable storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.”)
the combination of Wang, Zhao, Harney, and Hassan dees not explicitly disclose analyze a movement of a target object in a scene of the at least one original frame; predict a future trajectory of the target object; and generate other elements of the scene based on the future trajectory of the target object and previously-generated alternate frames of the at least one alternate frame.
However, Wu more explicitly teaches analyze a movement of a target object in a scene of the at least one original frame; (Wu, Figure 2; 3.1 Moving object detection; "we train a moving object detection network to identify moving objects and static scenes. Based on Cityscapes-Motion dataset and KITTI-Motion dataset [37] that provide annotations of moving areas, we build an encoder-decoder architecture to detect moving regions, with ResNet50 as the backbone [14].")
predict a future trajectory of the target object; and (Wu, (trajectory prediction in Figure 2)
generate other elements of the scene based on the future trajectory of the target object and previously-generated alternate frames of the at least one alternate frame. (Wu, Composition in Figure 2)
As Wang, Zhao, Harney, Hassan, and Wu are from the same field of endeavor, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to include analyze a movement of a target object in a scene of the at least one original frame; predict a future trajectory of the target object; and generate other elements of the scene based on the future trajectory of the target object and previously-generated alternate frames of the at least one alternate frame, in the context of video generation, by the combination of Wang, Zhao, Harney, and Hassan according to the teaching of Wu in order to predict video frames based on a sequence of continuous video frames in the past (Abstract of Wu).
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20240404283 A1) (hereinafter Wang) in view of Zhao et al. (ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond." arXiv preprint arXiv:2305.17098 (2023)) (hereinafter Zhao) and further in view of Harney (US 12626421 B1) (hereinafter Harney), Hassan et el. ("Synthesizing Physical Character-Scene Interactions." arXiv preprint arXiv:2302.00883 (2023); IDS REF) (hereinafter Hassan), Wu et el. ("Synthesizing Physical Character-Scene Interactions." arXiv preprint arXiv:2302.00883 (2023)) (hereinafter Wu), and Eggert et el. (US 20110129119 A1) (hereinafter Eggert.
Regarding claim 20, the combination of Wang, Zhao, Harney, Hassan, and Wu discloses the non-transitory computer-readable medium of Claim 19, wherein the program code, that when executed by the at least one processor, causes the electronic device to receive a video feed, comprises program code, that when executed by the at least one processor, causes the electronic device to: (Wang, para. [0086], “The components and their corresponding elements can comprise software, hardware, or both. For example, the components and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the multimodal video summarization system 900 can cause a client device and/or a server device to perform the methods described herein.”; para. [0101], “ In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory, etc.), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.”; para [0103], “Non-transitory computer-readable storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.”)
select a subset of frames from the plurality of frames using a large language model; (Wang, FIG. 5; para. [0022], “In some embodiments, the input 102 may include a user query 102D. The user query 102D is an additional text input but is distinguished from text 102B that transcribes the video 102A. An example user query 102D may be a user prompt asking about content of the video 102A. For example, a user query 102D may be a prompt asking if the video 102A includes/describes a particular concept, when a particular concept is described in the video 120A (e.g., at what time stamp), and the like. When the input 102 includes a user query 102D, the multimodal video summarization system 100 is considered a “query focused multimodal video summarization system,” because the summarized video 120A and summarized text 120B will be biased towards answering the user query 102D. That is, the summarized video 120A and summarized text 120B are dependent on the user query 102D in a query focused multimodal video summarization system."; para. [0035], "n some embodiments, the score predictor 110 concatenates the key text sentences to generate the summarized text 120B. In other embodiments, the multimodal summarization system 100 executes one or more machine learning models (such as any language model) to generate a refined/comprehensive text summary. For example, as opposed to concatenating sentences/phrases of text received as summarized text 120B, a language model (not shown) receives the summarized text 120B as a prompt and refines the summarized text 120B by adding punctuation, transitional language, removing redundant words, and the like."; para. [0051], "The video frame selector 406 identifies the k-most relevant frames associated with the video input based on identifying the top k-segments with the highest relevance score. As a result, the video frame selector 406 identifies the subset of most relevant frames to the video input as summarized video 120A of output 120.")
identify a status of a target object in each frame of the selected subsets of frames; and (Zhao, Figure 1; Figure 2)
The combination of Wang, Zhao, Harney, Hassan, and Wu does not explicitly disclose generate a multi-object tracking model.
However, Eggert more explicitly teaches generate a multi-object tracking model. (Eggert, claim 1, “A method for tracking objects based on sensory data (3) supplied from streaming sensors such as e.g. video camera(s) (30, 31), the method comprising the following steps: processing the sensory input data (3) using one or more tracker models (9), each tracker model comprising a tracker prediction and a measurement process, deciding (16) whether the sensory data (3) contain interesting parts not yet covered by a tracker model (9), and in the positive case, initializing new tracker objects,; para [0002], “Generally a tracking system in the present invention processes sensory data and generates control signals in order to make a sensor track one or more targets in the input field of the sensor(s), or
indicates the region of one or more tracked targets in the input field of the sensor(s).")
As Wang, Zhao, Harney, Wu, and Eggert are from the same field of endeavor, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to include generate a multi-object tracking model, in the context of video generation, by the combination of Wang, Zhao, Harney, Hassan, and Wu according to the teaching of Eggert in order to make a sensor track one or more targets in the input field of the sensor(s) (para. [0003] of Eggert).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hyorim Park whose telephone number is (571)272-3859. The examiner can normally be reached Monday - Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Hyorim Park/Examiner, Art Unit 2615
/ALICIA M HARRINGTON/Supervisory Patent Examiner, Art Unit 2615
