DETAILED ACTION
This communication is in response to the Applicant Arguments/Remarks filed 2/2/2026. Claims 1-20 are pending in the application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 2/2/2026 have been fully considered but they are not persuasive. Regarding the argument on pages 11-12 that “Kreis does not teach obtaining a set of video embeddings for a video clip, combining the set of prompt embeddings (representing a query) and a set of input embeddings to generate an input tensor, generating a set of output embeddings by applying a decoder to the input tensor, and converting the output embeddings into a response to the query that includes at least text describing a portion of the video with respect to the query”, examiner respectfully disagrees.
In response to applicant’s arguments against the references individually, one cannot show non-obviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). The teachings of Kreis were not applied to teach the step of “obtaining a set of video embeddings for a video clip”. In addition, it is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. "The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain." In re Heck, 699 F.2d 1331, 1332-33,216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275,277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claims above for the convenience of the Applicants. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claims, typically other passages and figures will apply as well.
Ghoshal teaches at para. 5: converting input text and/or images into vectors (equivalent to multi-modal encoder) within a multi-dimensional vector space, and comparing input content to a plurality of repository content to find a number of relevant content options within the content repository… relevant content items (e.g., images, audio and/or video clips, links to related articles, etc.); para. 58: content retrieval and embedding service.
Regarding the amended limitations in the independent claims 1 and 11 in relating to the generating, combining and converting steps, please see the new combination of references with columns and lines cited below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 10-12, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghoshal et al. (US 20200125575) in view of Fayyaz et al. (US 20240354317) and further in view of Kreis et al. (US 20240171788).
As per claims 1, 11, Ghoshal et al. (US 2020125575) teaches
a method, comprising: obtaining a query on content of a video and a request for one or more responses to the query; identifying one or more video clips in the video, each video clip including a segment of the video (para. 35: evaluate and rank content items from a content repository in response to input content received from user or client systems; para. 56, 179: a user may input one or more search terms into a search engine query interface, the recommendation system may recommend one or more relevant content items (e.g., web pages, documents, images, etc.) that most closely match the input search terms; para. 181: upon receiving user input (e.g., a search term/phrase, content being authored by a user) for which recommendation of content items is to be provided, the user input may be analyzed to identify one or more tags to be associated with the user input; fig. 3: requests may be received through a browser client);
obtaining a set of video embeddings representing the content of a video clip in a latent space, wherein the set of video embeddings are generated by: extracting frame data, audio data, or text data from the video content (fig. 4: topic vector space, image vector space, content retrieval/embeddings; para. 5: comparisons
may include thorough and exhaustive deep searches and/or more efficient tag-based filtered searches. Finally, relevant content items (e.g., images, audio and/or video
clips, links to related articles, etc.), may be retrieved and presented to the content author to be reviewed and embedded within the original authored content; para. 86: determine that the image tag "Mountaineer" sufficiently close within the word vector space to the extracted key words, and thus should be considered as an image tag match for the filtered feature space comparison; para. 125: using the retrieval /embedding component, the potentially related content resources may be provided to the user via the user interface);
Ghoshal teaches at para. 5: converting input text and/or images into vectors (equivalent to multi-modal encoder) within a multi-dimensional vector space, and comparing input content to a plurality of repository content to find a number of relevant content options within the content repository… relevant content items (e.g., images, audio and/or video clips, links to related articles, etc.); para. 58: content retrieval and embedding service. Ghoshal et al. does not explicitly teach the limitation multi-modal.
Fayyaz et al. teaches
applying a machine-learned multi-modal video encoder to the frame data, the audio data, or the text data to generate the set of video embeddings (para. 61: as a whole, the encoder system uses a unified framework for mapping different kinds of multimodal input items into target item embeddings, in which all input items are ultimately treated as language-bearing items; para. 57: the video preprocessor produces tokens that characterize individual video clips, each of which includes one or more frames. The video encoder (not shown) maps the video tokens into a video embedding, corresponding to one or more distributed vectors. The video encoder can use any technology to perform this task, including a CNN-based model, a transformer-based model, and so on).
generating a set of prompt embeddings from a machine-learned text encoder representing at least a portion of the query in a latent space, wherein the set of prompt embeddings are generated by applying the text encoder to text data in the query (para. 42: as input information, the language model receives the query embedding, the target item embeddings in the candidate set, and prompt information. The prompt information provides a text-based narrative that instructs the filtering system to find the most relevant target item embeddings and/or to find the least relevant target item embeddings; para. 46: the input-embedding system, this system includes a plurality of input-encoding subsystems. The subsystems include a text encoder for producing a text embedding based on text content, an image encoder for producing an image embedding based on image content, and an audio encoder for producing an audio embedding based on audio content, and so on. This list is non-exhaustive: other input encoders include a video encoder for processing video content, a three-dimensional-data encoder for processing three-dimensional data);
combining the set of prompt embeddings and a set of input embeddings from at least an output of the multi-modal video encoder to generate an input tensor (para. 3: the encoder system processes input items expressed using any input mode or any combination of two or more input modes. Similarly, the retrieval system allows a user to express an input query using any input mode or combination of input modes. Illustrative input modes include a text input mode, an image input mode, an audio input mode, a video input mode, etc.; para. 29-30: a “multimodal item,” as the term is used herein, refers to an item that includes one or more types of content produced by one or more corresponding input modes. For instance, one kind of multimodal item corresponds to an image item together with a textual caption. the user may explicitly instruct the encoder system to create target item embeddings for the user's document items, image items, audio items, video items, etc.; para. 90-93: each downstream transformer component operates on a sequence of input vectors (equivalent to 1D tensors) produced by the preceding transformer component in the pipeline. The attention component produces query information Q by multiplying the input vectors by a query weighting matrix Wº (equivalent to 2D tensor)).
applying at least a component of a machine-learned decoder to the input tensor to generate an output including a set of output embeddings (fig. 1: top K target item embeddings, query embedding, prompt information, filtering system to user interface system; fig. 3: selected embeddings; para. 52, 63-64: the lookup system performs an exhaustive search through all of the target item embeddings in the retrieval index 106 to find the K target item embeddings having the closest distance to the query input embedding. The mapping component maps the candidate set into the same language-based vector space as the filtering component, to produce a set of transformed target item embeddings ("transformed set" for brevity). The filtering component autoregressively maps the transformed set into output results. The output results identify the members of the candidate set that are most relevant to the input query
if any), and/or the members of the candidate set that are not relevant to the input query);
converting the set of output embeddings into a response to the query based on the video content of the video clip, wherein the response converted from the set of output embeddings includes at least text describing a portion of the video with respect to the query; providing the response to a user of the client device (para. 40, 45-46: the encoder system includes two main systems: an input-embedding system and an embedding mapping system. The input-embedding system maps the input item to an input-system embedding. The embedding-mapping system maps the input-system
Embedding to the target item embedding. Broadly stated, the encoder system uses the item-embedding system to convert text and non-text content into the language-based vector space of the embedding-mapping system. Referring first to the input-embedding system, this system includes a plurality of input-encoding sub systems. The subsystems include a text encoder for producing a text embedding based on text content, an image encoder for producing an image embedding based on image content, and an audio encoder for producing an audio embedding based on audio content, and so on. This list is non-exhaustive: other input encoders include a video encoder for processing video
content, a three-dimensional-data encoder for processing three-dimensional data; para. 42: the language model receives the query embedding, the target item embeddings in the candidate set, and prompt information. The prompt information provides a text-based narrative that instructs the filtering system to find the most relevant target item embeddings and/or to find the least relevant target item embeddings; para. 57).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal to include a machine-learned multi-modal video encoder to the frame data of Fayyaz in order to allow a user to retrieve target items based on an input query that includes content produced by any input mode or combination input modes. Further, the technique is extensible – See Fayyaz, para. 7.
Ghoshal and Fayyaz et al. do not explicitly teach the limitation tensor.
Kreis teaches
applying at least a component of a machine-learned decoder to the input tensor to generate an output including a set of output embeddings (para. 31-32: an LDM can include an encoder (e. g., an encoder neural network) and a decoder (e.g., a decoder neural network). An encoder of the LDM can compress and can maps the input (e.g., an image) from an image or pixel space to a lower dimensional, compressed latent space (e.g., latent tensors, latent representations, and so on) where the diffusion model can be trained more efficiently. A decoder of the LDM can map the latent encoding back to the image space. Accordingly, the diffusion model can be trained in the compressed latent space, which is memory-efficient. The video diffusion model fine-tuned from such an LDM can model an increased number of video frames at the same time given a fixed memory budget as compared to operating in image or pixel space directly, thus facilitating long-term video generation; para. 70-71, 107, 119: one or
more types of applications may include a machine learning application, including
training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc…); fig. 5);
converting the set of output embeddings into a response to the query based on the video content of the video clip, wherein the response converted from the set of output embeddings includes at least text describing a portion of the video with respect to the query; providing the response to a user of the client device (fig. 1: video data, encoder, decoder, output response 175; para. 3: an image diffusion model pre-trained on image data can be converted into a video generator by introducing a temporal dimension to a latent space of the image diffusion model. The video generator can be fine-tuned using encoded image sequences or video data; para. 18-19: generates a response or output based on input video data; para. 47: the training system can convert the image diffusion model to a video diffusion model; para. 70-71; figs. 5-6).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal and Fayyaz to include applying at least a component of a machine-learned decoder to the input tensor to generate an output of Kreis in order to effectively generate and provide relevant output video responses to the users.
As per claims 2, 12, Ghoshal et al. teaches
wherein applying the machine-learned video encoder further comprises: applying a visual encoder to the frame data to generate a set of visual embeddings, applying an audio encoder to the audio data to generate a set of audio embeddings, applying a text encoder to the text data to generate a set of text embeddings (para. 4-5: relevant content items (e.g., images, audio and/or video clips, links to related articles, etc.), may be retrieved and presented to the content author to be reviewed and embedded within the original authored content);
applying a multi-modal encoder to the set of visual embeddings, the set of audio embeddings, and the set of text embeddings to generate a set of video embeddings for the video clip (para. 5: converting input text and/or images into vectors (equivalent to multi-modal encoder) within a multi-dimensional vector space, and comparing input content to a plurality of repository content to find a number of relevant content options within the content repository… relevant content items (e.g., images, audio and/or video clips, links to related articles, etc.); para. 58: content retrieval and embedding service; para. 61-62: text topic model, trained image model; para. 55-56: converting the original content ( e.g., input text and/or images) into vectors within a multi-dimensional vector space, ( e) comparing such vectors to a plurality of other content vectors, each of which represents additional content in a content repository, in order to find and identify various potentially-relevant additional content related to the original content input authored by the user/author, and finally (f) retrieve and present the identified additional content to the author via the smart digital content recommendation tool; para. 125, 209: the input content received may be an image, graphic, audio input, or any other text and/or multimedia content which is generated or selected by the user via a client device).
Fayyaz et al. also teaches a multi-modal encoder at para. 61: as a whole, the encoder system uses a unified framework for mapping different kinds of multimodal input items into target item embeddings, in which all input items are ultimately treated as language-bearing items; para. 57: the video preprocessor produces tokens that characterize individual video clips, each of which includes one or more frames. The video encoder (not shown) maps the video tokens into a video embedding, corresponding to one or more distributed vectors. The video encoder can use any technology to perform this task, including a CNN-based model, a transformer-based model, and so on.
As per claims 10, 20, Ghoshal et al. does not teach attention output.
Fayyaz teaches
wherein the video encoder or the decoder includes a transformer architecture including one or more attention layers, each attention layer coupled to receive a set of inputs and generate a query, a key, a value, and generate an attention
output (para. 52, 54: encoders apply transformer-based models for mapping tokens into corresponding embeddings; para. 57: the video preprocessor produces tokens that characterize individual video clips, each of which includes one or more frames. The video encoder (not shown) maps the video tokens into a video embedding, corresponding to one or more distributed vectors. The video encoder can use any technology to perform this task, including a CNN-based model, a transformer-based model, and so on; para. 91-94: the attention component produces query information Q by multiplying the input vectors by a query weighting matrix Wº (equivalent to 2D tensor). The attention component takes the Softmax (normalized exponential function) of the scaled result, and then multiplies the result of the Softmax operation by V, to produce attention output information. The attention heads perform the computations described above using different respective sets of query, key, and value weight matrices).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal to include each attention layer coupled to receive a set of inputs and generate a query, a key, a value, and generate an attention output of Fayyaz to sufficiently understand input context, relationships, and sequential data and generate relevant outputs to the users.
Claim(s) 3, 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghoshal et al. (US 2020125575) in view of Fayyaz et al. (US 20240354317) and further in view of Kreis et al. (US 20240171788) and Ciofalo et al. (US 20170272476).
As per claims 3, 13, Kreis teaches at para. 35: the video diffusion model can be explicitly instructed on the frames used for conditioning and the frames to be generated using relative time step embedding (e.g., time in the video). Ghoshal, Fayyaz and Kreis do not explicitly teach said claims.
Ciofalo et al. teaches
wherein the frame data, the audio data, or the text data is associated with one or more time markers, each time marker representing a respective time stamp of when data occurred during the video clip, and wherein the response includes at least one time marker for a time stamp of the video clip that is relevant to the query (para. 33: it is possible that the linked video is 10 minutes long and contains content relating to multiple subjects, but that only ten seconds of the video contains content relating to the user's input. In such a case, the link may define a starting point and ending point within the video file (e.g., a video clip starting at the 2:31 mark and ending at the 2:41 mark), such that only the portion of the video responsive to the user's input is retrieved; para. 71: the method of FIG. 4 utilizes a vast array of online content to provide relevant responses to a user's queries from a person of interest to whom the queries are directed. Furthermore, the responses are generally presented to the user using video and/or audio featuring the person of interest, thereby simulating the look and feel of an interactive conversation with the person of interest).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal, Fayyaz, Kreis to include one or time markers representing a respective time stamp of when data occurred during the video clip that is relevant to the query of Ciofalo in order to effectively generate outputs and provide relevant clip responses to the users.
Claim(s) 4, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghoshal et al. (US 2020125575) in view of Fayyaz et al. (US 20240354317) and further in view of Kreis et al. (US 20240171788) and Pea et al. (US 2004012521).
As per claims 4, 14, Ghoshal, Fayyaz and Kreis do not explicitly teach said claims.
Pea et al. teaches
wherein the frame data, the audio data, or the text data is associated with one or more spatial markers (para. 5: interactive authoring, sharing and analysis of digital video content; para. 12: the abstract map may, for example, represent an outline of a scene from the visual data, automatically generated using an edge detection algorithm, in which case the markers might logically be plotted based upon the spatial location of the imagery captured in each corresponding traversal record),
each spatial marker representing a respective spatial region of data occurring during the video clip, and wherein the response includes at least one spatial marker for a spatial region of the video clip that is relevant to the query (para. 12: selecting a spatial area within the abstract map (such as by means of a slider bar), and in response displaying within the worksheet a filtered list of traversal records corresponding to those markers located within the spatial area of interest; para. 145: for the scenarios just described, the markers would comprise traversal thumbnail images 730 and 930 (FIGS. 7 and 9) or classification code dots 820 (FIG. 8). At 620, the markers are plotted in appropriate locations along the abstract map. At 630 the user interactively selects the marker of interest and at 640 plays back the corresponding traversal record, thereby linking and shifting between different levels of abstraction in order to explore the video record, as described in FIG. 10).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal, Fayyaz, Kreis to include at least one spatial marker for a spatial region of the video clip that is relevant to the query of Pea et al. in order to effectively generate outputs and provide relevant clip responses to the users.
Claim(s) 5-7, 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghoshal et al. (US 2020125575) in view of Fayyaz et al. (US 20240354317) and further in view of Kreis et al. (US 20240171788) and Amer et al. (US 10789755).
As per claims 5, 15, Fayyaz teaches at para. 42: the filtering system uses a language model that operates autoregressively. As input information, the language model receives the query embedding, the target item embeddings in the candidate set, and prompt information. The prompt information provides a text-based narrative that instructs the filtering system to find the most relevant target item embeddings and/or
to find the least relevant target item embeddings.
Ghoshal and Fayyaz do not explicitly teach wherein the decoder further includes a machine-learned alignment model.
Kreis teaches
wherein the decoder further includes a machine-learned alignment model (fig. 1: training system, video diffusion model: encoder, decoder; para. 4-6: align a plurality of images into frames of a first video using at least one first temporal attention layer of a neural network model; fig. 2: update neural network model to align images into frames of first video; para. 17: updating a neural network model to align a plurality of images into frames of a first video by updating at least one first temporal attention layer of the neural network model).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal, Fayyaz to include a machine-learned alignment model of Kreis et al. in order to effectively generate video outputs by aligning input images into frames of a video using e.g., relative time step embeddings.
Even if Ghoshal, Fayyaz and Kreis do not explicitly teach a machine-learned language model,
Amer teaches
a machine-learned language model (col. 6:15-27: training data includes data derived from actual video clips from movies (e.g., well-known or popular movies), which may be a particularly rich and convenient source for training data associated with human interactions. Training data includes human centric annotations, actors or characters appearing in the scenes or event sequences, objects, spatial and other relationships between actors and objects, interactions between characters, timestamps at which actions initiate and conclude, a description of the situation depicted in the scene, subtitles, text of the scene's narrative content, a summary or natural language description of the scene's narrative content, and other information; col. 12:10-13: a language model for action description encoding may be a 2-layer LSTM with 256 hidden units, and the word embeddings are learned from scratch; col. 16:1-9; col. 20:6-9: SLING is trained in an end to end fashion using bidirectional LSTMs for input encoding and a Transition Based Recurrent Unit (TRBU) for output decoding.)
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal, Fayyaz, Kreis to include a machine-learned language model of Amer et al. in order to effectively generate video outputs by using parameters identified during training to thereafter process new input data received from language parser and/or interaction module.
As per claims 6, 16, Ghoshal teaches at para. 79: tagging of text blocks, images, audio/video data, and/or other content with tags corresponding to any identified
content topics, categories, or features.
Ghoshal, Fayyaz do not explicitly teach a machine-learned alignment model.
Kreis et al. teaches
applying the machine-learned alignment model to the set of video embeddings to generate a set of video-language-aligned embeddings; wherein the set of input embeddings are the set of video-language-aligned embeddings; wherein applying the component of the machine-learned decoder comprises applying the machine-learned language model to the input tensor to generate the set of output embeddings (fig. 1: training system, video diffusion model: encoder, decoder; para. 4-6: align a plurality of images into frames of a first video using at least one first temporal attention layer of a neural network model; para. 17: updating a neural network model to align a plurality of images into frames of a first video by updating at least one first temporal attention layer of the neural network model; para. 31: an LDM can include an encoder (e. g., an encoder neural network) and a decoder (e.g., a decoder neural network). An encoder of the LDM can compress and can maps the input (e.g., an image) from an image or pixel space to a lower dimensional, compressed latent space (e.g., latent tensors, latent representations, and so on) where the diffusion model can be trained more efficiently; para. 71: the latent representation 504 can be applied in the generative denoising process in which latent embedding distribution modeled with diffusion model is conditioned on at least one of text prompts, bounding boxes, or conditioning signals).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal, Fayyaz to include a machine-learned alignment model of Kreis et al. in order to effectively generate video outputs by performing collaborative content creation for 3D assets, a system for text-to-video modeling etc.
Even if Ghoshal, Fayyaz, Kreis do not explicitly teach the machine-learned language model,
Amer teaches
the machine-learned language model (col. 6:15-27: training data includes data derived from actual video clips from movies (e.g., well-known or popular movies), which may be a particularly rich and convenient source for training data associated with human interactions. Training data includes human centric annotations, actors or characters appearing in the scenes or event sequences, objects, spatial and other relationships between actors and objects, interactions between characters, timestamps at which actions initiate and conclude, a description of the situation depicted in the scene, subtitles, text of the scene's narrative content, a summary or natural language description of the scene's narrative content, and other information; col. 12:10-13: a language model for action description encoding may be a 2-layer LSTM with 256 hidden units, and the word embeddings are learned from scratch; col. 16:1-9; col. 20:6-9: SLING is trained in an end to end fashion using bidirectional LSTMs for input encoding and a Transition Based Recurrent Unit (TRBU) for output decoding).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal, Fayyaz, Kreis to include a machine-learned language model of Amer et al. in order to effectively generate video outputs by using parameters identified during training to thereafter process new input data received from language parser and/or interaction module.
As per claims 7, 17, Ghoshal et al. teaches
wherein the set of input embeddings are the set of video embeddings (fig. 4: content input processing/analysis and content retrieval/embedding; para. 195: each machine learning model or model type, the trained models may be executed by one or more computing systems during which a content item is provided as input to the one or more models and the output from the models may identify the one or more tags to be associated with the content item, or the output of the models may be used to identify the one or more tags to be associated with the content item).
Ghoshal, Fayyaz do not explicitly teach wherein applying the component of the machine-learned decoder comprises applying the machine-learned alignment model to the input tensor.
Kreis teaches
wherein applying the component of the machine-learned decoder comprises applying the machine-learned alignment model to the input tensor to generate a set of video-language-aligned embeddings, the further comprising: applying the machine-learned language model the set of video-language-aligned embeddings to generate the set of output embeddings (fig. 1: training system, video diffusion model: encoder, decoder; para. 17: updating a neural network model to align a plurality of images into frames of a first video by updating at least one first temporal attention
layer of the neural network model; para. 31: an LDM can include an encoder (e. g., an encoder neural network) and a decoder (e.g., a decoder neural network). An encoder of the LDM can compress and can maps the input (e.g., an image) from an image or pixel space to a lower dimensional, compressed latent space (e.g., latent tensors, latent representations, and so on) where the diffusion model can be trained more efficiently; para. 35: the video diffusion model can be explicitly instructed on the frames used for
conditioning and the frames to be generated using relative time step embedding (e.g., time in the video)).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal, Fayyaz to include the applying the machine-learned alignment model to the input tensor to generate a set of video-language-aligned embeddings of Kreis in order to effectively generate video outputs by aligning input images into frames of a video using e.g., relative time step embeddings.
Claim(s) 8-9, 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghoshal et al. (US 2020125575) in view of Fayyaz et al. (US 20240354317) and further in view of Kreis et al. (US 20240171788), Amer et al. (US 10789755) and Mondal et al. (US 20230153352).
As per claims 8, 18, Amer teaches at col. 19:1-8: FIG. 1D, training sample includes visual information and textual information. Visual information may correspond to a video clip from a movie or other video, and textual information may correspond to a script, a description, or other textual representation of information associated with a story, an event, a scene, or a sequence of events or scenes; col. 31:1-19: data store.
Ghoshal, Fayyaz, Kreis, Amer do not teach said claims.
Mondal teaches
wherein the query is to generate a description of the video clip in text, and wherein the response is a clip description of the video clip, the method further comprising: for each video clip in the one or more video clips, generating clip descriptions for the video clip; and storing the clip descriptions for the one or more video clips in a datastore (para. 46: videos stored in the video database may or may not be annotated with multi-sentence text descriptions. The CBVR system may communicate with the video database to retrieve one or more videos that are relevant to a received word-based query, and provide the retrieved video(s) as part of a response to the query; para. 60: Each video has a multi-sentence text annotation, which may be referred to as a paragraph description. Each sentence in the paragraph description corresponds to a respective clip in the video and is associated with a respective timestamp of each clip ; para. 67: the partitioning into sentences may be performed by a preprocessing module that is external to the video representation generator 110. It should be noted that the index of each sentence corresponds to the index of the corresponding clip. That is, sentence-k corresponds to (i.e., is a text description of) clip-k; para. 80: each video in the training dataset is annotated with a ground-truth annotation including a multi-sentence text description of the video (where each sentence corresponds to a clip of the video) and clip timestamps corresponding to each sentence of the annotation. The generated outputs include generated video representation and a set of high-level clip representations; para. 100).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal, Fayyaz, Kreis, Amer to include storing the clip descriptions for the one or more video clips in a datastore of Mondal et al. in order to effectively allow searches and provide relevant clip responses to the users.
As per claims 9, 19, Ghoshal et al. does not teach said claims
Kreis teaches
applying the machine-learned language model or a second machine-learned language model to the clip descriptions for the one or more video clips to generate video-level information for the video (para. 28: using one or more neural networks or machine learning models (alternatively referred to herein as "models") to generate or synthesize high-resolution, long-term, AI-based videos; para. 37: the video diffusion model can be updated (e.g., trained) to be conditional on global class-like information, e.g., day, night, and so on);
applying at least a portion of the machine-learned decoder to a second set of video embeddings to update the clip description of the video clip, wherein applying at least the portion of the machine-learned decoder comprises including the video-level information in a set of inputs to the machine-learned alignment model or a set of inputs to the machine-learned language model (para. 4: align a plurality of images into frames of a first video using at least one first temporal attention layer of a neural network model; para. 47: the training system 100 can construct or otherwise arrive at the video diffusion model 110 based on the image diffusion model 102. The video
diffusion model 110 can include one or more neural networks. A neural network can include an input layer, an output layer, and/or one or more intermediate layers, such as
hidden layers, which can each have respective nodes. The video diffusion model 110 can include various neural network models, including models that are effective for operating on images and aligning images into a video. The image diffusion model 102 includes the encoder 104, the decoder etc.)
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ghoshal to include the machine-learned alignment model of Kreis in order to effectively process inputs, generate outputs and provide responses to the users.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Zhu et al. (US 12367240) teaches at col. 9:35-67: video data, audio data, and text data may be received, video data and text data may be received, audio data and text data may be received, and/or any other combination of different types of data, including data other than video, audio, and/or text data as well. That is, as aforementioned, any reference to aspects of an AVT may also be generally applicable to a multimodal transformer involving data combinations other than video and audio. The specific types of encoders that are employed may depend on the types of data being received. For example, at a high level, a video encoder (such as video encoder) may break down a frame of a series of frames included in video data into video embeddings.
Bakunov et al. (US 20240296535) teaches at para. 173: machine learning algorithms or tools, are used as part of the systems described herein to perform operations associated with searches and query responses; para. 109: Visual Question Answering (VQA) model is an AI tool that is trained to answer natural language questions about visual content, such as images or videos.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LINH BLACK whose telephone number is (571)272-4106. The examiner can normally be reached 9AM-5PM EST M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tony Mahmoudi can be reached on 571-272-4078. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LINH BLACK/Examiner, Art Unit 2163 6/9/2026
/ALEX GOFMAN/Primary Examiner, Art Unit 2163
Prosecution Insights