DETAILED ACTION
Response to Applicant Interview
Examiner notes that this Office Action replaces/supersedes the prior Office Action.
Allowable Subject Matter
Claims 8, 9, 16, and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance:
Claim 8 is allowable over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest the method of claim 1, further comprising: generating, based at least on the one or more poses, the one or more images to represent the one or more shapes from the perspective of a canonical viewpoint, as presented in the environment of the remaining limitations of claim 8. It is noted that the closest prior art, Menashof, shows the method of claim 1. However, Menashof fails to disclose or suggest receiving third data representative of one or more poses associated with the one or more shapes; and generating, based at least on the one or more poses, the one or more images to represent the one or more shapes from the perspective of a canonical viewpoint.
Claim 16 is allowable over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest the system of claim 10, further comprising: generate, based at least on the one or more poses, the one or more images to represent the one or more shapes from the perspective of a canonical viewpoint, as presented in the environment of the remaining limitations of claim 16. It is noted that the closest prior art, Menashof, shows the system of claim 10. However, Menashof fails to disclose or suggest receive third data representative of one or more poses associated with the one or more shapes; and generate, based at least on the one or more poses, the one or more images to represent the one or more shapes from the perspective of a canonical viewpoint.
The remaining claims depend from one of the above claims, either directly or indirectly, and are accordingly allowable.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 10-15, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Menashof et al. (US Pub. 2025/0148753), hereinafter Menashof, in view of Greasley (US Pub. 2021/0397842).
Regarding claim 1, Menashof discloses a method comprising: generating, based at least on one or more language models processing first data associated with one or more images depicting one or more shapes (Paragraph [0005]: the encoder includes a large language model (LLM) or other natural language model to generate descriptors of the pivot images. For example, the descriptors can include natural language descriptions of the objects, backgrounds, interactions, and concepts included in the pivot images): one or more first captions describing the one or more shapes (Paragraph [0024]: the encoder extracts every tenth frame of the video and causes a first generative machine model to generate a descriptor for the extracted frames (e.g., a natural language description of the soccer ball and the location). In this example, the extracted frames are selected based on an interval of time (e.g., every tenth frame), although, as described in greater detail below, other algorithms can be used for selecting frames of the video to be extracted); and one or more second captions describing the one or more shapes (Paragraph [0071]: the system implementing the method 300 generates descriptors based at least in part on the pivot images and/or video. In one example, an LLM or other machine learning model generates descriptors based at least in part on the pivot images. The descriptors, in an embodiment, include natural language descriptions of the pivot images. In other embodiments, the descriptors include structured data (e.g., a JavaScript Object Notation [JSON] file) that indicates attributes (e.g., location, movement, size, shape, etc.) of objects within the pivot images. For example, the descriptors include text-based data (e.g., data, pseudo-code, source code, etc.) that is provided to a generative model of an encoder to enable the generative model to reconstruct the video. In an embodiment, descriptors are generated for the video and/or portions thereof); and generating second data that associates the one or more shapes with the one or more first captions and the one or more second captions (Paragraph [0102]: each word or character in the input(s) 601 is mapped into the input embedding 602 in parallel or at the same time, unlike existing long short-term memory (LSTM) models, for example. The input embedding 602 maps a word to a feature vector representing the word. But the same word (for example, “apple”) in different sentences may have different meanings (for example, phone versus fruit). This is why a positional encoder 604 can be implemented. A positional encoder 604 is a vector that gives context to words (for example, “apple”) based on a position of a word in a sentence. For example, with respect to a message “I just sent the document,” because “I” is at the beginning of a sentence, embodiments can indicate a position in an embedding closer to “just,” as opposed to “document.” Some embodiments use a sine/cosine function to generate the positional encoder vector using the following two example equations; Paragraph [0110]: the initial embedding (for example, the input embedding 602) is constructed from three vectors: the token embeddings, the segment or context-question embeddings, and the position embeddings. In some embodiments, the following functionality occurs in the pre-training phase. The token embeddings are the pre-trained embeddings. The segment embeddings are the sentence numbers (that includes the input [s] 601) that is encoded into a vector (for example, first sentence, second sentence, and so forth, assuming a top-down and right-to-left approach). The position embeddings are vectors that represent the position of a particular word in such a sentence that can be produced by positional encoder 604. When these three embeddings are added or concatenated together, an embedding vector is generated that is used as input into the encoder/decoder block(s) 606. The segment and position embeddings are used for temporal ordering since all of the vectors are fed into the encoder/decoder block(s) 606 simultaneously, and language models need some sort of order preserved).
Menashof does not explicitly disclose the one or more first captions associated with one or more first numbers of words that are less than a threshold number of words; or the one or more second captions being associated with one or more second numbers of words that are equal to or greater than the threshold number of words.
However, Greasley teaches generating a text description for images (Paragraphs [0038]-[0039]), further comprising: the one or more first captions associated with one or more first numbers of words that are less than a threshold number of words (Paragraph [0038]: if the user 106 indicates a preference for a short listening experience, the electronic device 102 and/or the controller 104 may generate a terse text description, e.g., with a low word count, for example, less than a threshold number of words. On the other hand, if the user 106 indicates a preference for a longer listening experience, the electronic device 102 and/or the controller 104 may generate a more verbose text description, e.g., with a high word count, for example, greater than a threshold number of words); and the one or more second captions being associated with one or more second numbers of words that are equal to or greater than the threshold number of words (Paragraph [0039]: if the user 106 is moving at a low velocity or is stationary, the electronic device 102 and/or the controller 104 may generate a text description with a high word count (e.g., greater than a threshold number of words). If the user 106 is moving at a high velocity, the electronic device 102 and/or the controller 104 may generate a text description with a low word count (e.g., less than a threshold number of words) or may increase the speed (e.g., words per minute) of the narration to keep pace with the movement of the user 106; Paragraph [0053]: length of the text description may be determined based on the velocity of the user. For example, if the user is stationary or moving at a low velocity, the audio output generator 224 may generate a verbose text description with a high word count (e.g., greater than a threshold number of words). If the user is moving at a high velocity, the audio output generator 224 may generate a terse text description with a low word count (e.g., less than a threshold number of words). In some implementations, the audio output generator 224 selects a depth to which the ontology 226 is traversed). Greasley teaches that this will allow for user preference of descriptions to be realized (Paragraph [0038]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Menashof with the features of above as taught by Greasley so as to allow for user preference of descriptions to be realized as presented by Greasley.
Regarding claim 2, Menashof, in view of Greasley teaches the method of claim 1, Menashof discloses further comprising: generating one or more first embeddings associated with the one or more first captions and one or more second embeddings associated with the one or more second captions, wherein the second data associates the one or more shapes with the one or more first embeddings and the one or more second embeddings (Paragraph [0102]: each word or character in the input(s) 601 is mapped into the input embedding 602 in parallel or at the same time, unlike existing long short-term memory (LSTM) models, for example. The input embedding 602 maps a word to a feature vector representing the word. But the same word (for example, “apple”) in different sentences may have different meanings (for example, phone versus fruit). This is why a positional encoder 604 can be implemented. A positional encoder 604 is a vector that gives context to words (for example, “apple”) based on a position of a word in a sentence. For example, with respect to a message “I just sent the document,” because “I” is at the beginning of a sentence, embodiments can indicate a position in an embedding closer to “just,” as opposed to “document.” Some embodiments use a sine/cosine function to generate the positional encoder vector using the following two example equations; Paragraph [0110]: the initial embedding (for example, the input embedding 602) is constructed from three vectors: the token embeddings, the segment or context-question embeddings, and the position embeddings. In some embodiments, the following functionality occurs in the pre-training phase. The token embeddings are the pre-trained embeddings. The segment embeddings are the sentence numbers (that includes the input [s] 601) that is encoded into a vector (for example, first sentence, second sentence, and so forth, assuming a top-down and right-to-left approach). The position embeddings are vectors that represent the position of a particular word in such a sentence that can be produced by positional encoder 604. When these three embeddings are added or concatenated together, an embedding vector is generated that is used as input into the encoder/decoder block(s) 606. The segment and position embeddings are used for temporal ordering since all of the vectors are fed into the encoder/decoder block(s) 606 simultaneously, and language models need some sort of order preserved).
Regarding claim 3, Menashof, in view of Greasley teaches the method of claim 1, Menashof discloses further comprising: generating one or more embeddings associated with one or more combinations of the one or more first captions and the one or more second captions, wherein the second data associates the one or more shapes with the one or more embeddings (Paragraphs [0102]-[0103]: each word or character in the input(s) 601 is mapped into the input embedding 602 in parallel or at the same time, unlike existing long short-term memory (LSTM) models, for example. The input embedding 602 maps a word to a feature vector representing the word. But the same word (for example, “apple”) in different sentences may have different meanings (for example, phone versus fruit). This is why a positional encoder 604 can be implemented. A positional encoder 604 is a vector that gives context to words (for example, “apple”) based on a position of a word in a sentence. For example, with respect to a message “I just sent the document,” because “I” is at the beginning of a sentence, embodiments can indicate a position in an embedding closer to “just,” as opposed to “document.” Some embodiments use a sine/cosine function to generate the positional encoder vector using the following two example equations… passing the input(s) 601 through the input embedding 602 and applying the positional encoder 604, the output is a word embedding feature vector, which encodes positional information or context based on the positional encoder 604. These word embedding feature vectors are then passed to the encoder and/or decoder block(s) 606, where it goes through a multi-head attention layer 606-1 and a feedforward layer 606-2. The multi-head attention layer 606-1 is generally responsible for focusing or processing certain parts of the feature vectors representing specific portions of the input(s) 601 by generating attention vectors; Paragraph [0110]: the initial embedding (for example, the input embedding 602) is constructed from three vectors: the token embeddings, the segment or context-question embeddings, and the position embeddings. In some embodiments, the following functionality occurs in the pre-training phase. The token embeddings are the pre-trained embeddings. The segment embeddings are the sentence numbers (that includes the input [s] 601) that is encoded into a vector (for example, first sentence, second sentence, and so forth, assuming a top-down and right-to-left approach). The position embeddings are vectors that represent the position of a particular word in such a sentence that can be produced by positional encoder 604. When these three embeddings are added or concatenated together, an embedding vector is generated that is used as input into the encoder/decoder block(s) 606. The segment and position embeddings are used for temporal ordering since all of the vectors are fed into the encoder/decoder block(s) 606 simultaneously, and language models need some sort of order preserved).
Regarding claim 4, Menashof, in view of Greasley teaches the method of claim 1, Menashof discloses wherein the generating the one or more first captions and the one or more second captions is further based at least on the one or more machine learning models processing third data representative of one or more descriptions associated with the one or more shapes (Paragraphs [0086]-[0087]: the operations further comprise causing, at the decoder, a third machine learning model to generate a reconstructed video by at least providing as a first input to the third machine learning model the pivot image and the descriptor, where the third machine learning model uses the pivot image and at least a portion of the descriptor to output a second plurality of images that are combined to generate the reconstructed video…the first machine learning model comprises a large language model, the second machine learning model comprises a neural network, and the third machine learning model comprises a diffusion model).
Regarding claim 5, Menashof, in view of Greasley teaches the method of claim 4, Menashof discloses wherein the one or more descriptions may include at least one of: one or more categories associated with the one or more shapes; or one or more tags indicating one or more characteristics associated with the one or more shapes (Fig. 4; Paragraph [0075]: FIG. 4 is a flow diagram showing a method 400 for generating pivot images and descriptors for compressed video data, in accordance with at least one embodiment. The method 400 can be performed, for instance, by the encoder 124 of the video compression tool 104 of FIG. 1. As shown at block 402, the system implementing the method 300 obtains a video frame. In various embodiments, a video is obtained, and individual frames of the video (e.g., video frames) are extracted and/or processed. At block 404, the system implementing the method 400 determines and/or detects objects in the video frame. For example, as described above, an object detection model takes the video frame as an input and outputs data associated with objects in the video frame (e.g., labels, confidence intervals, tags, names, type information etc.)).
Regarding claim 6, Menashof, in view of Greasley teaches the method of claim 1, Menashof discloses wherein the generating the one or more first captions and the one or more second captions is further based at least on the one or more machine learning models processing third data representative of an output format, the output format associated with generating both the one or more first captions and the one or more second captions for the one or more shapes (Paragraph [0020]: compression techniques rely on eliminating or reducing redundant data and are limited by various constraints including video and audio fidelity. In addition, the focus on maintaining fidelity limits the effectiveness of compression techniques. In addition, video compression is necessary in order to share or otherwise transmit videos on the Internet because compression reduces the amount of data that is needed to stream or send the video to the viewer, and network bandwidth is a limited resource. One way to address this issue is by using a video coding format (e.g., a video compression format), which is a content representation format for storage or transmission of digital video content (e.g., in a data file or bitstream). Typically these formats use a video compression algorithm, such as discrete cosine transform (DCT) coding or motion compensation; Paragraph [0112]: after pre-training is performed, the encoder/decoder block(s) 606 performs prompt engineering or fine-tuning on a variety of QA data sets by converting different QA formats into a unified sequence-to-sequence format. For example, some embodiments perform the QA task by adding a new question-answering head or encoder/decoder block, just the way a masked language model head is added (in pre-training) for performing an MLM task, except that the task is a part of prompt engineering or fine-tuning. This includes the encoder/decoder block(s) 606 processing the inputs (e.g., pivot images 142 and/or descriptors 144 of FIG. 1) in order to make the predictions and generate a prompt response, as indicated in 604. Prompt engineering, in some embodiments, is the process of crafting and optimizing text prompts for language models to achieve desired outputs. In other words, prompt engineering comprises a process of mapping prompts (for example, a question) to the output (for example, an answer) that it belongs to for training. For example, if a user asks a model to generate a poem about a person fishing on a lake, the expectation is it will generate a different poem each time. Users may then label the output or answers from best to worst. Such labels are an input to the model to make sure the model is giving a more human-like or best answers, while trying to minimize the worst answers (for example, via reinforcement learning). In some embodiments, a “prompt” as described herein includes one or more of: a request (for example, a question or instruction [for example, “write a poem”]), target content, and one or more examples, as described herein).
Regarding claim 7, Menashof, in view of Greasley teaches the method of claim 1, Menashof discloses further comprising: generating, based at least on image data representative of the one or more images, one or more input tokens, wherein the first data represents the one or more input tokens (Paragraph [0060]: inference phase can be divided into two stages: a prompt stage and an auto-regressive stage. The prompt stage can include receiving and processing input as a batch of new tokens as part of the same inference. The prompt stage may operate based on a Key-Value (KV) cache technique, where a KV cache is created for tokens in a batch. During the prompt stage, the input is being digested. The auto-regressive state can include using the model to generate the tokens one-by-one, based on previous tokens, relying on reading the KV cache of previously-processed tokens, and adding the data of the new of only new tokens to the KV cache. This auto-regressive stage includes the model generating a response to the input from the prompt stage).
Regarding claim 10, Menashof discloses a system comprising: one or more processors (Fig. 7) to: generate, using one or more language models and based at least on first data associated with one or more images of one or more objects (Paragraph [0005]: the encoder includes a large language model (LLM) or other natural language model to generate descriptors of the pivot images. For example, the descriptors can include natural language descriptions of the objects, backgrounds, interactions, and concepts included in the pivot images): one or more first captions associated with the one or more shapes (Paragraph [0024]: the encoder extracts every tenth frame of the video and causes a first generative machine model to generate a descriptor for the extracted frames (e.g., a natural language description of the soccer ball and the location). In this example, the extracted frames are selected based on an interval of time (e.g., every tenth frame), although, as described in greater detail below, other algorithms can be used for selecting frames of the video to be extracted); and one or more second captions associated with the one or more shapes (Paragraph [0071]: the system implementing the method 300 generates descriptors based at least in part on the pivot images and/or video. In one example, an LLM or other machine learning model generates descriptors based at least in part on the pivot images. The descriptors, in an embodiment, include natural language descriptions of the pivot images. In other embodiments, the descriptors include structured data (e.g., a JavaScript Object Notation [JSON] file) that indicates attributes (e.g., location, movement, size, shape, etc.) of objects within the pivot images. For example, the descriptors include text-based data (e.g., data, pseudo-code, source code, etc.) that is provided to a generative model of an encoder to enable the generative model to reconstruct the video. In an embodiment, descriptors are generated for the video and/or portions thereof); and generate second data that associates the one or more images with the one or more first captions and the one or more second captions (Paragraph [0102]: each word or character in the input(s) 601 is mapped into the input embedding 602 in parallel or at the same time, unlike existing long short-term memory (LSTM) models, for example. The input embedding 602 maps a word to a feature vector representing the word. But the same word (for example, “apple”) in different sentences may have different meanings (for example, phone versus fruit). This is why a positional encoder 604 can be implemented. A positional encoder 604 is a vector that gives context to words (for example, “apple”) based on a position of a word in a sentence. For example, with respect to a message “I just sent the document,” because “I” is at the beginning of a sentence, embodiments can indicate a position in an embedding closer to “just,” as opposed to “document.” Some embodiments use a sine/cosine function to generate the positional encoder vector using the following two example equations; Paragraph [0110]: the initial embedding (for example, the input embedding 602) is constructed from three vectors: the token embeddings, the segment or context-question embeddings, and the position embeddings. In some embodiments, the following functionality occurs in the pre-training phase. The token embeddings are the pre-trained embeddings. The segment embeddings are the sentence numbers (that includes the input [s] 601) that is encoded into a vector (for example, first sentence, second sentence, and so forth, assuming a top-down and right-to-left approach). The position embeddings are vectors that represent the position of a particular word in such a sentence that can be produced by positional encoder 604. When these three embeddings are added or concatenated together, an embedding vector is generated that is used as input into the encoder/decoder block(s) 606. The segment and position embeddings are used for temporal ordering since all of the vectors are fed into the encoder/decoder block(s) 606 simultaneously, and language models need some sort of order preserved).
Menashof does not explicitly disclose the one or more first captions being associated with one or more first lengths; or the one or more second captions being associated with one or more second lengths that is different than the one or more first lengths.
However, Greasley teaches generating a text description for images (Paragraphs [0038]-[0039]), further comprising: the one or more first captions being associated with one or more first lengths (Paragraph [0038]: if the user 106 indicates a preference for a short listening experience, the electronic device 102 and/or the controller 104 may generate a terse text description, e.g., with a low word count, for example, less than a threshold number of words. On the other hand, if the user 106 indicates a preference for a longer listening experience, the electronic device 102 and/or the controller 104 may generate a more verbose text description, e.g., with a high word count, for example, greater than a threshold number of words); and the one or more second captions being associated with one or more second lengths that is different than the one or more first lengths (Paragraph [0039]: if the user 106 is moving at a low velocity or is stationary, the electronic device 102 and/or the controller 104 may generate a text description with a high word count (e.g., greater than a threshold number of words). If the user 106 is moving at a high velocity, the electronic device 102 and/or the controller 104 may generate a text description with a low word count (e.g., less than a threshold number of words) or may increase the speed (e.g., words per minute) of the narration to keep pace with the movement of the user 106; Paragraph [0053]: length of the text description may be determined based on the velocity of the user. For example, if the user is stationary or moving at a low velocity, the audio output generator 224 may generate a verbose text description with a high word count (e.g., greater than a threshold number of words). If the user is moving at a high velocity, the audio output generator 224 may generate a terse text description with a low word count (e.g., less than a threshold number of words). In some implementations, the audio output generator 224 selects a depth to which the ontology 226 is traversed). Greasley teaches that this will allow for user preference of descriptions to be realized (Paragraph [0038]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Menashof with the features of above as taught by Greasley so as to allow for user preference of descriptions to be realized as presented by Greasley.
Regarding claim 11, Menashof, in view of Greasley teaches the system of claim 10, Menashof discloses wherein the one or more processors are further to: generate one or more first embeddings associated with the one or more first captions and one or more second embeddings associated with the one or more second captions, wherein the second data associates the one or more shapes with the one or more first embeddings and the one or more second embeddings (Paragraph [0102]: each word or character in the input(s) 601 is mapped into the input embedding 602 in parallel or at the same time, unlike existing long short-term memory (LSTM) models, for example. The input embedding 602 maps a word to a feature vector representing the word. But the same word (for example, “apple”) in different sentences may have different meanings (for example, phone versus fruit). This is why a positional encoder 604 can be implemented. A positional encoder 604 is a vector that gives context to words (for example, “apple”) based on a position of a word in a sentence. For example, with respect to a message “I just sent the document,” because “I” is at the beginning of a sentence, embodiments can indicate a position in an embedding closer to “just,” as opposed to “document.” Some embodiments use a sine/cosine function to generate the positional encoder vector using the following two example equations; Paragraph [0110]: the initial embedding (for example, the input embedding 602) is constructed from three vectors: the token embeddings, the segment or context-question embeddings, and the position embeddings. In some embodiments, the following functionality occurs in the pre-training phase. The token embeddings are the pre-trained embeddings. The segment embeddings are the sentence numbers (that includes the input [s] 601) that is encoded into a vector (for example, first sentence, second sentence, and so forth, assuming a top-down and right-to-left approach). The position embeddings are vectors that represent the position of a particular word in such a sentence that can be produced by positional encoder 604. When these three embeddings are added or concatenated together, an embedding vector is generated that is used as input into the encoder/decoder block(s) 606. The segment and position embeddings are used for temporal ordering since all of the vectors are fed into the encoder/decoder block(s) 606 simultaneously, and language models need some sort of order preserved).
Regarding claim 12, Menashof, in view of Greasley teaches the system of claim 10, Menashof discloses wherein the one or more processors are further to: generate one or more embeddings associated with one or more combinations of the one or more first captions and the one or more second captions, wherein the second data associates the one or more shapes with the one or more embeddings (Paragraphs [0102]-[0103]: each word or character in the input(s) 601 is mapped into the input embedding 602 in parallel or at the same time, unlike existing long short-term memory (LSTM) models, for example. The input embedding 602 maps a word to a feature vector representing the word. But the same word (for example, “apple”) in different sentences may have different meanings (for example, phone versus fruit). This is why a positional encoder 604 can be implemented. A positional encoder 604 is a vector that gives context to words (for example, “apple”) based on a position of a word in a sentence. For example, with respect to a message “I just sent the document,” because “I” is at the beginning of a sentence, embodiments can indicate a position in an embedding closer to “just,” as opposed to “document.” Some embodiments use a sine/cosine function to generate the positional encoder vector using the following two example equations… passing the input(s) 601 through the input embedding 602 and applying the positional encoder 604, the output is a word embedding feature vector, which encodes positional information or context based on the positional encoder 604. These word embedding feature vectors are then passed to the encoder and/or decoder block(s) 606, where it goes through a multi-head attention layer 606-1 and a feedforward layer 606-2. The multi-head attention layer 606-1 is generally responsible for focusing or processing certain parts of the feature vectors representing specific portions of the input(s) 601 by generating attention vectors; Paragraph [0110]: the initial embedding (for example, the input embedding 602) is constructed from three vectors: the token embeddings, the segment or context-question embeddings, and the position embeddings. In some embodiments, the following functionality occurs in the pre-training phase. The token embeddings are the pre-trained embeddings. The segment embeddings are the sentence numbers (that includes the input [s] 601) that is encoded into a vector (for example, first sentence, second sentence, and so forth, assuming a top-down and right-to-left approach). The position embeddings are vectors that represent the position of a particular word in such a sentence that can be produced by positional encoder 604. When these three embeddings are added or concatenated together, an embedding vector is generated that is used as input into the encoder/decoder block(s) 606. The segment and position embeddings are used for temporal ordering since all of the vectors are fed into the encoder/decoder block(s) 606 simultaneously, and language models need some sort of order preserved).
Regarding claim 13, Menashof, in view of Greasley teaches the system of claim 10, Menashof discloses wherein the generation of the one or more first captions and the one or more second captions is further based at least on the one or more machine learning models processing third data representative of one or more descriptions associated with the one or more shapes (Paragraph [0071]: the system implementing the method 300 generates descriptors based at least in part on the pivot images and/or video. In one example, an LLM or other machine learning model generates descriptors based at least in part on the pivot images. The descriptors, in an embodiment, include natural language descriptions of the pivot images. In other embodiments, the descriptors include structured data (e.g., a JavaScript Object Notation [JSON] file) that indicates attributes (e.g., location, movement, size, shape, etc.) of objects within the pivot images. For example, the descriptors include text-based data (e.g., data, pseudo-code, source code, etc.) that is provided to a generative model of an encoder to enable the generative model to reconstruct the video. In an embodiment, descriptors are generated for the video and/or portions thereof; Paragraphs [0086]-[0087]: the operations further comprise causing, at the decoder, a third machine learning model to generate a reconstructed video by at least providing as a first input to the third machine learning model the pivot image and the descriptor, where the third machine learning model uses the pivot image and at least a portion of the descriptor to output a second plurality of images that are combined to generate the reconstructed video…the first machine learning model comprises a large language model, the second machine learning model comprises a neural network, and the third machine learning model comprises a diffusion model).
Regarding claim 14, Menashof, in view of Greasley teaches the system of claim 10, Menashof discloses wherein the generation of the one or more first captions and the one or more second captions is further based at least on the one or more machine learning models processing third data representative of an output format, the output format associated with generating both the one or more first captions and the one or more second captions for the one or more shapes (Paragraph [0020]: compression techniques rely on eliminating or reducing redundant data and are limited by various constraints including video and audio fidelity. In addition, the focus on maintaining fidelity limits the effectiveness of compression techniques. In addition, video compression is necessary in order to share or otherwise transmit videos on the Internet because compression reduces the amount of data that is needed to stream or send the video to the viewer, and network bandwidth is a limited resource. One way to address this issue is by using a video coding format (e.g., a video compression format), which is a content representation format for storage or transmission of digital video content (e.g., in a data file or bitstream). Typically these formats use a video compression algorithm, such as discrete cosine transform (DCT) coding or motion compensation; Paragraph [0112]: after pre-training is performed, the encoder/decoder block(s) 606 performs prompt engineering or fine-tuning on a variety of QA data sets by converting different QA formats into a unified sequence-to-sequence format. For example, some embodiments perform the QA task by adding a new question-answering head or encoder/decoder block, just the way a masked language model head is added (in pre-training) for performing an MLM task, except that the task is a part of prompt engineering or fine-tuning. This includes the encoder/decoder block(s) 606 processing the inputs (e.g., pivot images 142 and/or descriptors 144 of FIG. 1) in order to make the predictions and generate a prompt response, as indicated in 604. Prompt engineering, in some embodiments, is the process of crafting and optimizing text prompts for language models to achieve desired outputs. In other words, prompt engineering comprises a process of mapping prompts (for example, a question) to the output (for example, an answer) that it belongs to for training. For example, if a user asks a model to generate a poem about a person fishing on a lake, the expectation is it will generate a different poem each time. Users may then label the output or answers from best to worst. Such labels are an input to the model to make sure the model is giving a more human-like or best answers, while trying to minimize the worst answers (for example, via reinforcement learning). In some embodiments, a “prompt” as described herein includes one or more of: a request (for example, a question or instruction [for example, “write a poem”]), target content, and one or more examples, as described herein).
Regarding claim 15, Menashof, in view of Greasley teaches the system of claim 10, Greasley discloses wherein: the one or more first lengths are associated with one or more first numbers of words that are less than a threshold number of words (Paragraph [0038]: if the user 106 indicates a preference for a short listening experience, the electronic device 102 and/or the controller 104 may generate a terse text description, e.g., with a low word count, for example, less than a threshold number of words. On the other hand, if the user 106 indicates a preference for a longer listening experience, the electronic device 102 and/or the controller 104 may generate a more verbose text description, e.g., with a high word count, for example, greater than a threshold number of words); and the one or more second lengths are associated with one or more second numbers of words that are equal to or greater than the threshold number of words (Paragraph [0039]: if the user 106 is moving at a low velocity or is stationary, the electronic device 102 and/or the controller 104 may generate a text description with a high word count (e.g., greater than a threshold number of words). If the user 106 is moving at a high velocity, the electronic device 102 and/or the controller 104 may generate a text description with a low word count (e.g., less than a threshold number of words) or may increase the speed (e.g., words per minute) of the narration to keep pace with the movement of the user 106; Paragraph [0053]: length of the text description may be determined based on the velocity of the user. For example, if the user is stationary or moving at a low velocity, the audio output generator 224 may generate a verbose text description with a high word count (e.g., greater than a threshold number of words). If the user is moving at a high velocity, the audio output generator 224 may generate a terse text description with a low word count (e.g., less than a threshold number of words). In some implementations, the audio output generator 224 selects a depth to which the ontology 226 is traversed).
Regarding claim 18, Menashof, in view of Greasley teaches the system of claim 10, Menashof discloses wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs) (Paragraph [0005]: the encoder includes a large language model (LLM) or other natural language model to generate descriptors of the pivot images; a system for performing operations using one or more visual language models (VLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.
Regarding claim 19, Menashof discloses one or more processors (Fig. 7) comprising: processing circuitry to associate one or more shapes with one or more first captions and one or more second captions (Paragraph [0024]: the encoder extracts every tenth frame of the video and causes a first generative machine model to generate a descriptor for the extracted frames (e.g., a natural language description of the soccer ball and the location). In this example, the extracted frames are selected based on an interval of time (e.g., every tenth frame), although, as described in greater detail below, other algorithms can be used for selecting frames of the video to be extracted; Paragraph [0071]: the system implementing the method 300 generates descriptors based at least in part on the pivot images and/or video. In one example, an LLM or other machine learning model generates descriptors based at least in part on the pivot images. The descriptors, in an embodiment, include natural language descriptions of the pivot images. In other embodiments, the descriptors include structured data (e.g., a JavaScript Object Notation [JSON] file) that indicates attributes (e.g., location, movement, size, shape, etc.) of objects within the pivot images. For example, the descriptors include text-based data (e.g., data, pseudo-code, source code, etc.) that is provided to a generative model of an encoder to enable the generative model to reconstruct the video. In an embodiment, descriptors are generated for the video and/or portions thereof), wherein the one or more first captions and the one or more second captions are determined based at least on one or more language models processing data associated with one or more images depicting the one or more shapes (Paragraph [0005]: the encoder includes a large language model (LLM) or other natural language model to generate descriptors of the pivot images. For example, the descriptors can include natural language descriptions of the objects, backgrounds, interactions, and concepts included in the pivot images; Paragraph [0024]: the encoder extracts every tenth frame of the video and causes a first generative machine model to generate a descriptor for the extracted frames (e.g., a natural language description of the soccer ball and the location). In this example, the extracted frames are selected based on an interval of time (e.g., every tenth frame), although, as described in greater detail below, other algorithms can be used for selecting frames of the video to be extracted; Paragraph [0071]: the system implementing the method 300 generates descriptors based at least in part on the pivot images and/or video. In one example, an LLM or other machine learning model generates descriptors based at least in part on the pivot images. The descriptors, in an embodiment, include natural language descriptions of the pivot images. In other embodiments, the descriptors include structured data (e.g., a JavaScript Object Notation [JSON] file) that indicates attributes (e.g., location, movement, size, shape, etc.) of objects within the pivot images. For example, the descriptors include text-based data (e.g., data, pseudo-code, source code, etc.) that is provided to a generative model of an encoder to enable the generative model to reconstruct the video. In an embodiment, descriptors are generated for the video and/or portions thereof).
Menashof does not explicitly disclose one or more first captions that are associated with a first length and one or more second captions that are associated with a second length.
However, Greasley teaches generating a text description for images (Paragraphs [0038]-[0039]), further comprising: one or more first captions that are associated with a first length and one or more second captions that are associated with a second length (Paragraph [0038]: if the user 106 indicates a preference for a short listening experience, the electronic device 102 and/or the controller 104 may generate a terse text description, e.g., with a low word count, for example, less than a threshold number of words. On the other hand, if the user 106 indicates a preference for a longer listening experience, the electronic device 102 and/or the controller 104 may generate a more verbose text description, e.g., with a high word count, for example, greater than a threshold number of words; Paragraph [0039]: if the user 106 is moving at a low velocity or is stationary, the electronic device 102 and/or the controller 104 may generate a text description with a high word count (e.g., greater than a threshold number of words). If the user 106 is moving at a high velocity, the electronic device 102 and/or the controller 104 may generate a text description with a low word count (e.g., less than a threshold number of words) or may increase the speed (e.g., words per minute) of the narration to keep pace with the movement of the user 106; Paragraph [0053]: length of the text description may be determined based on the velocity of the user. For example, if the user is stationary or moving at a low velocity, the audio output generator 224 may generate a verbose text description with a high word count (e.g., greater than a threshold number of words). If the user is moving at a high velocity, the audio output generator 224 may generate a terse text description with a low word count (e.g., less than a threshold number of words). In some implementations, the audio output generator 224 selects a depth to which the ontology 226 is traversed). Greasley teaches that this will allow for user preference of descriptions to be realized (Paragraph [0038]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Menashof with the features of above as taught by Greasley so as to allow for user preference of descriptions to be realized as presented by Greasley.
Regarding claim 20, Menashof, in view of Greasley teaches the one or more processors of claim 19, Menashof discloses wherein the one or more processors are comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs) (Paragraph [0005]: the encoder includes a large language model (LLM) or other natural language model to generate descriptors of the pivot images); a system for performing operations using one or more visual language models (VLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW D SALVUCCI whose telephone number is (571)270-5748. The examiner can normally be reached M-F: 7:30-4:00PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW SALVUCCI/Primary Examiner, Art Unit 2613
Prosecution Insights