GENERATING HUMAN MOTION SEQUENCES UTILIZING UNSUPERVISED LEARNING OF DISCRETIZED FEATURES VIA A NEURAL NETWORK ENCODER-DECODER

Application No. 18/756,135

• Filed: Jun 27, 2024
• Priority: Feb 16, 2022 — continuation of 12/067,661
• Examiner: PEREN, VINCENT ROBERT
• Art Unit: 2617
• Tech Center: 2600 — Communications
• Assignee: Adobe Inc.
• OA Round: 1 (Non-Final)

Office Action

DETAILED ACTION
Status of Claims
Claims 1-20 are pending in this application, with claims 1, 10 and 17 being independent.
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Obligation Under 37 CFR 1.56 – Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Drawings
The drawings were received on June 27, 2024. These drawings are acceptable.

Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 17 is rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 9 and 18 of U.S. Patent No. 12,067,661. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of instant application are anticipated by and/or are obvious variants of the claims of U.S. Patent No. US 12,067,661.
Claims of the Instant Application

Claims of U.S. Patent No. 12,067,661

17. A non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause a computing device to perform operations comprising:

generating, utilizing a plurality of transformer neural network layers of
an encoder of a discretized motion model,
a sequence of latent feature representations of a human motion sequence in a continuous latent space from an unlabeled digital scene;

converting, utilizing a codebook of the discretized motion model, the sequence of latent feature representations into a sequence of discretized feature representations; and

generating, utilizing a decoder of the discretized motion model, digital content comprising a reconstructed human motion sequence based on the sequence of discretized feature representations.
18. A non-transitory computer readable storage medium storing instructions that, when executed by at least one processor, cause a computing device to perform operations comprising:

generating, utilizing an encoder of a discretized motion model,
a sequence of latent feature representations of a human motion sequence from an unlabeled digital scene;

determining, utilizing a distribution discretization layer of the discretized motion model, a plurality of sampling probabilities corresponding to a codebook of the discretized motion model based on the sequence of latent feature representations;

converting, utilizing the codebook of the discretized motion model, the sequence of latent feature representations into a sequence of discretized feature representations

by mapping the sequence of latent feature representations to a plurality of learned latent feature representations corresponding to human motions according to the plurality of sampling probabilities; and

generating, utilizing a decoder of the discretized motion model, digital content comprising a reconstructed human motion sequence based on the sequence of discretized feature representations.
17. A non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause a computing device to perform operations comprising:

generating, utilizing a plurality of transformer neural network layers of
an encoder of a discretized motion model,

a sequence of latent feature representations of a human motion sequence in a continuous latent space from an unlabeled digital scene;

converting, utilizing a codebook of the discretized motion model, the sequence of latent feature representations into a sequence of discretized feature representations;

and generating, utilizing a decoder of the discretized motion model, digital content comprising a reconstructed human motion sequence based on the sequence of discretized feature representations.
1. A computer-implemented method comprising:

generating, utilizing an encoder of a discretized motion model,

a sequence of latent feature representations of a human motion sequence from an unlabeled digital scene;

converting, utilizing a codebook of the discretized motion model, the sequence of latent feature representations into a sequence of discretized feature representations
by mapping the sequence of latent feature representations to a plurality of learned latent feature representations corresponding to human motions;

and generating, utilizing a decoder of the discretized motion model, digital content comprising a reconstructed human motion sequence based on the sequence of discretized feature representations.

9. The computer-implemented method as recited in claim 1, wherein
generating the sequence of latent feature representations comprises generating, utilizing a plurality of transformer neural network layers of the encoder, the sequence of latent feature representations in a continuous latent space.

Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.

The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:

Determining the scope and contents of the prior art;
Ascertaining the differences between the prior art and the claims at issue;
Resolving the level of ordinary skill in the pertinent art; and
Considering objective evidence present in the application indicating obviousness or nonobviousness.

Claims 1, 10 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over BHATTACHARYA et al. (US 2023/0135769, hereinafter “BHATTACHARYA”) in view of WU et al. (Wu, Chenfei, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. "Godiva: Generating open-domain videos from natural descriptions." arXiv preprint arXiv:2104.14806. 2021 Apr 30; hereinafter referred to as “WU”).
Regarding claim 1, BHATTACHARYA discloses a computer-implemented method (¶ [0019]: “The example neural network-based method can generate the emotive gestures at interactive rates on a commodity GPU.” ¶ [0063]: “method that takes in natural language text one sentence at a time and generates 3D pose sequences for virtual agents corresponding to emotive gestures aligned with that text. The example generative method also considers the intended acting task of narration or conversation, the intended emotion based on the text and the context, and the intended gender and handedness of the virtual agents to generate plausible gestures.” ¶ [0071]: “FIG. 10 is a flowchart illustrating an example method and technique for gesture generation, in accordance with various aspects of the techniques described in this disclosure. In some examples, the process 1000 may be carried out by the computing device 900 illustrated in FIG. 9, e.g., employing circuitry and/or software configured according to the block diagram illustrated in FIG. 9.”) comprising:
receiving an input text including one or more natural language phrases (¶ [0020]: “given a natural language text sentence associated with an acting task” ¶ [0021]: “the word at each position in the input sentence W” ¶ [0028]: “the input and target sequences.” ¶ [0029]: “the input sequence W.” ¶ [0072]: “At step 1002, process 1000 can receive a natural language input.” ¶ [0072]: “the natural language input can be a text sentence” ¶ [0064]: “receive a natural language input (e.g., text) 930.”) comprising an indication of a human motion sequence (e.g., ¶ [0019]: “emotive gestures (e.g., head gestures, hand gestures, full-body gestures, etc.) for virtual agents aligned with natural language inputs (e.g., text, speech, etc.).” ¶ [0020]: “the sequence of relative 3D joint rotations can correspond to a sequence of input words W.” ¶ [0028]: “the input and target sequences.”) (¶ [0019]: “emotive gestures (e.g., head gestures, hand gestures, full-body gestures, etc.) for virtual agents aligned with natural language inputs (e.g., text, speech, etc.). The example method generates emotionally expressive gestures (e.g., by utilizing the relevant biomechanical features for body expressions, also known as affective features). The example method can consider the intended task corresponding to the natural language input and the target virtual agents' intended gender and handedness in the generation pipeline.” ¶ [0020]: “In other words, a sequence of relative 3D joint rotations Q* underlying the poses of a virtual agent can be generated. Here, the sequence of relative 3D joint rotations can correspond to a sequence of input words W.” ¶ [0034]: “FIG. 2 shows the overall architecture 200 of an example transformer-based network. For example, the example network can take in sentences of natural language text 202 and transform the sentences to word embeddings 204 (e.g., using the pre-trained GloVe model). The example network can then use a transformer encoder 206 to transform the word embeddings 204 to latent representations 208, append the agent attributes 210 to these latent representations 208, and transform the combined representations into encoded features 212.” ¶ [0072]: “At step 1002, process 1000 can receive a natural language input. In some examples, the natural language input can include a sentence. In further examples, the natural language input can be a text sentence (e.g., an input using a keyboard, a touch screen, a microphone, or any suitable input device, etc.). However, it should be appreciated that the natural language input is not limited to a text sentence. It can be a sentence in a speech. In further examples, the natural language input can be multiple sentences.” ¶ [0073]: “At step 1004, process 1000 can receive a sequence of one or more word embeddings and one or more attributes. In some examples, process 1000 can convert the natural language input to the sequence of the one or more word embeddings using a embedding model. In further examples, process 1000 can obtain the one or more word embeddings based on the natural language input using the GloVe model pre-trained on the Common Crawl corpus. However, it should be appreciated that process 1000 can use any other suitable embedding model (e.g., Word2Vec, FastText, Bidirectional Encoder Representations from Transformers (BERT), etc.)” ¶ [0064]: “processes the input (e.g., the natural language input 930 and/or attributes using the gesture generation machine learning model to produce a predicted gesture 950 of a virtual agent aligned with the natural language input.”);
determining the human motion sequence (¶ [0022]: “a gesture can be represented as a sequence of poses or configurations of the 3D body joints. The sequence of poses or configurations can include body expressions as well as postures. In further examples, each pose can be represented with quaternions denoting 3D rotations of each joint relative to its parent in the directed pose graph as shown in FIG. 1.”) based on an intent (¶ [0019]: “the intended task corresponding to the natural language input” ¶ [0074]: “In some examples, the intended emotion indication and the acting task can depend on the natural language input.”) extracted from the input text utilizing a natural language processing model (¶ [0019]: “emotive gestures (e.g., head gestures, hand gestures, full-body gestures, etc.) for virtual agents aligned with natural language inputs (e.g., text, speech, etc.). The example method generates emotionally expressive gestures (e.g., by utilizing the relevant biomechanical features for body expressions, also known as affective features). The example method can consider the intended task corresponding to the natural language input and the target virtual agents' intended gender and handedness in the generation pipeline.” ¶ [0020]: “In other words, a sequence of relative 3D joint rotations Q* underlying the poses of a virtual agent can be generated. Here, the sequence of relative 3D joint rotations can correspond to a sequence of input words W.” ¶ [0021]: “word embeddings can be obtained via a suitable embedding model.” ¶ [0021]: “the word embeddings can be obtained using a GloVe model (e.g., pre-trained on the Common Crawl corpus). However, it should be appreciated that any other suitable embedding model (e.g., Word2Vec, FastText, BERT, etc.) can be used to obtain the word embeddings.” NOTE: One of ordinary skill in the art understands that BERT (Bidirectional Encoder Representations from Transformers) is natural language processing model based on an encoder only transformer architecture. ¶ [0034]: “FIG. 2 shows the overall architecture 200 of an example transformer-based network. For example, the example network can take in sentences of natural language text 202 and transform the sentences to word embeddings 204 (e.g., using the pre-trained GloVe model). The example network can then use a transformer encoder 206 to transform the word embeddings 204 to latent representations 208, append the agent attributes 210 to these latent representations 208, and transform the combined representations into encoded features 212.” ¶ [0073]: “At step 1004, process 1000 can receive a sequence of one or more word embeddings and one or more attributes. In some examples, process 1000 can convert the natural language input to the sequence of the one or more word embeddings using a embedding model. In further examples, process 1000 can obtain the one or more word embeddings based on the natural language input using the GloVe model pre-trained on the Common Crawl corpus. However, it should be appreciated that process 1000 can use any other suitable embedding model (e.g., Word2Vec, FastText, Bidirectional Encoder Representations from Transformers (BERT), etc.).” ¶ [0074]: “In some examples, process 1000 can receive one or more attributes. In some examples, the one or more attributes can include an intended emotion indication corresponding to the natural language input. In further examples, the intended emotion indication can include a categorical emotion term such as joy, anger, sadness, pride, etc. In some examples, a natural language input (e.g., a sentence) can be associated with one categorical emotion. However, it should be appreciated that a natural language input (e.g., a sentence) can be associated with multiple categorical emotions. In further examples, the intended emotion indication can include a set of values in a normalized valence-arousal-dominance (VAD) space. In some examples, a user can manually enter or select an indication indicative of the intended emotion indication corresponding to the natural language input (e.g., using a keyboard, a mouse, a touch screen, a voice command, etc.). In further examples, the user can change the intended emotion indication when a corresponding sentence can be mapped to a different intended emotion indication. For example, the user selects joy for a sentence. In further examples, the intended emotion indication can include one or more letters, one or more numbers, or any other suitable symbol. For example, the intended emotion indication can be ‘:)’ to indicate joy for a sentence. If next several sentences are mapped to the same intended emotion indication (i.e., joy), the user does not change the intended emotion indication until a different sentence is mapped to a different intended emotion indication (e.g., sadness). In other examples, process 1000 can recognize the natural language input and produce an indication indicative of the intended emotion indication (e.g., using a pre-trained machine learning model). In further examples, the one or more attributes can further include an acting task. For example, the acting task can include a narration indication and a conversation indication. In some examples, the intended emotion indication and the acting task can depend on the natural language input.” ¶ [0076]: “At step 1006, process 1000 can obtain a gesture generation machine learning model. In some examples, the gesture generation machine learning model can include a transformer network including an encoder and a decoder. However, it should be appreciated that the gesture generation machine learning model is not limited to a transformer network. For example, the gesture generation machine learning model can include a recurrent neural network (“RNN”), a long short-term memory (“LSTM”) model, a gated recurrent unit (“GRU”) model, a Markov process, a deep neural network (“DNN”), a convolutional neural network (“CNN”), a support vector machine (“SVM”), or any other suitable neural network model.”);
generating a sequence of latent feature representations (e.g., ¶ [0079]: “to produce the one or more latent representations.” ¶ [0081]: “produce the one or more latent representations.”) of the human motion sequence utilizing an encoder of a (e.g., ¶ [0028]: “Modeling the input text and output gestures as sequences can become a sequence transduction problem. This problem can be resolved by using a transformer-based network. The transformer network can include the encoder-decoder architecture for sequence-to-sequence modeling.” ¶ [0029]: “In the transformer encoder, the self-attention operates on the input sequence W.” ¶ [0031]: “The transformer encoder then can pass the MH output through two fully-connected (FC) layers.” ¶ [0031]: “The final encoded representation of the input sequence W can be denoted as FW.” ¶ [0032] To meet the given constraints on the acting task A, intended emotion E, gender G, and/or handedness H of the virtual agent, these variables can be appended to FW, and the combined representation can be passed through two fully-connected layers with trainable parameters WFC to obtain feature representations” ¶ [0034]: “FIG. 2 shows the overall architecture 200 of an example transformer-based network. For example, the example network can take in sentences of natural language text 202 and transform the sentences to word embeddings 204 (e.g., using the pre-trained GloVe model). The example network can then use a transformer encoder 206 to transform the word embeddings 204 to latent representations 208, append the agent attributes 210 to these latent representations 208, and transform the combined representations into encoded features 212.” ¶ [0035]: “In some examples, the word embedding layer can transform the words into feature vectors (e.g., using the pre-trained GloVe model).” ¶ [0077]: “At step 1008, process 1000 can provide the sequence of one or more word embeddings and the one or more attributes to the gesture generation machine learning model. In some examples, block 1010 is the gesture generation machine learning model. Steps 1012-1020 in the block 1010 are steps in the gesture generation machine learning model. Thus, process 1000 can perform steps 1010-1020 in block 1010 using the gesture generation machine learning model.” ¶ [0078]: “Steps 1012 and 1014 are performed in an encoder of the gesture generation machine learning model. At step 1012, process 1000 can receive, via an encoder of the gesture generation machine learning model, the sequence of the one or more word embeddings. In some examples process 1000 can signify a position of each word embedding in the sequence of one or more word embeddings (e.g., using a positional encoding scheme). In further examples, the position can be signified prior to using an encoder self-attention component in the encoder.” ¶ [0082]: “At step 1014, the gesture generation machine learning model can produce, via the encoder, an output based on the one or more word embeddings. In some examples, the encoder of the machine learning model can produce one or more latent representations based on the sequence of the one or more word embeddings.” ¶ [0083]: “At step 1016, the gesture generation machine learning model can generate one or more encoded features based on the output and the one or more attributes. In some examples, the gesture generation machine learning model can combine the one or more latent representations (FW) from the encoder with the one or more attributes (i.e., the acting task A, the intended emotion indication E, the gender indication G, and/or the handedness indication H). In further example, process 1000 can transforms the combined one or more latent representations into the one or more encoded features.”) (¶ [0020]: “given a natural language text sentence associated with an acting task of narration or conversation, an intended emotion, and attributes of the virtual agent, including gender and handedness, the virtual agent's corresponding gestures (e.g., body gestures) can be generated. In other words, a sequence of relative 3D joint rotations custom-character underlying the poses of a virtual agent can be generated. Here, the sequence of relative 3D joint rotations can correspond to a sequence of input words custom-character. In further examples, the sequence of relative 3D joint rotations can be subject to the acting task A and the intended emotion E based on the text, and the gender G and the handedness H of the virtual agent.” ¶ [0022]: “a gesture can be represented as a sequence of poses or configurations of the 3D body joints. The sequence of poses or configurations can include body expressions as well as postures. In further examples, each pose can be represented with quaternions denoting 3D rotations of each joint relative to its parent in the directed pose graph as shown in FIG. 1.” ¶ [0028]: “Modeling the input text and output gestures as sequences can become a sequence transduction problem. This problem can be resolved by using a transformer-based network. The transformer network can include the encoder-decoder architecture for sequence-to-sequence modeling. However, instead of using sequential chains of recurrent memory networks, or the computationally expensive convolutional networks, the example transformer uses a multi-head self-attention mechanism to model the dependencies between the elements at different temporal positions in the input and target sequences.”); and
generating, utilizing a decoder of the (e.g., ¶ [0033]: “The transformer decoder”), digital content comprising a reconstructed human motion sequence (¶ [0020]: “a sequence of relative 3D joint rotations Q* underlying the poses of a virtual agent can be generated. Here, the sequence of relative 3D joint rotations can correspond to a sequence of input words W.” ¶ [0033]: “the gesture sequence Q.” ¶ [0067]: “display a sequence of gestures of the virtual agent based on an output of the gesture generation machine learning model.”) based on learned mappings of the sequence of latent feature representations in a (¶ [0033]: “The transformer decoder can operate similarly using the target sequence Q,” ¶ [0034]: “The transformer decoder can take in these encoded features 212 and the past gesture history 214 to predict gestures 218 for the subsequent time steps using a transformer decoder 216. At each time step, the gesture can be represented by the set of rotations on all the body joints relative to their respective parents in the pose graph at that time step.” ¶ [0035]: “At the output of the decoder 216, the predicted values can be normalized so that the predicted values represent valid rotations. In some examples, the example network can be trained using the sum of three losses: the angle loss, the pose loss, and the affective loss. These losses can be computed between the gesture sequences generated by the example network and the original motion-captured sequences available as ground-truth in the training dataset.” ¶ [0064]: “FIG. 9 shows an example 900 of a system for gesture generation in accordance with some embodiments of the disclosed subject matter. As shown in FIG. 9, a computing device 910 can receive a natural language input (e.g., text) 930. In further examples, the computing device 910 can obtain a gesture generation machine learning model and/or attribute(s). The computing device 910 processes the input (e.g., the natural language input 930 and/or attributes using the gesture generation machine learning model to produce a predicted gesture 950 of a virtual agent aligned with the natural language input.” ¶ [0084]: “At step 1018, the gesture generation machine learning model can receive, via a decoder of the gesture generation machine learning model, the one or more encoded features and a first gesture of a virtual agent. In further examples, the first gesture can include a set of rotations on multiple body joints relative to one or more parent body joints.” ¶ [0085]: “In some examples, the decoder can generate the first emotive gesture at a preceding time step.” ¶ [0087]: “At step 1020, the gesture generation machine learning model can produce, via the decoder, a second emotive gesture based on the one or more encoded features and the first emotive gesture. In some examples, the fully connected layer of the decoder can produce the second emotive gesture. In further examples, the second gesture can include a set of rotations on multiple body joints relative to one or more parent body joints based on the first emotive gesture.” ¶ [0088]: “At step 1022, process 1000 can provide the second gesture of the virtual agent from the gesture generation machine learning model. In some examples, process 1000 can apply the set of rotations on multiple body joints to the virtual agent and display the movement of the virtual agent. In further examples, the second emotive gesture can include head movement of the virtual agent aligned with the natural language input. However, it should be appreciated that the second emotive gesture can include body movement, hand movement, and any other suitable movement.” ¶ [0091]: “At step 1122, process 1100 can train the gesture generation machine learning model based on the ground-truth gesture and the second emotive gesture. For example, the gesture generation machine learning model can be trained based on a loss function (L) summing an angle loss, a pose loss, and an affective loss. In some examples, a ground-truth gesture can include ground-truth relative rotation of a joint, and the second emotive gesture comprises a predicted relative rotation of the joint.”).
Whereas BHATTACHARYA is not entirely explicit as to, Wu teaches:
receiving an input text including one or more natural language phrases comprising an indication of a human motion sequence (pg. 1, 2nd paragraph of § 1: “text-to-video generation (T2V) task. Given a natural description, T2V requires the machine to understand it and create a semantically consistent video.” 1st paragraph of pg. 2: “generate open-domain videos from text using VQ-VAE and three-dimensional sparse attention.” pg. 4, 1st paragraph of § 3.2: “Given an input text t” pg. 6, 1st paragraph of § 4.2.1: “t denotes the input text.” pg. 6, 1st paragraph of § 4.4: “query "Play golf on grass", pg. 8, 1st paragraph of § 5: “generate open-domain videos from natural descriptions using VQ-VAE discrete visual tokens.”);
determining the human motion sequence based on an intent extracted from the input text utilizing a natural language processing model (pg. 1, 2nd paragraph of § 1: “text-to-video generation (T2V) task. Given a natural description, T2V requires the machine to understand it and create a semantically consistent video.” pg. 6, 1st paragraph of § 4.4: “query "Play golf on grass", pg. 7, 1st paragraph of § 4.4: “For example, GODIVA(64×64) first shows the grass field, then it gives the athlete a close-up shot, and finally the action of hitting the golf ball.” Figure 2: “Language Embedding” pg. 4, 1st paragraph of § 3.2: “the text embedding matrix,”);
generating a sequence of latent feature representations of the human motion sequence (pg. 2, 4th paragraph of § 1: “discrete video tokens.” pg. 4, Figure 2 caption: “the VQ-VAE discrete representation” pg. 4, 1st paragraph of § 3.2: “a sequence of discrete latent visual token embeddings” See the video representations of “humans” in Figure 3.) utilizing an encoder of a discretized motion model (e.g. pg. 4, Figure 2: “VQ-VAE Encoder” pg. 6, 1st paragraph of § 4.3: “the encoder E in Eq. (1)”) (pg. 1, 3rd paragraph of § 1: “a VQ-VAE pretrained model for the T2V task.” pg. 2, 4th paragraph of § 1: “a VQ-VAE auto-encoder is trained to represent continuous video pixels with discrete video tokens. Then, a three-dimensional sparse attention model is trained using language as input and the discrete video tokens as labels to generate videos,” pg. 4, Figure 2 caption: “the VQ-VAE discrete representation” pg. 4, 1st paragraph of § 3.2: “use the pretrained VQ-VAE Encoder (Eq. (1∼3)) to encode each frame the ground-truth videos in Eq. (7):” … “where the ground-truth video sequence x ∈ RL×H×W×C is encoded into a sequence of discrete latent visual token embeddings b ∈ RM×Db”. pg. 4, 1st paragraph of § 3.2: “pretrained VQ-VAE Encoder” ); and
generating, utilizing a decoder of the discretized motion model (pg. 4, Figure 2: “VQ-VAE Decoder”; pg. 4, 1st paragraph of § 3.2: “a decoder can be trained to generate videos” pg. 6, 1st paragraph of § 4.3: “the decoder D in Eq. (4)”), digital content comprising a reconstructed human motion sequence based on learned mappings of the sequence of latent feature representations in a discretized feature space (pg. 5, 1st paragraph of § 4.4.1: “v^ is the predicted video with L frames.” pg. 6, 1st paragraph of § 4.4: “T2V successfully generates the grass and action of "playing golf" in a resolution of 64×64,” pg. 7, 1st paragraph of § 4.4: “For example, GODIVA(64×64) first shows the grass field, then it gives the athlete a close-up shot, and finally the action of hitting the golf ball.” See Figure 3.).
Thus, in order to obtain a more computationally efficient system (as suggested by WU in the 2nd paragraph on pg. 2), it would have been obvious to one of ordinary skill in the art to have modified the system taught by BHATTACHARYA so as to use a trained encoder and decoder of a discretized motion model mapping sequences of latent feature representations in a discretized feature space, as taught by WU.

Regarding claim 10, BHATTACHARYA discloses a system (¶ [0064]: “FIG. 9 shows an example 900 of a system for gesture generation” ¶ [0064]: “computing device 910”) comprising:
one or more memory devices (e.g., FIG. 9: Memory 920” ¶ [0067]: “the computing device 910 can include a processor 912, a display 914, one or more inputs 916, one or more communication systems 918, and/or memory 920. In some embodiments, the processor 912 can be any suitable hardware processor or combination of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a microcontroller (MCU), etc.” ¶ [0069]: “In further examples, the memory 920 can include any suitable storage device or devices that can be used to store image data, instructions, values, machine learning models, etc., that can be used, for example, by the processor 912 to perform gesture generation or training the gesture generation machine learning model, to present a sequence of gestures 950 of the virtual agent using display 914, to receive the natural language input and/or attributes via communications system(s) 918 or input(s) 916, to transmit the sequence of gestures 950 of the virtual agent to any other suitable device(s) over the communication network 940, etc.”); and
one or more processors (¶ [0043]: “a GPU (e.g., Nvidia® GeForce® GTX1080Ti GPU).” FIG. 9: Processor 912”) coupled to the one or more memory devices that cause the system to perform operations (As shown in FIG. 9, processor 912 is coupled to memory 912. ¶ [0067]: “the computing device 910 can include a processor 912, a display 914, one or more inputs 916, one or more communication systems 918, and/or memory 920. In some embodiments, the processor 912 can be any suitable hardware processor or combination of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a microcontroller (MCU), etc.” ¶ [0069]: “the processor 912 can execute at least a portion of the computer program to perform one or more data processing and identification tasks described herein and/or to train/run the gesture generation machine learning model described herein, present the series of gestures 950 of the virtual agent to the display 914, transmit/receive information via the communications system(s) 918, etc.”) comprising:
receiving instructions (¶ [0020]: “given a natural language text sentence associated with an acting task” ¶ [0021]: “the word at each position in the input sentence W” ¶ [0028]: “the input and target sequences.” ¶ [0029]: “the input sequence W.” ¶ [0072]: “At step 1002, process 1000 can receive a natural language input.” ¶ [0072]: “the natural language input can be a text sentence” ¶ [0064]: “a computing device 910 can receive a natural language input (e.g., text) 930.” ¶ [0065]: “the computing device 910 can receive the natural language input 930,” ) comprising an indication of a human motion sequence (e.g., ¶ [0019]: “emotive gestures (e.g., head gestures, hand gestures, full-body gestures, etc.) for virtual agents aligned with natural language inputs (e.g., text, speech, etc.).” ¶ [0020]: “the sequence of relative 3D joint rotations can correspond to a sequence of input words W.” ¶ [0028]: “the input and target sequences.”) (¶ [0019]: “emotive gestures (e.g., head gestures, hand gestures, full-body gestures, etc.) for virtual agents aligned with natural language inputs (e.g., text, speech, etc.). The example method generates emotionally expressive gestures (e.g., by utilizing the relevant biomechanical features for body expressions, also known as affective features). The example method can consider the intended task corresponding to the natural language input and the target virtual agents' intended gender and handedness in the generation pipeline.” ¶ [0020]: “In other words, a sequence of relative 3D joint rotations Q* underlying the poses of a virtual agent can be generated. Here, the sequence of relative 3D joint rotations can correspond to a sequence of input words W.” ¶ [0034]: “FIG. 2 shows the overall architecture 200 of an example transformer-based network. For example, the example network can take in sentences of natural language text 202 and transform the sentences to word embeddings 204 (e.g., using the pre-trained GloVe model). The example network can then use a transformer encoder 206 to transform the word embeddings 204 to latent representations 208, append the agent attributes 210 to these latent representations 208, and transform the combined representations into encoded features 212.” ¶ [0072]: “At step 1002, process 1000 can receive a natural language input. In some examples, the natural language input can include a sentence. In further examples, the natural language input can be a text sentence (e.g., an input using a keyboard, a touch screen, a microphone, or any suitable input device, etc.). However, it should be appreciated that the natural language input is not limited to a text sentence. It can be a sentence in a speech. In further examples, the natural language input can be multiple sentences.” ¶ [0073]: “At step 1004, process 1000 can receive a sequence of one or more word embeddings and one or more attributes. In some examples, process 1000 can convert the natural language input to the sequence of the one or more word embeddings using a embedding model. In further examples, process 1000 can obtain the one or more word embeddings based on the natural language input using the GloVe model pre-trained on the Common Crawl corpus. However, it should be appreciated that process 1000 can use any other suitable embedding model (e.g., Word2Vec, FastText, Bidirectional Encoder Representations from Transformers (BERT), etc.)” ¶ [0074]: “In some examples, process 1000 can receive one or more attributes. In some examples, the one or more attributes can include an intended emotion indication corresponding to the natural language input. In further examples, the intended emotion indication can include a categorical emotion term such as joy, anger, sadness, pride, etc. In some examples, a natural language input (e.g., a sentence) can be associated with one categorical emotion. However, it should be appreciated that a natural language input (e.g., a sentence) can be associated with multiple categorical emotions. In further examples, the intended emotion indication can include a set of values in a normalized valence-arousal-dominance (VAD) space. In some examples, a user can manually enter or select an indication indicative of the intended emotion indication corresponding to the natural language input (e.g., using a keyboard, a mouse, a touch screen, a voice command, etc.). In further examples, the user can change the intended emotion indication when a corresponding sentence can be mapped to a different intended emotion indication. For example, the user selects joy for a sentence. In further examples, the intended emotion indication can include one or more letters, one or more numbers, or any other suitable symbol. For example, the intended emotion indication can be ‘:)’ to indicate joy for a sentence.” ¶ [0064]: “processes the input (e.g., the natural language input 930 and/or attributes using the gesture generation machine learning model to produce a predicted gesture 950 of a virtual agent aligned with the natural language input.”);
determining the human motion sequence (¶ [0022]: “a gesture can be represented as a sequence of poses or configurations of the 3D body joints. The sequence of poses or configurations can include body expressions as well as postures. In further examples, each pose can be represented with quaternions denoting 3D rotations of each joint relative to its parent in the directed pose graph as shown in FIG. 1.”) based on an intent (¶ [0019]: “the intended task corresponding to the natural language input” ¶ [0074]: “In some examples, the intended emotion indication and the acting task can depend on the natural language input.”) extracted from the instructions (¶ [0019]: “emotive gestures (e.g., head gestures, hand gestures, full-body gestures, etc.) for virtual agents aligned with natural language inputs (e.g., text, speech, etc.). The example method generates emotionally expressive gestures (e.g., by utilizing the relevant biomechanical features for body expressions, also known as affective features). The example method can consider the intended task corresponding to the natural language input and the target virtual agents' intended gender and handedness in the generation pipeline.” ¶ [0020]: “In other words, a sequence of relative 3D joint rotations Q* underlying the poses of a virtual agent can be generated. Here, the sequence of relative 3D joint rotations can correspond to a sequence of input words W.” ¶ [0021]: “word embeddings can be obtained via a suitable embedding model.” ¶ [0021]: “the word embeddings can be obtained using a GloVe model (e.g., pre-trained on the Common Crawl corpus). However, it should be appreciated that any other suitable embedding model (e.g., Word2Vec, FastText, BERT, etc.) can be used to obtain the word embeddings.” NOTE: One of ordinary skill in the art understands that BERT (Bidirectional Encoder Representations from Transformers) is natural language processing model based on an encoder only transformer architecture. ¶ [0034]: “FIG. 2 shows the overall architecture 200 of an example transformer-based network. For example, the example network can take in sentences of natural language text 202 and transform the sentences to word embeddings 204 (e.g., using the pre-trained GloVe model). The example network can then use a transformer encoder 206 to transform the word embeddings 204 to latent representations 208, append the agent attributes 210 to these latent representations 208, and transform the combined representations into encoded features 212.” ¶ [0073]: “At step 1004, process 1000 can receive a sequence of one or more word embeddings and one or more attributes. In some examples, process 1000 can convert the natural language input to the sequence of the one or more word embeddings using a embedding model. In further examples, process 1000 can obtain the one or more word embeddings based on the natural language input using the GloVe model pre-trained on the Common Crawl corpus. However, it should be appreciated that process 1000 can use any other suitable embedding model (e.g., Word2Vec, FastText, Bidirectional Encoder Representations from Transformers (BERT), etc.).” ¶ [0074]: “In some examples, process 1000 can receive one or more attributes. In some examples, the one or more attributes can include an intended emotion indication corresponding to the natural language input. In further examples, the intended emotion indication can include a categorical emotion term such as joy, anger, sadness, pride, etc. In some examples, a natural language input (e.g., a sentence) can be associated with one categorical emotion. However, it should be appreciated that a natural language input (e.g., a sentence) can be associated with multiple categorical emotions. In further examples, the intended emotion indication can include a set of values in a normalized valence-arousal-dominance (VAD) space. In some examples, a user can manually enter or select an indication indicative of the intended emotion indication corresponding to the natural language input (e.g., using a keyboard, a mouse, a touch screen, a voice command, etc.). In further examples, the user can change the intended emotion indication when a corresponding sentence can be mapped to a different intended emotion indication. For example, the user selects joy for a sentence. In further examples, the intended emotion indication can include one or more letters, one or more numbers, or any other suitable symbol. For example, the intended emotion indication can be ‘:)’ to indicate joy for a sentence. If next several sentences are mapped to the same intended emotion indication (i.e., joy), the user does not change the intended emotion indication until a different sentence is mapped to a different intended emotion indication (e.g., sadness). In other examples, process 1000 can recognize the natural language input and produce an indication indicative of the intended emotion indication (e.g., using a pre-trained machine learning model). In further examples, the one or more attributes can further include an acting task. For example, the acting task can include a narration indication and a conversation indication. In some examples, the intended emotion indication and the acting task can depend on the natural language input.” ¶ [0076]: “At step 1006, process 1000 can obtain a gesture generation machine learning model. In some examples, the gesture generation machine learning model can include a transformer network including an encoder and a decoder. However, it should be appreciated that the gesture generation machine learning model is not limited to a transformer network. For example, the gesture generation machine learning model can include a recurrent neural network (“RNN”), a long short-term memory (“LSTM”) model, a gated recurrent unit (“GRU”) model, a Markov process, a deep neural network (“DNN”), a convolutional neural network (“CNN”), a support vector machine (“SVM”), or any other suitable neural network model.”);
generating a sequence of latent feature representations of the human motion sequence (e.g., ¶ [0079]: “to produce the one or more latent representations.” ¶ [0081]: “produce the one or more latent representations.”) utilizing an encoder of a motion model (e.g., ¶ [0028]: “Modeling the input text and output gestures as sequences can become a sequence transduction problem. This problem can be resolved by using a transformer-based network. The transformer network can include the encoder-decoder architecture for sequence-to-sequence modeling.” ¶ [0029]: “In the transformer encoder, the self-attention operates on the input sequence W.” ¶ [0031]: “The transformer encoder then can pass the MH output through two fully-connected (FC) layers.” ¶ [0031]: “The final encoded representation of the input sequence W can be denoted as FW.” ¶ [0032] To meet the given constraints on the acting task A, intended emotion E, gender G, and/or handedness H of the virtual agent, these variables can be appended to FW, and the combined representation can be passed through two fully-connected layers with trainable parameters WFC to obtain feature representations” ¶ [0034]: “FIG. 2 shows the overall architecture 200 of an example transformer-based network. For example, the example network can take in sentences of natural language text 202 and transform the sentences to word embeddings 204 (e.g., using the pre-trained GloVe model). The example network can then use a transformer encoder 206 to transform the word embeddings 204 to latent representations 208, append the agent attributes 210 to these latent representations 208, and transform the combined representations into encoded features 212.” ¶ [0035]: “In some examples, the word embedding layer can transform the words into feature vectors (e.g., using the pre-trained GloVe model).” ¶ [0077]: “At step 1008, process 1000 can provide the sequence of one or more word embeddings and the one or more attributes to the gesture generation machine learning model. In some examples, block 1010 is the gesture generation machine learning model. Steps 1012-1020 in the block 1010 are steps in the gesture generation machine learning model. Thus, process 1000 can perform steps 1010-1020 in block 1010 using the gesture generation machine learning model.” ¶ [0078]: “Steps 1012 and 1014 are performed in an encoder of the gesture generation machine learning model. At step 1012, process 1000 can receive, via an encoder of the gesture generation machine learning model, the sequence of the one or more word embeddings. In some examples process 1000 can signify a position of each word embedding in the sequence of one or more word embeddings (e.g., using a positional encoding scheme). In further examples, the position can be signified prior to using an encoder self-attention component in the encoder.” ¶ [0082]: “At step 1014, the gesture generation machine learning model can produce, via the encoder, an output based on the one or more word embeddings. In some examples, the encoder of the machine learning model can produce one or more latent representations based on the sequence of the one or more word embeddings.” ¶ [0083]: “At step 1016, the gesture generation machine learning model can generate one or more encoded features based on the output and the one or more attributes. In some examples, the gesture generation machine learning model can combine the one or more latent representations (FW) from the encoder with the one or more attributes (i.e., the acting task A, the intended emotion indication E, the gender indication G, and/or the handedness indication H). In further example, process 1000 can transforms the combined one or more latent representations into the one or more encoded features.”) (¶ [0020]: “given a natural language text sentence associated with an acting task of narration or conversation, an intended emotion, and attributes of the virtual agent, including gender and handedness, the virtual agent's corresponding gestures (e.g., body gestures) can be generated. In other words, a sequence of relative 3D joint rotations custom-character underlying the poses of a virtual agent can be generated. Here, the sequence of relative 3D joint rotations can correspond to a sequence of input words custom-character. In further examples, the sequence of relative 3D joint rotations can be subject to the acting task A and the intended emotion E based on the text, and the gender G and the handedness H of the virtual agent.” ¶ [0022]: “a gesture can be represented as a sequence of poses or configurations of the 3D body joints. The sequence of poses or configurations can include body expressions as well as postures. In further examples, each pose can be represented with quaternions denoting 3D rotations of each joint relative to its parent in the directed pose graph as shown in FIG. 1.” ¶ [0028]: “Modeling the input text and output gestures as sequences can become a sequence transduction problem. This problem can be resolved by using a transformer-based network. The transformer network can include the encoder-decoder architecture for sequence-to-sequence modeling. However, instead of using sequential chains of recurrent memory networks, or the computationally expensive convolutional networks, the example transformer uses a multi-head self-attention mechanism to model the dependencies between the elements at different temporal positions in the input and target sequences.”);

generating, utilizing a decoder of the (e.g., ¶ [0033]: “The transformer decoder”), digital content comprising a reconstructed human motion sequence (¶ [0020]: “a sequence of relative 3D joint rotations Q* underlying the poses of a virtual agent can be generated. Here, the sequence of relative 3D joint rotations can correspond to a sequence of input words W.” ¶ [0033]: “the gesture sequence Q.” ¶ [0067]: “display a sequence of gestures of the virtual agent based on an output of the gesture generation machine learning model.”) (¶ [0033]: “The transformer decoder can operate similarly using the target sequence Q,” ¶ [0034]: “The transformer decoder can take in these encoded features 212 and the past gesture history 214 to predict gestures 218 for the subsequent time steps using a transformer decoder 216. At each time step, the gesture can be represented by the set of rotations on all the body joints relative to their respective parents in the pose graph at that time step.” ¶ [0035]: “At the output of the decoder 216, the predicted values can be normalized so that the predicted values represent valid rotations. In some examples, the example network can be trained using the sum of three losses: the angle loss, the pose loss, and the affective loss. These losses can be computed between the gesture sequences generated by the example network and the original motion-captured sequences available as ground-truth in the training dataset.” ¶ [0064]: “FIG. 9 shows an example 900 of a system for gesture generation in accordance with some embodiments of the disclosed subject matter. As shown in FIG. 9, a computing device 910 can receive a natural language input (e.g., text) 930. In further examples, the computing device 910 can obtain a gesture generation machine learning model and/or attribute(s). The computing device 910 processes the input (e.g., the natural language input 930 and/or attributes using the gesture generation machine learning model to produce a predicted gesture 950 of a virtual agent aligned with the natural language input.” ¶ [0084]: “At step 1018, the gesture generation machine learning model can receive, via a decoder of the gesture generation machine learning model, the one or more encoded features and a first gesture of a virtual agent. In further examples, the first gesture can include a set of rotations on multiple body joints relative to one or more parent body joints.” ¶ [0085]: “In some examples, the decoder can generate the first emotive gesture at a preceding time step.” ¶ [0087]: “At step 1020, the gesture generation machine learning model can produce, via the decoder, a second emotive gesture based on the one or more encoded features and the first emotive gesture. In some examples, the fully connected layer of the decoder can produce the second emotive gesture. In further examples, the second gesture can include a set of rotations on multiple body joints relative to one or more parent body joints based on the first emotive gesture.” ¶ [0088]: “At step 1022, process 1000 can provide the second gesture of the virtual agent from the gesture generation machine learning model. In some examples, process 1000 can apply the set of rotations on multiple body joints to the virtual agent and display the movement of the virtual agent. In further examples, the second emotive gesture can include head movement of the virtual agent aligned with the natural language input. However, it should be appreciated that the second emotive gesture can include body movement, hand movement, and any other suitable movement.” ¶ [0091]: “At step 1122, process 1100 can train the gesture generation machine learning model based on the ground-truth gesture and the second emotive gesture. For example, the gesture generation machine learning model can be trained based on a loss function (L) summing an angle loss, a pose loss, and an affective loss. In some examples, a ground-truth gesture can include ground-truth relative rotation of a joint, and the second emotive gesture comprises a predicted relative rotation of the joint.”).
Whereas BHATTACHARYA is not entirely explicit as to, Wu teaches:
receiving instructions comprising an indication of a human motion sequence (p. 1, 2nd paragraph of § 1: “text-to-video generation (T2V) task. Given a natural description, T2V requires the machine to understand it and create a semantically consistent video.” 1st paragraph of p. 2: “generate open-domain videos from text using VQ-VAE and three-dimensional sparse attention.” p. 4, 1st paragraph of § 3.2: “Given an input text t” p. 6, 1st paragraph of § 4.2.1: “t denotes the input text.” p. 6, 1st paragraph of § 4.4: “query "Play golf on grass", p. 8, 1st paragraph of § 5: “generate open-domain videos from natural descriptions using VQ-VAE discrete visual tokens.”);
determining the human motion sequence based on an intent extracted from the instructions (p. 1, 2nd paragraph of § 1: “text-to-video generation (T2V) task. Given a natural description, T2V requires the machine to understand it and create a semantically consistent video.” p. 6, 1st paragraph of § 4.4: “query "Play golf on grass", p. 7, 1st paragraph of § 4.4: “For example, GODIVA(64×64) first shows the grass field, then it gives the athlete a close-up shot, and finally the action of hitting the golf ball.” Figure 2: “Language Embedding” p. 4, 1st paragraph of § 3.2: “the text embedding matrix,”);
generating a sequence of latent feature representations of the human motion sequence (p. 2, 4th paragraph of § 1: “discrete video tokens.” p. 4, Figure 2 caption: “the VQ-VAE discrete representation” p. 4, 1st paragraph of § 3.2: “a sequence of discrete latent visual token embeddings” See the video representations of “humans” in Figure 3.) utilizing an encoder of a discretized motion model (e.g. pg. 4, Figure 2: “VQ-VAE Encoder” p. 6, 1st paragraph of § 4.3: “the encoder E in Eq. (1)”) (p. 1, 3rd paragraph of § 1: “a VQ-VAE pretrained model for the T2V task.” p. 2, 4th paragraph of § 1: “a VQ-VAE auto-encoder is trained to represent continuous video pixels with discrete video tokens. Then, a three-dimensional sparse attention model is trained using language as input and the discrete video tokens as labels to generate videos,” p. 4, Figure 2 caption: “the VQ-VAE discrete representation” p. 4, 1st paragraph of § 3.2: “use the pretrained VQ-VAE Encoder (Eq. (1∼3)) to encode each frame the ground-truth videos in Eq. (7):” … “where the ground-truth video sequence x ∈ RL×H×W×C is encoded into a sequence of discrete latent visual token embeddings b ∈ RM×Db”. p. 4, 1st paragraph of § 3.2: “pretrained VQ-VAE Encoder”);
mapping the sequence of latent feature representations (p. 3, § 3.1: “where y(l) … is the latent variable”) to a plurality of learned latent feature representations (p. 3, § 3.1: “Then, y(l) is quantized to get a more compact latent representation, as denoted in Eq. (2).” p. 3, § 3.1: “where B” … “is the codebook where the ith region of the latent variable y i(l) “ … “is searched to find the nearest indexes z(l)” p. 3, § 3.1: “Then, z(l) is embedded by the codebook in Eq. (3).”) corresponding to human motions (p. 6, 1st paragraph of § 4.4: “T2V successfully generates the grass and action of "playing golf" in a resolution of 64×64,” p. 7, 1st paragraph of § 4.4: “For example, GODIVA(64×64) first shows the grass field, then it gives the athlete a close-up shot, and finally the action of hitting the golf ball.”); and
generating, utilizing a decoder of the discretized motion model (pg. 4, Figure 2: “VQ-VAE Decoder”; p. 4, 1st paragraph of § 3.2: “a decoder can be trained to generate videos” p. 6, 1st paragraph of § 4.3: “the decoder D in Eq. (4)”), digital content comprising a reconstructed human motion sequence based on the plurality of learned latent feature representations (p. 4, § 3.2: “encoded into a sequence of discrete latent visual token embeddings b” p. 5, 1st paragraph of § 4.4.1: “v^ is the predicted video with L frames.” p. 6, 1st paragraph of § 4.4: “T2V successfully generates the grass and action of "playing golf" in a resolution of 64×64,” p. 7, 1st paragraph of § 4.4: “For example, GODIVA(64×64) first shows the grass field, then it gives the athlete a close-up shot, and finally the action of hitting the golf ball.” See Figure 2 and Figure 3.).
Thus, in order to obtain a more computationally efficient system (as suggested by WU in the 2nd paragraph on pg. 2), it would have been obvious to one of ordinary skill in the art to have modified the system taught by BHATTACHARYA so as to use a trained encoder and decoder of a discretized motion model mapping sequences of latent feature representations to a plurality of learned latent feature representations in a discretized feature space, as taught by WU.
Regarding claim 15 (depends on claim 10), BHATTACHARYA discloses:
wherein generating the sequence of latent feature representations comprises generating the sequence of latent feature representations utilizing a plurality of transformer neural network layers of the encoder of the discretized motion model (¶ [0028]: “Using the Transformer Network: Modeling the input text and output gestures as sequences can become a sequence transduction problem. This problem can be resolved by using a transformer-based network. The transformer network can include the encoder-decoder architecture for sequence-to-sequence modeling. However, instead of using sequential chains of recurrent memory networks, or the computationally expensive convolutional networks, the example transformer uses a multi-head self-attention mechanism to model the dependencies between the elements at different temporal positions in the input and target sequences.” ¶ [0034]: “Training the Transformer-Based Network: FIG. 2 shows the overall architecture 200 of an example transformer-based network. For example, the example network can take in sentences of natural language text 202 and transform the sentences to word embeddings 204 (e.g., using the pre-trained GloVe model). The example network can then use a transformer encoder 206 to transform the word embeddings 204 to latent representations 208, append the agent attributes 210 to these latent representations 208, and transform the combined representations into encoded features 212.” ¶ [0076]: “At step 1006, process 1000 can obtain a gesture generation machine learning model. In some examples, the gesture generation machine learning model can include a transformer network including an encoder and a decoder. However, it should be appreciated that the gesture generation machine learning model is not limited to a transformer network. For example, the gesture generation machine learning model can include a recurrent neural network (“RNN”), a long short-term memory (“LSTM”) model, a gated recurrent unit (“GRU”) model, a Markov process, a deep neural network (“DNN”), a convolutional neural network (“CNN”), a support vector machine (“SVM”), or any other suitable neural network model.” ¶ [0083]: “a fully connected layer in the machine learning model can transform the combined one or more latent representations into the one or more encoded features. The fully connected layer can be multiple fully connected layers.”).
Regarding claim 16 (depends on claim 10), BHATTACHARYA discloses:
receiving the instructions comprises receiving computing instructions from a three-dimensional modeling program (claim 3: “generating three-dimensional pose sequences based on the second emotive gesture for the virtual agent corresponding to emotive gestures aligned with the natural language text input.” ¶ [0003]: “Interactions between humans and virtual agents are being used in various applications, including online learning, virtual interviewing and counseling, virtual social interactions, and large-scale virtual worlds. Current game engines and animation engines can generate humanlike movements for virtual agents.” ¶ [0004]: “receiving a sequence of one or more word embeddings and one or more attributes; obtaining a gesture generation machine learning model; providing the sequence of one or more word embeddings and the one or more attributes to the gesture generation machine learning model; and providing a second emotive gesture of the virtual agent from the gesture generation machine learning model.”); and
determining the human motion sequence comprises converting the instructions into the human motion sequence (¶ [0019]: “interactively generate emotive gestures (e.g., head gestures, hand gestures, full-body gestures, etc.) for virtual agents aligned with natural language inputs (e.g., text, speech, etc.).” ¶ [0022]: “a sequence of poses or configurations of the 3D body joints.”).

Claims 6-7, 9 are rejected under 35 U.S.C. 103 as being unpatentable over BHATTACHARYA et al. (US 2023/0135769) in view of WU et al. (Wu, Chenfei, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. "Godiva: Generating open-domain videos from natural descriptions." arXiv preprint arXiv:2104.14806. 2021 Apr 30), further in view of KAVUKCUOGLU et al. (US 2020/0184316, hereinafter “KAVUKCUOGLU”).
Regarding claim 6 (depends on claim 1), whereas BHATTACHARYA and WU are not entirely explicit as to, KAVUKCUOGLU teaches:
wherein generating the digital content comprises generating, utilizing the decoder, the reconstructed human motion sequence from the learned mappings of the sequence of latent feature representations (see the rejection of claim 1 above) according to a plurality of weights corresponding to the learned mappings (¶ [0048]: “In order for the decoder neural network to be able to generate high quality reconstructions from decoder inputs, a training system 190 trains the encoder neural network 110 and the decoder neural network 170 jointly to determine trained values of the encoder network parameters and the decoder network parameters while also adjusting the latent embedding vectors in the memory 130 (and 152) to allow the latent embedding vectors to effectively represent features of input data items. This training is described in more detail below with reference to FIG. 4.” ¶ [0075]: “The system generates the reconstruction of the input data item by processing the decoder input using the decoder neural network (step 306).” ¶ [0076]: “FIG. 4 is a flow diagram of an example process 400 for training the encoder neural network, the decoder neural network, and updating the latent embedding vectors. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 190 of FIG. 1, appropriately programmed, can perform the process 400.” ¶ [0077]: “The system can repeatedly perform the process 400 to repeatedly update the values of the encoder network parameters, the decoder network parameters, and the latent embedding vectors.” ¶ [0083]: “The system determines a reconstruction update to the current values of the decoder network parameters and the encoder network parameters (step 406).” ¶ [0084]: “In particular, the system determines the reconstruction updates by determining a gradient with respect to the current values of the decoder network parameters and the encoder network parameters of a reconstruction error between the training reconstruction and the training data item, i.e., to optimize the reconstruction error.” ¶ [0101]: “Once the system has performed the process 400 for each training data item in a mini-batch of training data items, the system applies the updates to the current values of the encoder network parameters and the decoder network parameters and to the current embedding vectors, e.g., in accordance with the update rule employed by the optimizer used by the system in the training, e.g., the Adam optimizer or another gradient descent-based optimizer.” ¶ [0102]: “The system repeats the process 400 for multiple mini-batches to determine the trained encoder and decoder network parameter values and the final set of latent embedding vectors.”).
Thus, in order to obtain the trained decoder for generating the digital content, it would have been obvious to one of ordinary skill in the art to have trained the decoder taught by the combination of BHATTACHARYA and WU to obtain the trained weights of the decoder corresponding to learned mappings, as more clearly taught by KAVUKCUOGLU.
Regarding claim 7 (depends on claim 1), whereas BHATTACHARYA and WU are not entirely explicit as to, KAVUKCUOGLU teaches: determining a reconstruction loss (¶ [0084]: “a reconstruction error” ¶ [0085]: “the reconstruction error is a reconstruction loss) based on differences between the human motion sequence (¶ [0082]: “training decoder input”) and the reconstructed human motion sequence (¶ [0082]: “the training reconstruction of the training data item.”) (¶ [0082]: “The system then generates a training decoder input that includes the nearest current latent embedding vectors and processes the training decoder input through the decoder neural network in accordance with the current values of the decoder network parameters to generate the training reconstruction of the training data item.” ¶ [0083]: “The system determines a reconstruction update to the current values of the decoder network parameters and the encoder network parameters (step 406).” ¶ [0084]: “In particular, the system determines the reconstruction updates by determining a gradient with respect to the current values of the decoder network parameters and the encoder network parameters of a reconstruction error between the training reconstruction and the training data item, i.e., to optimize the reconstruction error.”);
adjusting parameters of the discretized motion model based on the reconstruction loss (¶ [0082]: “The system then generates a training decoder input that includes the nearest current latent embedding vectors and processes the training decoder input through the decoder neural network in accordance with the current values of the decoder network parameters to generate the training reconstruction of the training data item.” ¶ [0083]: “The system determines a reconstruction update to the current values of the decoder network parameters and the encoder network parameters (step 406).” ¶ [0084]: “In particular, the system determines the reconstruction updates by determining a gradient with respect to the current values of the decoder network parameters and the encoder network parameters of a reconstruction error between the training reconstruction and the training data item, i.e., to optimize the reconstruction error.” ¶ [0045]: “The decoder neural network 170 has been trained to process the decoder input 162 to generate the reconstruction 172 of the input data item 102 in accordance with a set of parameters (referred to in this specification as “decoder network parameters”). The decoder neural network 170 can be the same type of neural network as the encoder neural network 110, but configured to generate a reconstruction from a decoder input rather than an encoder output (which is the same size as the decoder input) from an input data item (which is the same size as the reconstruction).” ¶ [0048]: “In order for the decoder neural network to be able to generate high quality reconstructions from decoder inputs, a training system 190 trains the encoder neural network 110 and the decoder neural network 170 jointly to determine trained values of the encoder network parameters and the decoder network parameters while also adjusting the latent embedding vectors in the memory 130 (and 152) to allow the latent embedding vectors to effectively represent features of input data items. This training is described in more detail below with reference to FIG. 4.” ¶ [0101]: “Once the system has performed the process 400 for each training data item in a mini-batch of training data items, the system applies the updates to the current values of the encoder network parameters and the decoder network parameters and to the current embedding vectors, e.g., in accordance with the update rule employed by the optimizer used by the system in the training, e.g., the Adam optimizer or another gradient descent-based optimizer.” ¶ [0102]: “The system repeats the process 400 for multiple mini-batches to determine the trained encoder and decoder network parameter values and the final set of latent embedding vectors.”); and
modifying one or more learned mappings of a codebook of the discretized motion model (¶ [0042]: “the latent embedding vectors in a latent embedding memory 152.” ¶ [0088]: “the latent embedding vectors currently stored in the memory”) based on the reconstruction loss (¶ [0088]: “In some other implementations, the system determines a subgradient through the operation of selecting the nearest current latent embedding vector for each latent variable and uses the subgradient to determine the gradient with respect to the current values of the encoder network parameters.” ¶ [0089]: “By determining this update, the system encourages the encoder and decoder neural networks to generate higher quality reconstructions given the current latent embedding vectors, i.e., the latent embedding vectors currently stored in the memory.” ¶ [0090]: “The system determines updates to the current latent embedding vectors that are stored in the memory (step 408).” ¶ [0091]: “In particular, in some implementations, for each latent variable, the system determines an update to the nearest current latent embedding vector for the latent variable by determining a gradient with respect to the nearest current latent embedding vector of an error between the training encoded vector for the latent variable and the nearest current latent embedding vector for the latent variable, i.e., to minimize the error.” ¶ [0042]: “The decoder subsystem 160 is configured to receive the discrete latent representation 122 and generate a decoder input 162 using the latent embedding vectors in a latent embedding memory 152.” ¶ [0043]: “The latent embedding memory 152 generally stores the same latent embedding vectors as the latent embedding memory 130.” ¶ [0048]: “In order for the decoder neural network to be able to generate high quality reconstructions from decoder inputs, a training system 190 trains the encoder neural network 110 and the decoder neural network 170 jointly to determine trained values of the encoder network parameters and the decoder network parameters while also adjusting the latent embedding vectors in the memory 130 (and 152) to allow the latent embedding vectors to effectively represent features of input data items. This training is described in more detail below with reference to FIG. 4.” ¶ [0101]: “Once the system has performed the process 400 for each training data item in a mini-batch of training data items, the system applies the updates to the current values of the encoder network parameters and the decoder network parameters and to the current embedding vectors, e.g., in accordance with the update rule employed by the optimizer used by the system in the training, e.g., the Adam optimizer or another gradient descent-based optimizer.” ¶ [0102]: “The system repeats the process 400 for multiple mini-batches to determine the trained encoder and decoder network parameter values and the final set of latent embedding vectors.”).
Thus, in order to obtain the trained discretized encoder-decoder (e.g., VQ-VAE) human motion model taught by the combination of BHATTACHARYA and WU for generating the digital content, it would have been obvious to one of ordinary skill in the art to have trained the discretized encoder-decoder (e.g., VQ-VAE) by determining a reconstruction loss based on differences between the human motion sequence and the reconstructed human motion sequence, adjusting parameters of the discretized motion model based on the reconstruction loss, and modifying one or more learned mappings of a codebook of the discretized motion model based on the reconstruction loss, as more clearly taught by KAVUKCUOGLU.
Regarding claim 9 (depends on claim 1), although WU teaches “a VQ-VAE is trained to represent continuous video pixels with discrete video tokens,” BHATTACHARYA and WU fail to explicitly disclose: wherein generating the sequence of latent feature representations comprises generating, utilizing a plurality of convolutional neural network layers of the encoder, the sequence of latent feature representations in a continuous latent space.
However, whereas BHATTACHARYA and WU are not entirely explicit as to, KAVUKCUOGLU clearly teaches:
wherein generating the sequence of latent feature representations comprises generating, utilizing a plurality of convolutional neural network layers of the encoder (See “CNN’ layers of “ENCODER” 112 in FIG. 1B.), the sequence of latent feature representations in a continuous latent space (¶ [0033]: “Unlike a system that generates continuous latent representations and instead of using the encoder output as the representation of the input data item 102, the encoder subsystem 120 generates the discrete latent representation 122 of the input data item 102 using the encoder output 112 and the latent embedding vectors in the memory 130.” ¶ [0034]: “In particular, for each latent variable, the encoder subsystem 120 determines, from the set of latent embedding vectors in the memory 130, a latent embedding vector that is nearest to the encoded vector for the latent variable. For example, the subsystem 120 can determine the latent embedding vector that is nearest to a given encoded vector using a nearest neighbor lookup on the set of latent embedding vectors or any other appropriate distance metric.” NOTE: In other words, the encoder output 112 (i.e., the encoded vector for the latent variable) is a continuous latent representation until it is discretized using a nearest neighbor lookup on the set of latent embedding vectors in the memory 130 (i.e., the codebook of discrete latent representations). ¶ [0004]: “The latent representation is referred to as a discrete latent representation because, unlike a continuous representation, the value for each of the latent variables is selected from a discrete set of possible values. More specifically, the value for each of the latent variables is a vector selected from a discrete set of latent embedding vectors.” ¶ [0011]: “The described system can generate latent representations of input data items that allow high quality reconstructions of the input data items to be generated even though the latent representations are much smaller than latent representations generated by conventional neural network-based encoders that allow reconstructions of comparable (or even lesser) quality to be generated, e.g., continuous latent variable models. In particular, because of the way that the described systems select the value for each latent variable from a discrete set of latent embedding vectors, the latent representations are discrete and can be stored using very little memory or transmitted using very little bandwidth while still allowing high quality reconstructions to be generated.”).
Thus, in order to obtain a more versatile system, it would have been obvious to one of ordinary skill in the art to modify the system taught by the combination of BHATTACHARYA and WU so that generating the sequence of latent feature representations comprises generating, utilizing a plurality of convolutional neural network layers of the encoder, the sequence of latent feature representations in a continuous latent space, as clearly taught by KAVUKCUOGLU.

Claims 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over AHUJA et al. (Ahuja C, Morency LP. Language2pose: Natural language grounded pose forecasting. In 2019 International conference on 3D vision (3DV) 2019 Sep 16 (pp. 719-728). IEEE; hereinafter “AHUJA”) in view of KAVUKCUOGLU et al. (US 2020/0184316, hereinafter “KAVUKCUOGLU”).
Regarding claim 17, AHUJA discloses a non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause a computing device to perform operations (p 719, Abstract: “a neural network architecture”. One of ordinary skill in the art would understand that the process implemented by the neural network architecture is conventionally implemented by at least one processor executing instructions stored in memory.) comprising:
generating, utilizing a plurality of transformer neural network layers of an encoder of a (e.g., p. 720, Figure 2: “Pose Encoder qe” of “model Joint Language-to-Pose (or JL2P).” p. 723, 1st paragraph of § 5.2: “For pose encoder (qe) a network of Gated Recurrent Units (GRUs) [9] is used in our model JL2P.” ), a sequence of latent feature representations of a human motion sequence in a continuous latent space (e.g., p. 720, Figure 2 caption: “Language and pose are mapped to a joint embedding space Z” p. 722, 1st paragraph of § 4.1: “To learn a joint embedding space of language and pose, the sentence X1:N and pose Y1:T are first mapped to a latent representation using a sentence encoder pe (X1:N;Φe) and a pose encoder qe (Y1:T;Ψe) respectively. These estimate the latent representation or embeddings zx and zy respectively in the embedding space Z”) from an unlabeled digital scene (e.g., the sequence of poses entered in Pose Encoder qe shown in Figure 2 are unlabeled. p. 722, 1st paragraph of § 4.1: “pose Y1:T “);

generating, utilizing a decoder of the (e.g., p. 720, Figure 2: “Pose Decoder qd” of “model Joint Language-to-Pose (or JL2P).”), digital content comprising a reconstructed human motion sequence (e.g., p. 720, Figure 2 caption: “generate a pose sequence”; p. 721, 2nd paragraph of § 3: “predict a T-sized sequence of 3D poses Y1:T =[ y1, y2, ... yT ]” ) based on the sequence of (e.g., p. 720, Figure 2 caption: “embedding space Z”) (e.g., p. 720, Figure 2 caption: “Language and pose are mapped to a joint embedding space Z, which can now be used by a trained pose decoder qd to generate a pose sequence. At train time both pe and qe are used to create the joint embedding using a training curriculum. But at inference time z ∈ Z is encoded by pe and decoded by qd, giving us a model which can generate a animation (or sequence of poses) from a free form description (or language).” p. 722, 1st paragraph of § 4.2: “Once we have the embedding zx or zy, a pose decoder qd(; Ψd) is used to generate an animation from the joint embedding space Z.”).
Whereas AHUJA is not explicit as to, KAVUKCUOGLU teaches:
a non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause a computing device to perform operations (¶ [0005]: “one or more computers and one or more storage devices storing instructions that when executed by the one or more computers” ¶ [0109]: “The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output.” ¶ [0110]: “Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit.” ¶ [0111]: “Computer readable media suitable for storing computer program instructions”) comprising:
generating, utilizing a plurality of transformer neural network layers of an encoder of a discretized motion model (e.g., ¶ [0027]: “an encoder neural network 110,” ¶ [0030]: “the encoder neural network 110 can be a convolutional neural network that processes the image to generate the encoder output.” ¶ [0032]: ” the encoder neural network 110 can be a convolutional neural network with layers that propagate information across time, e.g., recurrent layers or pooling layers, or a neural network that includes three-dimensional convolutional layers” ), a sequence of latent feature representations of a human motion sequence in a continuous latent space (e.g., ¶ [0031]: “the encoder neural network 110 generates the encoder output 112 as a sequence of encoded vectors, with each position in the sequence corresponding to a different latent variable.” ¶ [0033]: “generates continuous latent representations”, ¶ [0033]: “the encoder output 112” ¶ [0028]: “The encoder neural network 110 is a neural network that has been configured through training to process the input data item 102 to generate an encoder output 112 for the input data item 102 in accordance with a set of parameters (referred to in this specification as “encoder network parameters”).” ¶ [0031]: “the encoder neural network 110 generates the encoder output 112 as a sequence of encoded vectors, with each position in the sequence corresponding to a different latent variable.” ¶ [0032]: “When the input data item 102 is a video, each latent variable corresponds to a distinct point in a three-dimensional feature map. That is, the encoder output 112 includes a respective encoded vector for each point in the three-dimensional feature map and each point corresponds to a different latent variable. In these cases, the encoder neural network 110 can be a convolutional neural network with layers that propagate information across time, e.g., recurrent layers or pooling layers, or a neural network that includes three-dimensional convolutional layers that receives the video as a sequence of frames and generates the three-dimensional feature map.” ¶ [0054]: “an action being performed by an entity depicted in the existing portion of the video.” NOTE: If the input data item 102 is a video, the video could clearly be a video of a human motion sequence, i.e., an action performed by an entity, which could be a human. ¶ [0066]: “The system processes the input data item using an encoder neural network to generate an encoder output for the input data item (step 204). In particular, as described above, the encoder neural network is configured to process the input data item to generate an output that includes a respective encoded vector for each of one or more latent variables.”) from an unlabeled digital scene (¶ [0024]: “The encoder system 100 receives an input data item 102 and encodes the input data item 102 to generate a discrete latent representation 122 of the input data item 102.” ¶ [0032]: “When the input data item 102 is a video,”);
converting, utilizing a codebook of the discretized motion model (¶ [0027]: “a latent embedding vector memory 130 that stores a set of latent embedding vectors.” ¶ [0043]: “The latent embedding memory 152 generally stores the same latent embedding vectors as the latent embedding memory 130. When the encoder system 100 and decoder system 150 are implemented on the same set of computers, the memory 130 and the memory 152 can be the same memory.”), the sequence of latent feature representations into a sequence of discretized feature representations (¶ [0037: “in the discrete latent representation 122, data that identifies, for each latent variable, the nearest latent embedding vector to the encoded vector for the latent variable.”) (¶ [0004]: “generates discrete latent representations of input data items, e.g., images, audio data, videos, electronic documents, and so on. Generally, each discrete latent representation identifies a respective value for each of one or more latent variables, where the number of latent variables is fixed. The latent representation is referred to as a discrete latent representation because, unlike a continuous representation, the value for each of the latent variables is selected from a discrete set of possible values. More specifically, the value for each of the latent variables is a vector selected from a discrete set of latent embedding vectors.” ¶ [0033]: “Unlike a system that generates continuous latent representations and instead of using the encoder output as the representation of the input data item 102, the encoder subsystem 120 generates the discrete latent representation 122 of the input data item 102 using the encoder output 112 and the latent embedding vectors in the memory 130.” ¶ [0034]: “In particular, for each latent variable, the encoder subsystem 120 determines, from the set of latent embedding vectors in the memory 130, a latent embedding vector that is nearest to the encoded vector for the latent variable. For example, the subsystem 120 can determine the latent embedding vector that is nearest to a given encoded vector using a nearest neighbor lookup on the set of latent embedding vectors or any other appropriate distance metric.” ¶ [0059]: “The system 100 then generates the discrete latent representation 122 using the encoder output 112 and the set of latent embedding vectors stored in the memory 130. In particular, in the example of FIG. 1B, the memory 130 stores K latent embedding vectors e1 through eK.” ¶ [0060]: “To generate the latent representation 122, the system 100 identifies, for each of the latent variables, the latent embedding vector of the K latent embedding vectors that is nearest to the encoded vector for the latent variable, e.g., using a nearest neighbor look-up. The system 100 then generates the latent representation 122 that identifies, for each of the latent variables, the nearest latent embedding vector to the encoded vector for the latent variable.” ¶ [0067]: “The system generates a discrete latent representation of the input data item using the encoder output (step 206).” ¶ [0068]: “In particular, for each latent variable, the system selects the latent embedding vector stored in the latent embedding vector memory that is nearest to the encoded vector for the latent variable.” ¶ [0069]: “The system then generates a discrete latent representation that identifies, for each of the latent variables, the nearest latent embedding vector to the encoded vector for the latent variable.”); and
generating, utilizing a decoder of the discretized motion model (e.g., ¶ [0041]: “The decoder system 150 includes a decoder subsystem 160 and a decoder neural network 170.”), digital content comprising a reconstructed human motion sequence (¶ [0045]: “The decoder neural network 170 has been trained to process the decoder input 162 to generate the reconstruction 172 of the input data item 102 in accordance with a set of parameters (referred to in this specification as “decoder network parameters”). The decoder neural network 170 can be the same type of neural network as the encoder neural network 110, but configured to generate a reconstruction from a decoder input rather than an encoder output (which is the same size as the decoder input) from an input data item (which is the same size as the reconstruction).” ¶ [0047]: “Generating a reconstruction of an input data item from a discrete latent representation of the data item is described in more detail below with reference to FIGS. 1B and 3.” NOTE: Where the input data item is a video of human motion sequence, i.e., a sequence of image frames of a human moving, the reconstruction of the input data would be a reconstructed human motion sequence. ¶ [0062]: “The system 150 then processes the decoder input 162 using the decoder neural network 170 to generate the reconstruction 172 of the input data item 102, i.e., an image the same size as the input image that is an estimate of the input based on the latent representation 122.”) based on the sequence of discretized feature representations (¶ [0042]: “The decoder subsystem 160 is configured to receive the discrete latent representation 122 and generate a decoder input 162 using the latent embedding vectors in a latent embedding memory 152.” ¶ [0061]: “The system 150 then generates the decoder input 162 using the latent embedding vectors and the latent representation 122. In particular, the system 150 generates the decoder input 162 as a three-dimensional feature map having a D dimensional vector at each of multiple spatial locations. The D dimensional vector at any given spatial location is the latent embedding vector identified for the corresponding latent variable in the latent representation 122.” ¶ [0073]: “The system obtains the discrete latent representation of the data item (step 302).” ¶ [0074]: “The system generates a decoder input from the discrete latent representation using the latent embedding vectors (step 304).” ¶ [0075]: “The system generates the reconstruction of the input data item by processing the decoder input using the decoder neural network (step 306).”).
Thus, in order to obtain a more computationally efficient human motion generation system with reduced memory and/or bandwidth requirements (as suggested in paragraphs [0011] and [0012] of KAVUKCUOGLU), it would have been obvious to one of ordinary skill in the art to have modified the human motion modelling system taught by AHUJA so as to include utilizing a codebook to convert the sequence of latent feature representations into a sequence of discretized feature representations and generate the reconstructed human motion sequence based on the sequence of discretized feature representations, as taught by KAVUKCUOGLU.
Regarding claim 18 (depends on claim 17), KAVUKCUOGLU further teaches:
generating the sequence of latent feature representations comprises encoding the sequence of latent feature representations into a continuous space embedding including a dimensionality corresponding to a number of codebook vectors (¶ [0033]: “Unlike a system that generates continuous latent representations and instead of using the encoder output as the representation of the input data item 102, the encoder subsystem 120 generates the discrete latent representation 122 of the input data item 102 using the encoder output 112 and the latent embedding vectors in the memory 130.” ¶ [0034]: “In particular, for each latent variable, the encoder subsystem 120 determines, from the set of latent embedding vectors in the memory 130, a latent embedding vector that is nearest to the encoded vector for the latent variable. For example, the subsystem 120 can determine the latent embedding vector that is nearest to a given encoded vector using a nearest neighbor lookup on the set of latent embedding vectors or any other appropriate distance metric.” ¶ [0058]: “In particular, the encoder neural network 110 processes the input image to generate the encoder output 112. As can be seen in FIG. 1B, the encoder output 112 is a respective D dimensional vector for each spatial location in a two-dimensional feature map, with each spatial location corresponding to a respective latent variable. Thus, the encoder output 112 includes a respective encoded vector for each of multiple latent variables.” ¶ [0059] The system 100 then generates the discrete latent representation 122 using the encoder output 112 and the set of latent embedding vectors stored in the memory 130. In particular, in the example of FIG. 1B, the memory 130 stores K latent embedding vectors e1 through eK.” ¶ [0060]: “To generate the latent representation 122, the system 100 identifies, for each of the latent variables, the latent embedding vector of the K latent embedding vectors that is nearest to the encoded vector for the latent variable, e.g., using a nearest neighbor look-up. The system 100 then generates the latent representation 122 that identifies, for each of the latent variables, the nearest latent embedding vector to the encoded vector for the latent variable.” ¶ [0061]: “The system 150 then generates the decoder input 162 using the latent embedding vectors and the latent representation 122. In particular, the system 150 generates the decoder input 162 as a three-dimensional feature map having a D dimensional vector at each of multiple spatial locations. The D dimensional vector at any given spatial location is the latent embedding vector identified for the corresponding latent variable in the latent representation 122.”).
Allowable Subject Matter
Claims 2-5, 8, 11-14 and 19-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
At present, it is not apparent to the examiner which part of the application could serve as a basis for new and allowable claims. However, should the applicant nevertheless regard some particular matter as patentable, the examiner encourages applicant to appropriately amend the claims to include such matter and to indicate in the REMARKS the difference(s) between the prior art and the claimed invention as well as the significance thereof.
Furthermore, should applicant decide to amend the claims, examiner respectfully requests that the applicant please indicate in the REMARKS from which page(s), line(s) or claim(s) of the originally filed application that any amendments are derived. See MPEP § 2163(II)(A) (There is a strong presumption that an adequate written description of the claimed invention is present in the specification as filed, Wertheim, 541 F.2d at 262, 191 USPQ at 96; however, with respect to newly added or amended claims, applicant should show support in the original disclosure for the new or amended claims.).
A shortened statutory period for reply to this action is set to expire THREE MONTHS from the mailing date of this action. Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action. Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 USC § 133).
Relevant Prior Art
The following prior art, although not relied upon, is made of record since it is considered pertinent to applicant's disclosure:
del Val Santos et al. (US 20200302667) discloses a computer-implemented method for generating a machine-learned model to generate facial position data based on audio data comprising training a conditional variational autoencoder having an encoder and decoder. The training comprises receiving a set of training data items, each training data item comprising a facial position descriptor and an audio descriptor; processing one or more of the training data items using the encoder to obtain distribution parameters; sampling a latent vector from a latent space distribution based on the distribution parameters; processing the latent vector and the audio descriptor using the decoder to obtain a facial position output; calculating a loss value based at least in part on a comparison of the facial position output and the facial position descriptor of at least one of the one or more training data items; and updating parameters of the conditional variational autoencoder based at least in part on the calculated loss value.
Lin et al. (Lin AS, Wu L, Corona R, Tai K, Huang Q, Mooney RJ. Generating animated videos of human activities from natural language descriptions. Learning. 2018 Dec;1(2018):1.) discloses a system that maps a natural language description to an animation of a humanoid skeleton. Our system is a sequence-to-sequence model that is pretrained with an autoencoder objective and then trained end-to-end.
Ahn et al. (Ahn H, Ha T, Choi Y, Yoo H, Oh S. Text2action: Generative adversarial synthesis from language to action. In 2018 IEEE International Conference on Robotics and Automation (ICRA) 2018 May 21 (pp. 5915-5920). IEEE.) discloses a generative model which learns the relationship between language and human action in order to generate a human action sequence given a sentence describing human behavior. The proposed generative model is a generative adversarial network (GAN), which is based on the sequence to sequence (SEQ2SEQ) model. Using the proposed generative network, we can synthesize various actions for a robot or a virtual agent using a text encoder recurrent neural network (RNN) and an action decoder RNN.
Plappert et al. (Plappert M, Mandery C, Asfour T. Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks. Robotics and Autonomous Systems. 2018 Nov 1;109:13-26.) discloses a generative model that learns a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks (RNNs) and sequence-to-sequence learning.
(Mama R, Tyndel MS, Kadhim H, Clifford C, Thurairatnam R. NWT: towards natural audio-to-video generation with representation learning. arXiv preprint arXiv:2106.04283. 2021 Jun 8.) discloses an expressive speech-to-video model comprising a discrete variational autoencoder with adversarial loss, dVAE-Adv, which learns a new discrete latent representation.
Yan et al. (Yan W, Zhang Y, Abbeel P, Srinivas A. VideoGPT: Video generation using VQ-VAE and Transformers. arXiv preprint arXiv:2104.10157. 2021 Apr 20.) discloses VideoGPT which uses a VQVAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention.
Ding et al. (Ding M, Yang Z, Hong W, Zheng W, Zhou C, Yin D, Lin J, Zou X, Shao Z, Yang H, Tang J. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems. 2021 Dec 6; 34:19822-35.) discloses a 4-billion-parameter Transformer with VQ-VAE tokenizer for text-to image generation.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT PEREN who can be reached by telephone at [phone], or via email at [e-mail]. The examiner can normally be reached on Monday-Friday from 10:00 A.M. to 6:00 P.M.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KING POON, can be reached at telephone number [phone]. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/VINCENT PEREN/
Examiner, Art Unit 2617

/KING Y POON/Supervisory Patent Examiner, Art Unit 2617

Prosecution Timeline

• Jun 27, 2024: Application Filed
• May 29, 2026: Non-Final Rejection mailed — §103
• Jul 14, 2026: Interview Requested