CTNF 18/374,424 CTNF 95890 DETAILED ACTION This action is responsive to the Claims filed on 09/28/2023. Claims 1-20 are pending in the case. Claims 1, 9, and 19 are independent claims. Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 102 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-12-aia AIA (a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention. 07-15-aia AIA Claim(s) 1-10, 19 and 20 is/are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Piergiovanni et al. “Learning Real-World Robot Policies by Dreaming” Claim 1 Piergiovanni teaches, A method comprising: obtaining a seed image representing a seeded application state (Section 4 Dreaming model pg 2 “Our dreaming model is a combination of a convolutional autoencoder and an action-conditioned future representation regressor. It enables encoding of an image frame (i.e., robot observation) into the representation abstracting scene (i.e., robot state)” the image from received is a seed image of an application, see figure 1 pg 3 picturing the obtained image.) mapping the seed image to at least one seed image token using an image encoder; ( Section 4 Dreaming model pg 2 “It enables encoding of an image frame (i.e., robot observation) into the representation abstracting scene (i.e., robot state), …More specifically, our dreaming model consists of the following four function components… State Encoder Encθ : I → s” pg 3 Section A “We learn a state representation based on a variational autoencoder (VAE) [34] which takes an image as input and learns a latent state representation optimized to reconstruct the given image” the Encoder, Enc, mapps the image to a seed token representation. The latent representation amounts to the token representation as described in paragraph 0052 of the specification.) inputting the at least one seed image token as a prompt to a neural dreaming model (pg 3 Section A “The encoder outputs µ,σ = Enc(I) and we sample from s ∼ N(µ,σ) to obtain the state representation during the training.” The encoder, which outputs the token and which is part of the illustrated dreaming model, is inputs the token representation to subsequent layers as prompt to the neural dreaming model as indicated in Figure 1, the caption noting the components which receive the latent representation as input “ Rectangles in the figure are the CNN layers” PNG media_image1.png 328 800 media_image1.png Greyscale ) that has been trained to predict training sequences obtained from one or more executions of one or more applications (pg 3 “We learn a state-transition model… Our state-transition model takes the current state, st, and action, at, as input and outputs the next state, st+1. Since our state representation is convolutional, we train a CNN… Using this loss enables our learning to minimize the difference between the the future-regressed state and the true future state” the model is trained via minimization to learn the future state or training sequences obtained from the state representation of the actions.) the training sequences including images output by the one or more applications during the one or more executions and inputs to the one or more applications during the one or more executions; ( pg 5 “To train the dreaming model, we collect a dataset consisting of 40,000 images… We allow the robot to take random actions in the environment and we store the starting image, action and resulting image pairs” the model is trained with image pairs from the dataset including staring images, i.e input images to the application, and resulting images, i.e output images to the application.) generating subsequent image tokens with the neural dreaming model; and decoding the subsequent image tokens with an image decoder to obtain subsequent images. (pg 3 “Fig. 1: Illustration of our dreaming model. (a) The encoder, action representation, future regressor and decoder modules” as shown in the figure, the decoder generates subsequent image tokens which are decoded into subsequent images, i.e I_t+1) Claim 2 Piergiovanni teaches claim 1 Piergiovanni teaches, wherein the neural dreaming model comprises a transformer decoder. (pg 3 “Fig. 1: Illustration of our dreaming model. (a) The encoder, action representation, future regressor and decoder modules” the decoder modules are transformation decoders, the decoders not only decode images but the latent representation of the actions) Claim 3 Piergiovanni teaches claim 1 Piergiovanni teaches, wherein the neural dreaming model is a multi-modal model and the generating also involves generating subsequent input tokens. (pg 3 “Fig. 1: Illustration of our dreaming model.” PNG media_image1.png 328 800 media_image1.png Greyscale the model is multi-modal because it operates on two modes of input, actions and images. The model generates future image and future predicted state tokens corresponding to subsequent input tokens.) Claim 4 Piergiovanni teaches claim 3 Piergiovanni teaches, sequentially generating further subsequent image tokens and further subsequent input tokens with the neural dreaming model conditioned on previously-generated image tokens and previously-generated input tokens (pg 9 “To train the actor-critic and REINFORCE networks, for each iteration we use a batch of 256 trajectories run up to 30 steps. These trajectories are obtained from our dreaming model and was not from the real-world environment; we are able to generate as many trajectories as we want for any policy as needed…In Fig 10, we show three example sequences of the target transfers.” Also see figure 3 and 4 pg 4. The trajectories are sequentially generated steps or iterations of running/training the model to generate future image tokens which as shown in the model is conditioned on prior input and image tokens. These figures show the sequence of generated images in response to the generated tokens) Claim 5 Piergiovanni teaches claim 1 Piergiovanni teaches, receiving actual user input tokens representing actual user inputs; inputting the actual user input tokens to the neural dreaming model; (pg 5 “To train the dreaming model, we collect a dataset consisting of 40,000 images (400 random trajectories) with the various target objects in different locations in the room. We allow Fig. 5: Shared encoder with target specific decoders (TSD). the robot to take random actions in the environment and we store the starting image, action and resulting image pairs… We use this data to train our dreaming model” the dreaming model receives actual robot actions, corresponding to actual user inputs claimed. This data is input to the model via training.) and generating the subsequent image tokens based at least on the actual user inputs. (pg 3 Figure 1 caption “Fig. 1: Illustration of our dreaming model…. Future image reconstruction loss for the future image prediction” as shown subsequent or future images based on subsequent image tokens is generated further based on the above described actual user inputs.) Claim 6 Piergiovanni teaches claim 5 Piergiovanni teaches, sequentially generating further subsequent image tokens with the neural dreaming model conditioned on previously-generated image tokens and previously-received actual user input tokens. (pg 9 “To train the actor-critic and REINFORCE networks, for each iteration we use a batch of 256 trajectories run up to 30 steps. These trajectories are obtained from our dreaming model and was not from the real-world environment; we are able to generate as many trajectories as we want for any policy as needed…In Fig 10, we show three example sequences of the target transfers.” Also see figure 3 and 4 pg 4. The trajectories are sequentially generated steps or iterations of running/training the model to generate future image tokens which as shown in the model is conditioned on prior input and image tokens. These figures show the sequence of generated images in response to the generated tokens. Each token is generated sequentially with each addition step in the iterations.) Claim 7 Piergiovanni teaches claim 1 Piergiovanni teaches, wherein the neural dreaming model has been trained using token prediction loss when predicting image tokens and input tokens from the training sequences. (pg 3 caption Figure 1 “Future image reconstruction loss for the future image prediction.” Here future image reconstruction loss is the token prediction loss when predicting image tokens and input tokens.) Claim 8 Piergiovanni teaches claim 1 Piergiovanni teaches, wherein the image encoder and the image decoder have been trained using reconstruction loss from the images in the training sequences. (pg 3 caption Figure 1 “Future image reconstruction loss for the future image prediction.” Here future image reconstruction loss is the token reconstruction loss from the images in the training sequence.) Claim 9 Piergiovanni teaches claim 1 Piergiovanni teaches, displaying the subsequent images. (pg 4 Figure 4, “Fig. 4: Comparison of (top) a real trajectory and (bottom) a dreamed trajectory” The figure displays the generated dreamed subsequent images of the trajectory ) Claim 10 Piergiovanni teaches, A system comprising: a hardware processing unit; and a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the hardware processing unit to ( pg 7 “Once learned, our action policy CNN runs in real-time on a Nvidia Jetson TX2 mobile GPU.” A GPU is a hardware processing comprising the claimed memory and instructions) The remaining limitations of the claim are rejected for the reasons set forth in the rejection of claim 1. Claim 19 Piergiovanni teaches, A computer-readable storage medium storing computer-readable instructions which, when executed by a hardware processing unit, cause the hardware processing unit to perform acts comprising: ( pg 7 “Once learned, our action policy CNN runs in real-time on a Nvidia Jetson TX2 mobile GPU.” A GPU is a hardware processing comprising the claimed memory and instructions) accessing training data reflecting one or more executions of one or more applications (Section 4 Dreaming model pg 2 “Our dreaming model is a combination of a convolutional autoencoder and an action-conditioned future representation regressor. It enables encoding of an image frame (i.e., robot observation) into the representation abstracting scene (i.e., robot state)” the image from received is a seed image of an application, see figure 1 pg 3 picturing the obtained image. pg 3 Section A “We learn a state representation based on a variational autoencoder (VAE) [34] which takes an image as input and learns ) the training data including training sequences of images output by the one or more applications and inputs provided to the one or more applications during the one or more executions ( pg 5 “To train the dreaming model, we collect a dataset consisting of 40,000 images… We allow the robot to take random actions in the environment and we store the starting image, action and resulting image pairs” the model is trained with image pairs from the dataset including staring images, i.e input images to the application, and resulting images, i.e output images to the application.) mapping the images to training image tokens and the inputs to training input tokens ( Section 4 Dreaming model pg 2 “It enables encoding of an image frame (i.e., robot observation) into the representation abstracting scene (i.e., robot state), …More specifically, our dreaming model consists of the following four function components… State Encoder Encθ : I → s” pg 3 Section A “We learn a state representation based on a variational autoencoder (VAE) [34] which takes an image as input and learns a latent state representation optimized to reconstruct the given image” the Encoder, Enc, mapps the image to a seed token representation. The latent representation amounts to the token representation as described in paragraph 0052 of the specification.) training a generative model to predict the training image tokens and the training input tokens sequentially according to the training sequences; and outputting the trained generative model. (pg 3 Section A “The encoder outputs µ,σ = Enc(I) and we sample from s ∼ N(µ,σ) to obtain the state representation during the training.” The encoder, which outputs the token and which is part of the illustrated dreaming model, is inputs the token representation to subsequent layers as prompt to the neural dreaming model as indicated in Figure 1, the caption noting the components which receive the latent representation as input “ Rectangles in the figure are the CNN layers” PNG media_image1.png 328 800 media_image1.png Greyscale pg 3 “We learn a state-transition model … Our state-transition model takes the current state, st, and action, at, as input and outputs the next state, st+1. Since our state representation is convolutional, we train a CNN… Using this loss enables our learning to minimize the difference between the the future-regressed state and the true future state” the model is trained via minimization to learn the future state or training sequences obtained from the state representation of the actions, resulting in outputting the trained or learned generative model. pg 3 “Fig. 1: Illustration of our dreaming model. (a) The encoder, action representation, future regressor and decoder modules” as shown in the figure, the decoder generates subsequent image tokens which are decoded into subsequent images, i.e I_t+1) Claim 20 Piergiovanni teaches, training an image encoder/decoder to map the images in the training sequences to the training image tokens using reconstruction loss (pg 3 Figure 1 “(b) Image reconstruction loss for the autoencoder.” As shown in the figure the image reconstruction loss is to map Images from an initial image to a subsequent image via the encoder decoder, a loss function describes the training objective.) and training the generative model using next token prediction loss for the training image tokens and the training input tokens. (pg 3 caption Figure 1 “(d) Future image reconstruction loss for the future image prediction.” Here future image reconstruction loss is the token reconstruction loss from the images in the training sequence.) Claim Rejections - 35 U.S.C. § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 11-17 are rejected under 35 U.S.C. § 103 as being unpatentable over Piergiovanni further in view of Lifshitz “STEVE-1: A Generative Model for Text-to-Behavior in Minecraft” Claim 11 Piergiovanni teaches claim 10 Piergiovanni does not explicitly teach, the seed image represents output by a video game that is at least partially implemented by the generative model. Lifshitz however when addressing policy state learning conditioned on action and images teaches, the seed image represents output by a video game that is at least partially implemented by the generative model. (pg 4 “Our goal is to create a generative model of behavior in Minecraft conditioned on text instructions y. We do this by utilizing a dataset of Minecraft trajectory segments, some of which contain instruction labels… where τ is a trajectory of observations and actions…. MineCLIP is trained using a contrastive objective on pairs of Minecraft videos and transcripts from the web” here the seed image used in training is from Minecraft a video game the model includes a generative model of behavior in the game.) Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the action conditioned image generative model of Piergiovanni to comprise seed data of a video game as described by Lifshitz . One would have been motivated to make such a combination both systems are related to generative image action policy learning. Further Piergiovanni notes that “significant progress has been made in deep reinforcement learning (RL), enabling learning of control policies using raw image data from physics engines and video game environments” (pg 1 introduction). Further, Lifshitz notes “effective for creating instruction-following sequential decision-making agents… sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines” (Abstract) Claim 12 Piergiovanni teaches claim 11 Lifshitz when combined with Piergiovanni for predicting images teaches wherein the generative model has been trained to predict images output by video games and video game controller inputs that are present in the training sequences.. (pg 4 “Our goal is to create a generative model of behavior in Minecraft conditioned on text instructions y. We do this by utilizing a dataset of Minecraft trajectory segments, some of which contain instruction labels… where τ is a trajectory of observations and actions…. MineCLIP is trained using a contrastive objective on pairs of Minecraft videos and transcripts from the web” pg 5 “We gather a gameplay dataset with 54M frames… along with associated actions from two sources:… During gameplay, keypresses and mouse movements are recorded.” here the seed image used in training is from Minecraft a video game the model includes a generative model of behavior in the game. The actions are video game controller inputs, in this case mouse and keyboard) Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the action conditioned image generative model of Piergiovanni to comprise seed data of a video game as described by Lifshitz . One would have been motivated to make such a combination both systems are related to generative image action policy learning. Further Piergiovanni notes that “significant progress has been made in deep reinforcement learning (RL), enabling learning of control policies using raw image data from physics engines and video game environments” (pg 1 introduction). Further, Lifshitz notes “effective for creating instruction-following sequential decision-making agents… sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines” (Abstract) Claim 13 Piergiovanni/Lifshitz teaches claim 11 Further , Piergiovanni teaches, when executed by the hardware processing unit, cause the hardware processing unit to: map the seed image to the at least one seed image token using an image encoder. ( Section 4 Dreaming model pg 2 “It enables encoding of an image frame (i.e., robot observation) into the representation abstracting scene (i.e., robot state), …More specifically, our dreaming model consists of the following four function components… State Encoder Encθ : I → s” pg 3 Section A “We learn a state representation based on a variational autoencoder (VAE) [34] which takes an image as input and learns a latent state representation optimized to reconstruct the given image” the Encoder, Enc, mapps the image to a seed token representation. The latent representation amounts to the token representation as described in paragraph 0052 of the specification.) Claim 14 Piergiovanni/Lifshitz teaches claim 13 Piergiovanni teaches , the image encoder having been trained using reconstruction loss from the image tokens in the training sequences (Figure 1 caption “Fig. 1: Illustration of our dreaming model. (a) The encoder, action representation, future regressor and decoder modules. (b) Image reconstruction loss for the autoencoder. (c) L2 loss for the future regressor. (d) Future image reconstruction loss for the future image prediction” the dreaming model is trained based on (b) the reconstruction loss corresponding to, training the image encoder using reconstruction loss) and the generative model having been trained using token prediction loss when predicting the image tokens and the input tokens in the training sequences. (pg 3 caption Figure 1 “Future image reconstruction loss for the future image prediction.” Here future image reconstruction loss is the token prediction loss from the images in the training sequence. This loss is computed in part when predicting input tokens and image tokens) Claim 15 Piergiovanni/Lifshitz teaches claim 14 Lifshitz teaches, the input tokens obtained from the training sequences having values representing different input mechanisms of a video game controller. ( pg 3 “We leverage the MineRL environment to research the creation of agents that can follow open-ended instructions in complex visual environments using only low-level actions (mouse and keyboard). To train STEVE-1’s, we fine-tune VPT, a foundation model of Minecraft behavior that is pretrained on 70k hours of web videos of Minecraft along with estimated mouse and keyboard actions.” The actions for training include two mechanisms, mouse and keyboard, which to be incorporated into the model necessarily have values associated with them.) Claim 16 Piergiovanni/Lifshitz teaches claim 12 Piergiovanni teaches, generate future image tokens and future input tokens given past image tokens produced by the generative model and past input tokens produced by the generative model. (pg 9 “To train the actor-critic and REINFORCE networks, for each iteration we use a batch of 256 trajectories run up to 30 steps. These trajectories are obtained from our dreaming model and was not from the real-world environment; we are able to generate as many trajectories as we want for any policy as needed…In Fig 10, we show three example sequences of the target transfers.” Also see figure 3 and 4 pg 4. The trajectories are sequentially generated steps or iterations of running/training the model to generate future image tokens which as shown in the model is conditioned on prior input and image tokens. These figures show the sequence of generated images in response to the generated tokens. Each token is generated sequentially with each addition step in the iterations.) Claim 17 Piergiovanni/Lifshitz teaches claim 12 Piergiovanni teaches, generate future image tokens given past image tokens produced by the generative model and actual user inputs (pg 3 Figure 1 caption “Fig. 1: Illustration of our dreaming model” as shown in the figure future image tokens are generated from prior images and based on actual user actions a_t pg 4 figure 2 caption “Fig. 2: Illustration of how our dreaming works for the policy learning. A random start state is sampled and then actions are sampled from the policy”) Lifshitz teaches, received from a video game controller. (pg 4 “Our goal is to create a generative model of behavior in Minecraft conditioned on text instructions y. We do this by utilizing a dataset of Minecraft trajectory segments, some of which contain instruction labels… where τ is a trajectory of observations and actions…. MineCLIP is trained using a contrastive objective on pairs of Minecraft videos and transcripts from the web” pg 5 “We gather a gameplay dataset with 54M frames… along with associated actions from two sources:… During gameplay, keypresses and mouse movements are recorded.”) 07-21-aia AIA Claim (s) 18 is rejected under 35 U.S.C. § 103 as being unpatentable over Piergiovanni further in view of Ramesh “ Hierarchical Text-Conditional Image Generation with CLIP Latents” Claim 18 Piergiovanni/Lifshitz teaches claim 10 Piergiovanni does not explicitly teach, receive a natural language description of an application scenario; and generate the seed image from the natural language description using a text-to-image synthesis model. Ramesh however when addressing text and image encoding teaches, receive a natural language description of an application scenario; and generate the seed image from the natural language description using a text-to-image synthesis model. (abstract pg 1 “To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding.” An image caption as shown in the reference is text in natural language of an application scenario. Accordingly the generated image via the decoder, which makes up the text-to-image synthesis model uses the text description) Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the generative image model of Piergiovanni to an image model conditioned on text as described by Ramesh . One would have been motivated to make such a combination both systems are related to generative image learning. Further, Ramesh notes “the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion… finding that the latter are computationally more efficient and produce higher-quality samples” (abstract Ramesh) Conclusion Prior art not relied upon: Hafner et al “ Mastering Diverse Domains through World Models” describes world models of video game environments trained to predict discrete representations of a world state based on given actions and sensory inputs. Wang et al. “Deep Action Conditional Neural Network for Frame Prediction in Atari Games” describes future image frame prediction conditioned on actions in a video game Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNATHAN R GERMICK whose telephone number is (571)272-8363. The examiner can normally be reached M-F 9:30-4:30. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached on 571-272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /J.R.G./ Examiner, Art Unit 2122 /KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122 Application/Control Number: 18/374,424 Page 2 Art Unit: 2122 Application/Control Number: 18/374,424 Page 3 Art Unit: 2122 Application/Control Number: 18/374,424 Page 4 Art Unit: 2122 Application/Control Number: 18/374,424 Page 5 Art Unit: 2122 Application/Control Number: 18/374,424 Page 6 Art Unit: 2122 Application/Control Number: 18/374,424 Page 7 Art Unit: 2122 Application/Control Number: 18/374,424 Page 8 Art Unit: 2122 Application/Control Number: 18/374,424 Page 9 Art Unit: 2122 Application/Control Number: 18/374,424 Page 10 Art Unit: 2122 Application/Control Number: 18/374,424 Page 11 Art Unit: 2122 Application/Control Number: 18/374,424 Page 12 Art Unit: 2122 Application/Control Number: 18/374,424 Page 13 Art Unit: 2122
Prosecution Insights