DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-7, 9, 11-13, 15-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Chi et al (Diffusion Policy: Visuomotor Policy Learning via Action Diffusion published 2023-03-07).
Regarding Claim 1, Chi teaches:
a method implemented using one or more processors (see at least " In our real-world experiments, using DDIM with 100 training iterations and 10 inference iterations enables 0.1s inference latency on a Nvidia 3080 GPU" on page 4 section 3.4) and comprising:
retrieving a plurality of images that capture an environment in which a robot operates from multiple different perspectives (see at least "The visual encoder maps the raw image sequence into a latent embedding Ot and is trained end-to-end with the diffusion policy. Different camera views use separate encoders, and images in each timestep are encoded independently and then concatenated to form Ot." On page 4 section 3.2 ) ;
processing data indicative of the plurality of images and a proprioceptive state of the robot using a transformer-encoder to generate latent embeddings representing the plurality of images and proprioceptive state of the robot; (see at least latent embeddings on page 4 section 3.2 and " The proprioceptive observation space is extended to include the poses of both end-effectors and the gripper widths of both grippers. We also extend the observation space to include the actual and desired values of these quantities. The image observation space is comprised of two scene cameras and two wrist cameras, one attached to each arm. The action space is extended to include the desired poses of both end effectors and the desired gripper widths of both grippers." On page 11 section 7.1)
processing the latent embeddings and data indicative of a diffusion timestep using a transformer-decoder to generate robot control data, wherein the robot control data comprises a series of actions to be performed by the robot over a time interval (see at least " Actions with noise Ak t are passed in as input tokens for the transformer decoder blocks, with the sinusoidal embedding for diffusion iteration k prepended as the first token. The observation Ot is transformed into observation embedding sequence by a shared MLP, which is then passed into the transformer decoder stack as input features." On page 4 section 3.1 ) ;
causing the robot to be operated in accordance with the robot control data. (see at least " We evaluated Diffusion Policy in the realworld performance on 4 tasks across 2 hardware setups– with training data from different demonstrators for each setup. On the realworld Push-T task, we perform ablations examining Diffusion Policy on 2 architecture options and 3 visual encoder options; we also benchmarked against 2 baseline methods with both position-control and velocity-control action spaces." On page 9 section 6 )
Regarding Claim 2, Chi teaches:
The method of claim 1, wherein the series of actions comprise a series of absolute joint positions of a plurality of joints of the robot. (see at least " For the non-haptic control, a custom mid-level controller is implemented to generate desired joint positions from desired end effector poses from the learned policies. At each time step, we solve a differential kinematics problem (formulated as a Quadratic Program) to compute the desired joint velocity to track the desired end effector velocity. The resulting joint velocity is Euler integrated into joint position, which is tracked by a joint-level controller on the robot." on page 18 section D.1 )
Regarding Claim 3, Chi teaches:
The method of claim 2,
wherein the series of actions further comprise a series of gripper positions for two or more grippers. (see at least "For the non-haptic control, a custom mid-level controller is implemented to generate desired joint positions from desired end effector poses from the learned policies. At each time step, we solve a differential kinematics problem (formulated as a Quadratic Program) to compute the desired joint velocity to track the desired end effector velocity. The resulting joint velocity is Euler integrated into joint position, which is tracked by a joint-level controller on the robot." and bimanual tasks on page 18 section D.1 )
Regarding Claim 4, Chi teaches:
The method of claim 3,
wherein the series of gripper positions are continuous (see at least "We combine the policy’s capability to predict high-dimensional action sequences with receding-horizon control to achieve robust execution. This design allows the policy to continuously re-plan its action in a closed-loop manner while maintaining temporal action consistency achieving a balance between long-horizon planning and responsiveness." On page 2 section 1) .
Regarding Claim 5, Chi teaches:
The method of claim 1, wherein the series of actions comprise:
joint commands and/or torque commands; (see at least "At each time step, we solve a differential kinematics problem (formulated as a Quadratic Program) to compute the desired joint velocity to track the desired end effector velocity. The resulting joint velocity is Euler integrated into joint position, which is tracked by a joint-level controller on the robot." on page 18 )
Cartesian commands for an end effector of the robot;
a target robot pose; or (see at least "Our UR5 based experiment setup is shown in Fig 6. Diffusion Policy predicts robot commands at 10 Hz and these commands then linearly interpolated to 125 Hz for robot execution." On page 9 section 6.1 )
code specifying reward functions for motion controller optimization; or
selected predefined robot primitives.
Regarding Claim 6, Chi teaches:
the method of claim 1,
wherein the transformer-encoder and transformer-decoder form a diffusion policy. (see at least diffusion transformer encoder on page 8 section 5.4 and diffusion transformer decoder in Fig. 2 and page 4 section 3.1 )
Regarding Claim 7, Chi teaches:
the method of claim 1, further comprising
processing each of the plurality of images using a respective convolutional neural network to generate feature maps. (see at least "The visual encoder maps the raw image sequence into a latent embedding Ot and is trained end-to-end with the diffusion policy. Different camera views use separate encoders, and images in each timestep are encoded independently and then concatenated to form Ot. We used a standard ResNet-18 (without pretraining) as the encoder with the following modifications:" and CNN backbone models on page 4 section 3.2 )
Regarding Claim 9, Chi teaches:
the method of claim 1,
wherein the transformer-decoder comprises a diffusion denoiser. (see denoiser on page 6 section 4.5)
Regarding Claim 11, Chi teaches:
the method of claim 1,
wherein the robot is a simulated robot or a real robot. (see evaluating the policy in both simulated and real world environments on page 2 section 1)
Regarding Claim 12, Chi teaches:
the method of claim 1,
wherein one or both of the transformer-encoder and transformer decoder are trained using training data collected using imitation learning. (see imitation learning and demonstration data used for training on page 6 in section 5.1)
Regarding Claim 13, Chi teaches:
the method of claim 12,
wherein the imitation learning comprises teleoperation of one or more robots using a puppeteering interface. (see teleoperated demonstration data used for training on page 6 in section 5.1 and teleoperation using hand-controllers and VR headset on page 11 section 7.2 and page 18 section D)
Regarding Claim 15, Chi teaches:
the method of claim 13,
wherein the imitation learning comprises one or more of the following tasks:
folding a shirt; (see at least bimanual shirt folding on page 12 section 7.5 )
hanging a shirt on a hanger;
shoelace tying;
robot finger placement;
gear insertion; or
stacking random collections of dishware.
Regarding Claim 16, Chi teaches:
the method of claim 1,
wherein at least the transformer-decoder is trained with a diffusion loss. (see end to end training using diffusion loss functions on page 3 sections 2.2 and 2.3 )
Regarding Claim 17, Chi teaches:
the method of claim 16,
wherein both the transformer-encoder and transformer-decoder are trained with diffusion loss. (see end to end training using diffusion loss functions on page 3 sections 2.2 and 2.3 )
Regarding Claim 18, Chi teaches:
a method implemented using one or more processors (see at least " In our real-world experiments, using DDIM with 100 training iterations and 10 inference iterations enables 0.1s inference latency on a Nvidia 3080 GPU" on page 4 section 3.4) and comprising:
retrieving a plurality of images that capture, from multiple different perspectives, an environment in which a robot was operated to perform a sequence of actions (see at least "The visual encoder maps the raw image sequence into a latent embedding Ot and is trained end-to-end with the diffusion policy. Different camera views use separate encoders, and images in each timestep are encoded independently and then concatenated to form Ot." On page 4 section 3.2 );
processing data indicative of the plurality of images and a proprioceptive state of the robot using a transformer-encoder to generate latent embeddings representing the plurality of images and proprioceptive state of the robot; (see at least latent embeddings on page 4 section 3.2 and " The proprioceptive observation space is extended to include the poses of both end-effectors and the gripper widths of both grippers. We also extend the observation space to include the actual and desired values of these quantities. The image observation space is comprised of two scene cameras and two wrist cameras, one attached to each arm. The action space is extended to include the desired poses of both end effectors and the desired gripper widths of both grippers." On page 11 section 7.1)
adding noise to the sequence of actions performed by the robot to generate a plurality of noisy actions; (see Gaussian noise being added on page 2 section 2.1 and page 3 section 2.2 and page 4 section 4.1)
processing the latent embeddings and the plurality of noisy actions using a diffusion-based transformer decoder to predict noise values; (see denoising process including predicting noise values and minimizing the loss as a function of the predicted noise in page 2-3 section 2.1-2.2 and diffusion decoders in Fig. 2 )
based on the predicted noise values, training the diffusion-based transformer decoder. (see training DDPM to minimize loss based on predicted noise values on page 3 section 2.2)
Regarding Claim 19, Chi teaches:
the method of claim 18,
wherein predicted actions are determined using the predicted noise values, and the diffusion-based transformer-decoder is trained based on a comparison of the predicted actions and the sequence of actions performed by the robot. (see closed-loop action-sequence prediction by the diffusion model and visual observation conditioning on page 3 section 2.3 and Fig. 2)
Regarding Claim 20, Chi also teaches:
a system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors (see system hardware on page 4 section 3.4 and Table 6 on page 9) , cause the one or more processors to:
implement the method of Claim 1 (see Claim 1 analysis for rejection of the method)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chi et al (Diffusion Policy: Visuomotor Policy Learning via Action Diffusion published 2023-03-07) in view of Hester et al (US 11921824, hereinafter Hester.
Regarding Claim 8, Chi teaches:
the method of claim 7, further comprising
Chi does not appear to explicitly teach all of the following, but Hester does teach
flattening the feature maps into a sequence of tokens that comprise the data indicative of the plurality of images that is processed using the transformer encoder. (see at least " In various examples, the RGB 106a sensor data, the RGB 106b sensor data, the RGB 106c sensor data, and the RGB 106d sensor data may be input into a backbone network to generate a plurality of tokens representing the various frames of 2D image data. In the example depicted in FIG. 1B, sensor data from each respective channel (e.g., from each image sensor) is sent to a respective backbone network (a ResNet-18 backbone in the example). Left RGB 106a sensor data may be sent to ResNet-18 backbone 108a. ResNet-18 backbone 108a may output a plurality of tokens that represent the input frame (e.g., the left RGB 106a sensor data). The plurality of tokens may comprise feature data representing the image. Additionally, the ResNet-18 backbone 108a may generate position embeddings representing each token. The position embeddings may represent the portion of the 2D frame of image data that a particular token represents.” in col. 7 lines 40-55)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Chi to incorporate the teachings of Hester wherein feature data from sensor data is collapsed into tokens for input to a transformer encoder. The motivation to incorporate the teachings of Hester would be to extract features from the raw sensor data in a format that can be used as input from multiple sensors being fused by a transformer encoder, which makes the system robust against calibration errors (see col. 5 lines 20-28)
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chi et al (Diffusion Policy: Visuomotor Policy Learning via Action Diffusion published 2023-03-07) in view of Ji et al (US 20240169682, hereinafter Ji).
Regarding Claim 10, Chi teaches:
the method of claim 1,
Chi does not appear to explicitly teach all of the following, but Ji does teach:
wherein the diffusion timestep is represented as a one-hot vector. (see at least " In some embodiments, the transformer encoder 402 takes as input a sequence of linear projections 404 having size (M+1), where the first element (denoted “0” in FIG. 4) indicates a diffusion time step t represented as a one-hot encoded embedding of size 1×T (where T is, again, the maximum diffusion time step), and the other elements (denoted “1” to “M” in FIG. 4) are respectively M landmarks (e.g., 3D face landmarks) obtained at the respective diffusion time step t. " in par. 0064)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Chi to incorporate the teachings of Ji wherein the diffusion time step is one-hot encoded and input into the diffusion model to represent where the model is in the denoising iterations. The motivation to incorporate the teachings of Ji would be to improve the denoising process (see par. 0007)
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chi et al (Diffusion Policy: Visuomotor Policy Learning via Action Diffusion published 2023-03-07) in view of Li et al (US 20250249595, hereinafter Li).
Regarding Claim 14, Chi teaches:
the method of claim 13,
Chi does not appear to explicitly teach all of the following, but Li does teach:
wherein the puppeteering interface comprises two leader arms of a first size that are synchronized with two follower arms of a second size that is greater than the first size. (see at least “As shown, a leader console 260 (e.g., a machine with a robotic arm controlled by a human 265) performs a task, which is followed by a follower console or station 270. The leader module 260 demonstrates a series of tasks, movements, or trials for a particular manipulation of a non-rigid material.” In par. 0040 and "The human 265 may control the leader console 260 by directly handing the robotic arms of the machine. For example, the human may perform different movements by directly manipulating the robotic arms in 3D space." in par. 0041 and Fig. 2B)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method taught by Chi to incorporate the teachings of Li wherein demonstration data for a bimanual task is collected using a leader console with two smaller versions of the robotic arms that are directly manipulated by a human while training data is recorded. The motivation to incorporate the teachings of Li would be to improve the training and subsequent performance of a generative ML model trained to control a robot to manipulate non-rigid materials (see par. 0014)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN M KATZ whose telephone number is (571)272-2776. The examiner can normally be reached Mon-Thurs. 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN M KATZ/Primary Examiner, Art Unit 3657
Prosecution Insights