DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“object dynamic motion module” in claims 8, 16, and 20.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Response to Arguments
Applicant’s arguments, see pages 8-9, filed 08/24/2026, with respect to the interpretation of claim(s) 6, 8, 14, 16, and 20 under 35 USC 112(f) have been fully considered and are persuasive. Therefore, the interpretation has been withdrawn. However, upon further consideration, a new ground(s) of interpretation is made in view of the amendment of 08/24/2026.
Applicant’s arguments, see pages 9-10, filed 08/24/2026, with respect to claims 4-6, 8, 12-14, 16, and 19-20 have been fully considered and are persuasive. The 35 USC 112(b) rejection of 06/29/2026 has been withdrawn.
Applicant’s arguments, see page 10, filed 08/24/2026, with respect to claims 8, 16, and 19-20 have been fully considered and are persuasive. The minor objections of 06/29/2026 have been withdrawn.
Applicant’s arguments, see pages 10-12, filed 08/24/2026, with respect to the rejection(s) of claim(s) 1-20 under 35 USC 102(a)(1) and 35 USC 103 have been fully considered and are persuasive. Therefore, the rejections have been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of the amendment of 08/24/2026, which alters the scope of the claims.
Response to Amendment
The amendment of 08/24/2026 has been accepted.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 7, 9-11, 15, and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Shah (‘MUTEX: Learning Unified Policies from Multimodal Task Specifications’), and in further view of Di Palo (‘Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics’), and in further view of Chen (‘SeqTrack: Sequence to Sequence Learning for Visual Object Tracking’).
Consider claims 1, 9, and 17, Shah discloses a system for generating robotic instructions from contextual data, the system comprising: at least one memory storing instructions; and at least one processor communicatively coupled with the at least one memory and configured to execute the instructions to perform operations comprising (6.2 Training Details, GPUs):
[claim 17: A non-transitory computer-readable media (CRM) storing instructions thereon, which, when executed by at least one processor of a computing device, cause the computing device to generate robotic instructions from contextual data by performing operations comprising (6.2 Training Details, GPUs):]
[claim 1: A method for generating robotic instructions from contextual data comprising (Abstract, 6.2 Training Details; “MUTEX, a unified approach to policy learning from multimodal task specifications”):]
receiving, at one or more processors, video data demonstrating one or more tasks (FIG. 2, 3 Video Demonstration; Section 3);
receiving, at the one or more processors, text data related to the one or more tasks (FIG. 2 Text Instructions; Section 3);
encoding, at the one or more processors, the video data and the text data, wherein the encoding is generated using at least one cross-attentional transformer (FIG. 2; “The resultant tokens obtained from projection layers are combined with observation tokens through N blocks of self- and cross-attention layers. The encoder’s hidden state is passed to MUTEX’s transformer encoder that is queried for actions (behavior learning loss) and the masked features and tokens (masked modeling loss), promoting action-specific cross-modal learning. In the second stage of training, all modalities are enriched with information from video features through a cross-modal matching loss.”; Section 3; “employing a transformer-based architecture with stacked cross- and self-attention layers”);
receiving, at the one or more processors, image data providing environmental data for at least one robotic task (FIG. 2 Robot Observations; Section 3);
encoding, at the one or more processors, vision data corresponding to the image data (FIG. 2 Observation Encoder, Policy Encoder; Section 3; 6.3 Policy Encoder; “Policy Encoder combines the robot observations with the task specification embeddings received from the modality specific encoders”); and
generating, at the one or more processors, based upon the video data and the text data that was encoded and the vision data that was encoded, robotic instructions (FIG. 2; “MUTEX predicts closed-loop actions to achieve the task based on the provided observations and a task specification modality (one or more) at test time”; Section 3; 6.3 Model Architecture; “Policy Decoder takes in the input from the policy encoder and the learnable queries, enabling MUTEX to output a variable number of tokens. These outputs are used to predict the masked token values during masked modeling and action values. The learnable query vectors indicate the type of corresponding output- {Action, Text Goal, Text Instructions, Image Goal, Video Demonstration, Speech Goal, Speech Instructions}”).
Shah fails to specifically disclose wherein the robotic instructions comprise an output sequence of tokens representing robot-executable commands, the output sequence of tokens being generated autoregressively by a decoder comprising a causal self-attention layer that attends to previous tokens in the output sequence and a cross-attention layer that attends to visual embeddings of the environmental data, and the output sequence of tokens including one or more N-degree-of-freedom (N-DoF) pose tokens representing a position and an orientation of an end-effector of a robot.
In related art, Di Palo discloses wherein the robotic instructions comprise an output sequence of tokens representing robot-executable commands, the output sequence of tokens being generated autoregressively by a decoder, and the output sequence of tokens including one or more N-degree-of-freedom (N-DoF) pose tokens representing a position and an orientation of an end-effector of a robot (Di Palo FIG. 4, III. Method; “Let O represent the sequence of observations received by the robot and A the sequence of end-effector actions - in our case a trajectory of SE(3) poses TW expressed in the world frame W… autoregressively compute a new sequence of tokens… The action tokens, i.e. the triplets eW, will then be transformed into SE(3) poses of the end-effector and executed sequentially… The robot will then execute the sequence of predicted action tokens … transformed into a trajectory of SE(3) poses.”; F. Action: Investigations on Action Tokens; What is the optimal representation of poses to use as action tokens?).
Shah discloses a unified policy capable of executing tasks specified as goal descriptions or more detailed instructions in text, speech, or visual modalities. Results indicate that “MUTEX can effectively interpret human video demonstrations and perform tasks successfully with the robot” (Shah Introduction). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the action token representation of Di Palo into the action output of Shah to directly generate robot trajectories from visual observations (Di Palo FIG. 4, III. Method; Shah 6.4 Multimodal Dataset Details).
In related art, Chen discloses a decoder comprising a causal self-attention layer that attends to previous tokens in the output sequence and a cross-attention layer that attends to visual embeddings of the environmental data (Chen FIG. 2; “The causal decoder autoregressively generates the sequence of the bounding box tokens using the extracted features. (b) The detailed transformer block in the causal decoder. The input embeddings interact in a casual manner through a masked multi-head attention. The visual feature is incorporated into the decoder through a multi-head attention layer.”; 3. Method; “A causal attention mask is imposed on the self-attention modules in the decoder to restrict the tokens to only attend to their proceeding tokens.”).
Shah discloses a policy decoder (Shah FIG. 2) and Di Palo discloses “autoregressively comput[ing] a new sequence of tokens (Di Palo III. Method).” Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the decoder architecture of Chen into the decoder of Shah, as modified by Di Palo, to ensure that the output of each sequence element only depends on its previous sequence elements (Chen 3.3 Model Architecture, Decoder).
Consider claims 2, 10, and 18, Shah, as modified by Di Palo and Chen, discloses the claimed invention further comprising:
splitting each frame of the video data into a predetermined number of patches (Shah 3.1, [59] SimMIM; 6.2; Masked Modelling; 6.3 CLIP Large ViT-L/14; Chen 3.2. Image and Sequence Representation);
flattening each of the predetermined number of patches into a vector (Chen 3.2. Image and Sequence Representation);
projecting each vector into a linear projection (Chen 3.2. Image and Sequence Representation); and
executing a multi-head attention module based upon the linear projections of each of the predetermined number of patches (Shah 6.3 Model Architecture; CLIP Large ViT-L/14 model; Chen 3.2. Image and Sequence Representation).
Consider claims 3 and 11, Shah, as modified by Di Palo and Chen, discloses the claimed invention further comprising determining a plurality of portions of an input sequence that includes a plurality of temporal dependencies (Chen 4.3, Temporal Sequence; Di Palo III. Method).
Consider claims 7 and 15, Shah, as modified by Di Palo and Chen, discloses the claimed invention wherein the patches are non-overlapping and of a same size (Chen 3.2. Image and Sequence Representation, 4.1. Implementation Details).
Claims 4 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Shah, in further view of Di Palo and Chen, as applied to claims 1-3, 7, 9-11, 15, and 17-18 above, and further in view of Dosovitskiy (‘An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale’).
Consider claims 4 and 12, Shah, while referencing using BERT to “mask out words (Shah 3.1),” fails to explicitly disclose wherein the encoding of the text data includes text embeddings from input text using a Bidirectional Encoder Representations from Transformers (BERT) encoder.
In related art, Dosovitskiy discloses wherein the encoding of the text data includes text embeddings from input text using Bidirectional Encoder Representations from Transformers (BERT) encoder. (Dosovitskiy (Devlin) FIG. 7; 2, 4.1 Model Variants, 4.6; B.1.2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the explicit use of BERT of Dosovitskiy in text encoding of Shah, as modified by Di Palo and Chen, to “mask out words [58] that are then passed through pretrained CLIP language model [15] to extract the features (Shah 3.1).”
Claims 5 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Shah, in further view of Di Palo and Chen, as applied to claims 1-3, 7, 9-11, 15, and 17-18 above, and further in view of Steyer (‘Grid-Based Object Tracking With Nonlinear Dynamic State and Shape Estimation’).
Consider claims 5 and 13, Shah, as modified by Di Palo and Chen, discloses the claimed invention wherein the encoding of the video data and the text data creates a representation of dynamic motion of an object, the dynamic motion including (Di Palo III. Method; Chen 3.5 Prior Knowledge Integration, 4.3 Temporal Sequence).
In related art, Steyer discloses the dynamic motion including at least one of velocity, acceleration and direction of the object across a sequence of video frames (Steyer IV. Dynamic State Estimation of Object Tracks).
Chen discloses constructing “a temporal sequence containing the coordinates of the target object in four historical frames. … In this way, the model perceives the historically moving trajectory of the target object.” Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate dynamic motion including velocity and acceleration of Steyer into the robotic instruction-sequence generation of Shah, as modified by Di Palo and Chen, to track the robot using predicted trajectories (Di Palo FIG. 4, III. Method; Shah 6.4 Multimodal Dataset Details; Steyer IV.).
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Shah, in further view of Di Palo, Chen, and Steyer, as applied to claims 5 and 13 above, and further in view of Ye (‘Cross-Modal Self-Attention Network for Referring Image Segmentation’).
Consider claims 6 and 14, Shah, as modified by Di Palo and Chen, fails to specifically disclose contextualizing the text data using the at least one cross-attentional transformer attending to the video data and outputting features; and
contextualizing the features using a transformer block with self-attention.
In related art, Ye discloses contextualizing the text data using the at least one cross-attentional transformer attending to the video data and outputting features (Ye 3. Our Model); and
contextualizing the features using a transformer block with self-attention (Ye 3. Our Model).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the self-attention contextualization of Ye into the instruction generation method of Shah, as modified by Di Palo, Chen, and Steyer, to understand the relationships between language and spatial feature representations, to achieve task specifications across multiple modalities (Shah 5 Conclusion).
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Shah, in further view of Di Palo and Chen, as applied to claims 1-3, 7, 9-11, 15, and 17-18 above, and further in view of Zhou (‘TransVOD: End-to-End Video Object Detection With Spatial-Temporal Transformers).
Consider claims 8 and 16, Shah, as modified by Di Palo and Chen, fails to specifically disclose generating by an object dynamic motion module, a fused object representation from an object bounding box representation and a set of learned object queries.
In related art, Zhou discloses generating by an object dynamic motion module, a fused object representation from an object bounding box representation and a set of learned object queries (Zhou Introduction, 3. Method).
Chen teaches, “The encoder is to extract visual features of video frames, while the decoder is to generate the sequence of bounding box values using the extracted features (Chen Introduction).” Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the fused object representation of Zhou into the instruction generation method of Shah, as modified by Di Palo and Chen, to provide context for the objects involved in the task.
Claim 19 consists of a combination of the limitations of above rejected claims 3 and 11, 4 and 12, 5 and 13, and 6 and 14, and so is similarly rejected in view of Shah, in view of Di Palo and Chen, and in further view of Steyer, and in further view of Ye. It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the referenced features of Shah, Di Palo, Chen, Dosovitskiy, Stayer, and Ye to predictably yield the claimed system.
Claim 20 is a combination of the limitations of above rejected claims 7 and 15, and 8 and 16, and so is similarly rejected in view of Shah, in view of Di Palo and Chen, and in further view of Zhou. It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the referenced features of Shah, Di Palo, Chen, and Zhou to predictably yield the claimed system.
Relevant Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Brohan (‘RT-2: Vision-language-action models transfer web knowledge to robotic control’).
Jain (‘Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers’).
Zhang (‘Autoregressive Action Sequence Learning for Robotic Manipulation’).
Herzig (‘Object-region video transformers’).
WO2024163919A1 discloses method for controlling robots using multi-modal language models.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ASHLEY HYTREK whose telephone number is (703)756-4562. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Steve Koziol can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ASHLEY HYTREK/Examiner, Art Unit 2665
/Stephen R Koziol/Supervisory Patent Examiner, Art Unit 2665
