Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/16/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement has been considered by the Examiner.
Claim Objections
Claim 8 is objected to because of the following informalities: claim 8 is objected to for the limitation “an additional sensor observation from one or more sensors in the environment” while claim 6 on which it depends recites “one or more image sensors” making it unclear if the “one or more” sensors are intended to be the same sensors or different sensors. For example, it is unclear if the additional observations are from the same image sensors, or from distinct sensors in the environment. Appropriate correction is required, with the examiner suggesting amendments to further clarify the sensor types.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 19 is rejected under 35 U.S.C. § 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because the broadest reasonable interpretation of the “one or more storage devices communicatively couped to the one or more computers,” encompasses signals per se. See MPEP §2106.03.11 (“Non-limiting examples of claims that are not directed to any of the statutory categories include: ... Transitory forms of signal transmission (often referred to as "signals per se"), such as a propagating electrical or electromagnetic signal or carrier wave”). Claim 19 in particular recites “one or more storage devices communicatively couped to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for controlling an agent acting in an environment to perform a task of a plurality of possible tasks, the operations comprising” Paragraph [0156] of the specification states that “Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium” which may comprise transitory signals as it is not a requirement to use the tangible non-transitory storage medium, and therefore is directed towards signals per se. In order to overcome this rejection, Examiner recommends that Applicant amends claim 19 to disclose “one or more non-transitory storage mediums ”.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-7, 9-13, 15-16, and 19-21 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Paxton et al. (US Pre-Granted Publication No. US 2023/0297074 A1 hereinafter “Paxton”).
Regarding claim 1 Paxton discloses:
A computer-implemented method of controlling an agent acting in an environment to perform a task of a plurality of possible tasks, the method comprising: (Paxton [0020] wherein the robot completes a variety of tasks or goals) obtaining a task description, wherein the task description identifies the task to be performed; and, at each of a plurality of sub-task execution time steps: (Paxton [0020] [0030] [0048] [0138] wherein the robot follows tasks as laid out through learned elements and long term instructions as a sequence determination) processing the task description and an observation characterizing a state of the environment at the time step, (Paxton [0028] wherein task descriptions are based on imaging data and identified cues in the environment) using a dispatcher neural network system, to generate an executor instruction, wherein the executor instruction comprises a set of tokens that encodes a representation of aspects of the environment relevant to the sub-task; (Paxton [0032-0033] wherein the system uses a neural network to identify image features for symbols or tokens to indicate how to operate the robot) processing the executor instruction, using an executor neural network system, to generate an action selection output for performing a set of one or more sub-task actions for executing a sub-task of the task; (Paxton [0038-0039] [0033] wherein the neural network determines how to interpret images, and complete a task) and controlling the agent using actions selected according to the action selection output to perform the set of one or more sub-task actions for executing the sub-task. (Paxton [0033] [0039] [0049] wherein the robot is controlled to perform the task).
Regarding claim 2 Paxton discloses all of the limitations of claim 1 and Paxton further discloses:
The method of claim 1, wherein processing the task description and the observation characterizing the state of the environment at the time step, using the dispatcher neural network system, (Paxton [0028] [0032-0033] wherein the system uses a neural network to identify image features for symbols or tokens to indicate how to operate the robot) comprises dividing the task into sub-tasks using the dispatcher neural network system (Paxton [0038-0039] [0033] [0025] [0048-0049] wherein the tasks can be split into sequences for the neural network to operate) and generating the executor instruction by selecting, from the observation characterizing the state of the environment at the time step, (Paxton [0049] wherein the system neural network can break images into sequences and determine the environment to operate) a subset of information from the observation relevant to the sub-task being executed at the time step; (Paxton [0049] wherein the first step of the sequence is analyzed and performed) and wherein processing the executor instruction, using the executor neural network system, to generate the action selection output comprises processing the subset of information from the observation, (Paxton [0032-0033] wherein the system uses a neural network to identify image features for symbols or tokens to indicate how to operate the robot) relevant to the sub-task being executed at the time step, to generate the action selection output for performing the set of one or more sub-task actions. (Paxton [0033] [0039] [0049] wherein the robot is controlled to perform the task).
Regarding claim 3 Paxton discloses all of the limitations of claim 1 and Paxton further discloses:
The method of claim 1, wherein processing the task description and the observation characterizing the state of the environment at the time step, using the dispatcher neural network system, to generate the executor instruction comprises (Paxton [0028] [0032-0033] wherein the system uses a neural network to identify image features for symbols or tokens to indicate how to operate the robot) generating the set of tokens such that each token represents one or more aspects of an entity in the environment that is relevant to performing the task. (Paxton [0033] [0035] wherein tokens for specifying the robot action are created and performed by the robot).
Regarding claim 4 Paxton discloses all of the limitations of claim 3 and Paxton further discloses:
The method of claim 3, wherein generating the set of tokens comprises generating one or more respective tokens to represent the one or more aspects of each respective entity in the environment, (Paxton [0033] [0035] wherein tokens for specifying the robot action are created and performed by the robot) wherein different tokens represent different respective entities in the environment. (Paxton [0031-0033] wherein the symbols or tokens can be entities or steps of the plan).
Regarding claim 5 Paxton discloses all of the limitations of claim 4 and Paxton further discloses:
The method of claim 4, wherein generating respective tokens representing different respective entities in the environment comprises processing the observation characterizing the state of the environment at the time step and the task description, using the dispatcher neural network system, to select a subset of the entities in the environment based on the task description, for use in performing the sub-task. (Paxton [0031-0032] [0049] wherein the images of the environment with identified features are used with the neural network to perform robot operations).
Regarding claim 6 Paxton discloses all of the limitations of claim 4 and Paxton further discloses:
The method of claim 4, wherein the observation characterizing the state of the environment at the time step comprises an image observation from one or more image sensors; (Paxton [0032] wherein multiple cameras can be used by the system) and wherein generating respective tokens representing different respective entities in the environment comprises: processing the task description and pixels of the image observation at the time step, (Paxton [0031-0032] [0141] wherein image inferences can include pixel level segmentation) using the dispatcher neural network system, to select one or more objects in the image observation, based on the task description, for use in performing the sub-task; (Paxton [0028] [0031-0032] wherein the image data is used to identify objects and cues to operate the robot) and generating the set of tokens such that each of the one of the one or more objects is characterized by one or more of the tokens. (Paxton [0033] [0035] wherein tokens for specifying the robot action are created and performed by the robot).
Regarding claim 7 Paxton discloses all of the limitations of claim 6 and Paxton further discloses:
The method of claim 6, wherein a location of each of the one or more objects is characterized by a respective token. (Paxton [0032-0033] wherein objects can be identified by symbols or tokens).
Regarding claim 9 Paxton discloses all of the limitations of claim 1 and Paxton further discloses:
The method of claim 1, wherein generating the set of tokens comprises selecting each token from an observation description language comprising tokens that describe one or more of: objects represented by the observation, (Paxton [0032-0033] wherein the objects are specified by machine language, including aspects such as a color and location) aspects of objects represented by the observation, and a scene represented by the observation. (Paxton [0032-0033] wherein the objects are specified by machine language, including aspects such as a color and location in the scene i.e. environment).
Examiner notes that due to the “one or more of” language, only one of the observation, aspect of the object, and scene are needed to fully teach the limitation.
Regarding claim 10 Paxton discloses all of the limitations of claim 1 and Paxton further discloses:
The method of claim 1, wherein at least some tokens of the set of tokens encode a learned representation of the aspects of the environment relevant to the sub-task. (Paxton [0032-0033] wherein the objects are specified by machine language, including aspects such as a color and location in the scene i.e. environment).
Regarding claim 11 Paxton discloses all of the limitations of claim 1 and Paxton further discloses:
The method of claim 1, wherein the set of tokens includes an executor identifier token to identify one of a plurality of the executor neural network systems; (Paxton [0032-0033] [0035] [0130] wherein the tokens identify objects to be used by one or more neural networks, with the ability to use various network pipelines to train and then implement the robot operations) the method further comprising: processing the task description and the observation characterizing the state of the environment at the time step, using the dispatcher neural network system, to generate the executor instruction including the executor identifier token; (Paxton [0128] [0130] [0134] wherein the multiple training systems with the neural network can be selected to operate the system) and processing the executor instruction using the identified executor neural network system to generate the action selection output. (Paxton [0130-0133] [0138] wherein the system uses the network to train and perform operations of the robotic system).
Regarding claim 12 Paxton discloses all of the limitations of claim 1 and Paxton further discloses:
The method of claim 1, wherein the task description comprises a text sequence that characterizes the task to be performed by the agent in the environment; (Paxton [0031] [0141] wherein the system can create and use text sequences to determine the robot actions) the method further comprising: generating an encoded representation of the text sequence; (Paxton [0031] wherein the text information can be encoded) wherein processing the task description and the observation characterizing the state of the environment at the time step, using the dispatcher neural network system comprises processing the observation characterizing the state of the environment conditioned on the encoded representation of the text sequence to generate the executor instruction comprising the set of tokens; (Paxton [0031-0033] wherein the data from the images or audio instructions to operate the system can be encoded into the neural network and encoded with the images) and wherein the set of tokens comprises a sequence of tokens each representing a different respective aspect of the observation characterizing the state of the environment. (Paxton [0031-0033] wherein the symbols or tokens can be entities or steps of the plan).
Regarding claim 13 Paxton discloses all of the limitations of claim 1 and Paxton further discloses:
The method of claim 1, wherein the task description comprises a text sequence that characterizes the task to be performed by the agent in the environment, (Paxton [0031] [0141] wherein the system can create and use text sequences to determine the robot actions) wherein dispatcher neural network system comprises a transformer-based multimodal machine learning model having a first modality input to receive the text sequence (Paxton [0031] wherein the text instructions are used by the neural network to determine robot operations) and a second modality input to receive the observation characterizing the state of the environment at the time step, and the method further comprising: (Paxton [0032] wherein the image observations alongside the text are used to identify objects and features to operate the system) jointly processing, using the transformer-based multimodal machine learning model, an encoded version of the text sequence and an encoded version of the observation to generate the executor instruction. (Paxton [0031-0033] wherein the data from the images or audio instructions to operate the system can be encoded into the neural network and encoded with the images).
Regarding claim 15 Paxton discloses all of the limitations of claim 1 and Paxton further discloses:
The method of claim 1, further comprising: training at least the executor neural network system to perform one or more sub-tasks of the plurality of tasks using demonstration data from one or more demonstration agents trained to perform a particular example of the one or more sub-tasks, by: (Paxton [0024] [0111-0115] wherein the system can use training information from different models or different facilities to learn how to operate a certain task) training at least the executor neural network system to perform the one or more sub-tasks using an imitation learning technique based on training data comprising the demonstration data characterizing interactions of the one or more demonstration agents performing the particular example of the one or more sub-tasks in a corresponding environment to the environment of the agent. (Paxton [0024] [0111-0115] [0120] wherein the system can use training information from different models or different facilities to learn how to operate a certain task, including training at a first facility and being implemented at a second facility).
Regarding claim 16 Paxton discloses all of the limitations of claim 1 and Paxton further discloses:
The method of claim 1, wherein the environment is a real-world environment, (Paxton [0039] [0130] wherein the robot is operated in a real world environment with real world data) the observations comprise observations from one or more sensors in the real-world environment, (Paxton [0032] wherein the system includes cameras or other sensors to input environment information) the agent comprises a machine operating in the real-world environment to perform the task, and the sub-task actions are actions of the machine in the real-world environment. (Paxton [0032-0034] wherein the information is used by a real robot operating in the environment).
Regarding claim 19 Paxton discloses:
A system comprising: one or more computers; (Paxton [0020] wherein the robot completes a variety of tasks or goals) and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations (Paxton [0058-0059] wherein the system includes data storage to operate the robot) for controlling an agent acting in an environment to perform a task of a plurality of possible tasks, the operations comprising: (Paxton [0020] wherein the robot completes a variety of tasks or goals) obtaining a task description, wherein the task description identifies the task to be performed; and, at each of a plurality of sub-task execution time steps: (Paxton [0020] [0030] [0048] [0138] wherein the robot follows tasks as laid out through learned elements and long term instructions as a sequence determination) processing the task description and an observation characterizing a state of the environment at the time step, (Paxton [0028] wherein task descriptions are based on imaging data and identified cues in the environment) using a dispatcher neural network system, to generate an executor instruction, wherein the executor instruction comprises a set of tokens that encodes a representation of aspects of the environment relevant to the sub-task; (Paxton [0032-0033] wherein the system uses a neural network to identify image features for symbols or tokens to indicate how to operate the robot) processing the executor instruction, using an executor neural network system, to generate an action selection output for performing a set of one or more sub-task actions for executing a sub-task of the task; (Paxton [0038-0039] [0033] wherein the neural network determines how to interpret images, and complete a task) and controlling the agent using actions selected according to the action selection output to perform the set of one or more sub-task actions for executing the sub-task. (Paxton [0033] [0039] [0049] wherein the robot is controlled to perform the task).
Regarding claim 20 Paxton discloses:
A computer-implemented machine control system, for controlling a machine acting in a real-world environment to perform a task of a plurality of possible tasks, (Paxton [0058-0059] wherein the system includes data storage to operate the robot) the system comprising: a task description input to receive a task description, wherein the task description identifies the task to be performed; (Paxton [0020] [0030] [0048] [0138] wherein the robot follows tasks as laid out through learned elements and long term instructions as a sequence determination) a sensor input to receive, from one or more sensors in the real-world environment, an observation characterizing a state of the environment; (Paxton [0032] wherein multiple cameras can be used by the system in the environment) and wherein the system is configured to, at each of a plurality of time steps: process the task description and an observation characterizing a state of the environment at the time step, (Paxton [0028] wherein task descriptions are based on imaging data and identified cues in the environment) using a dispatcher neural network system, to generate an executor instruction, wherein the executor instruction comprises a set of tokens that encodes a representation of relevant aspects of the environment to the task; (Paxton [0032-0033] wherein the system uses a neural network to identify image features for symbols or tokens to indicate how to operate the robot) process the executor instruction, using an executor neural network system, to generate an action selection output for performing a set of one or more sub-task actions for executing a sub-task of the task; (Paxton [0038-0039] [0033] wherein the neural network determines how to interpret images, and complete a task) and generate, using the action selection output, a control output for controlling the machine to perform the set of one or more sub-task actions for executing the sub-task. (Paxton [0033] [0039] [0049] wherein the robot is controlled to perform the task).
Regarding claim 21 Paxton discloses all of the limitations of claim 20 and Paxton further discloses:
The system of claim 20, wherein the one or more sensors comprise an image sensor, (Paxton [0032] wherein multiple cameras can be used by the system) wherein the observation characterizing the state of the environment at the sub-task execution time step comprises an image observation from image sensor; (Paxton [0032] wherein multiple cameras can be used by the system) and wherein generating respective tokens representing different respective entities in the environment comprises: processing the task description and pixels of the image observation of the environment at the sub-task execution time step, (Paxton [0031-0032] [0141] wherein image inferences can include pixel level segmentation) using the dispatcher neural network system, to select one or more objects in the image observation, based on the task description, for use in performing the task; (Paxton [0028] [0031-0032] wherein the image data is used to identify objects and cues to operate the robot) and generating the set of tokens such that each token characterizes one of the one or more objects. (Paxton [0033] [0035] wherein tokens for specifying the robot action are created and performed by the robot).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 8 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Paxton in view of Kranski et al. (US Pre-Granted Publication No. US 2022/0314434 A1 hereinafter “Kranski”).
Regarding claim 8 Paxton discloses all of the limitations of claim 6 and Paxton further discloses:
The method of claim 6, wherein the observation characterizing the state of the environment at the time step includes an additional sensor observation from one or more sensors in the environment; (Paxton [0032] wherein sensors from the environment can view the region or environment) …
Paxton does not appear to disclose:
… the method further comprising: processing the additional sensor observation using a sensor encoder neural network to generate a set of sensor feature vectors representing the additional sensor observation; and processing the executor instruction and the set of sensor feature vectors, using the executor neural network system, to generate the action selection output.
However, in the same field of endeavor of robotic controls Kranski discloses:
“the method further comprising: processing the additional sensor observation using a sensor encoder neural network to generate a set of sensor feature vectors representing the additional sensor observation; (Kranski [0017] [0032] wherein sensors are able to encode features as a vector for controlling the robot in an environment) and processing the executor instruction and the set of sensor feature vectors, using the executor neural network system, to generate the action selection output.” (Kranski [0017] [0032] wherein sensors are able to encode features as a vector for controlling the robot in an environment)
It would have been obvious for one having ordinary skill in the art prior to the effective filing date of the invention to combine the sensor vector for the machine learning model of Kranski with the system of Paxton with a reasonable expectation of success because one of ordinary skill would have been motivated to make this modification in order to provide improved performance of the system, and optimize control states of the robot through machine learning vectors (Kranski [0017-0019]).
Regarding claim 14 Paxton discloses all of the limitations of claim 1 but Paxton does not appear to further disclose:
… training at least the executor neural network system to perform one or more sub-tasks of the plurality of tasks using a reinforcement learning technique based on rewards provided by the dispatcher neural network system to the executor neural network system.
However, in the same field of endeavor of robotic controls Kranski discloses:
“training at least the executor neural network system to perform one or more sub-tasks of the plurality of tasks using a reinforcement learning technique based on rewards provided by the dispatcher neural network system to the executor neural network system.” (Kranski [0027-0028] wherein the robot learns based on reinforcement learning models from feedback machine learning data i.e. the dispatcher network to improve robotic controls i.e. the execution system)
It would have been obvious for one having ordinary skill in the art prior to the effective filing date of the invention to combine the sensor vector for the machine learning model of Kranski with the system of Paxton with a reasonable expectation of success because one of ordinary skill would have been motivated to make this modification in order to provide improved performance of the system, and protect the system from damage, while identifying unwanted conditions or fail states of the robot operation (Kranski [0017-0019] [0027-0028]).
Allowable Subject Matter
Claims 17-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 17-18 the relevant art Paxton in view of Kranski discloses a control system for a robot (Paxton [0020]) using sensed information from the environment (Paxton [0028] [0032-0033]) to control the robot (Paxton [0033] [0039] [0049]) but fails to disclose wherein using the parallel processing of a first and second hardware device to output sub tasks at a current sub task step, or sharing information with other agents in other environments at a time step to control instructions for the other robots. Specifically, the relevant art fails to disclose “implementing the dispatcher neural network system on a first hardware computing device; implementing the executor neural network system on a second, different hardware computing device, in communication with the first hardware computing device to receive the executor instruction; and using the first hardware computing device to process the observation characterizing a state of the environment at a next sub-task execution time step to generate the executor instruction for the next sub-task execution time step in parallel with the processing the executor instruction, using the executor neural network system on the second hardware computing device, to generate an action selection output for performing the set of one or more sub-task actions for a current sub-task execution time step” or “further comprising: implementing the dispatcher neural network system on a first hardware computing device, and wherein generating the executor instruction comprising the set of tokens comprises selecting each token from an observation description language; implementing the executor neural network system on a second, different hardware computing device, in communication with the first hardware computing device to receive the executor instruction; and sharing the dispatcher neural network system with one or more other agents in one or more other respective environments to perform a respective task, wherein the sharing comprises, for each other agent, processing a task description for the respective task of the other agent and an observation characterizing a state of the respective environment of the other agent at a time step, using the dispatcher neural network system, to generate a respective executor instruction for the other agent”.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 2023/0182296 A1 discloses a robotic arm training technique based on goal conditions
US 2022/0066456 A1 discloses operating a robot in an environment based on generated sensor data, such as LIDAR data
US 11,745,332 B2 discloses a robot implementation directed to maintaining end effector poses interacting with the environment
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kyle T Johnson whose telephone number is (303)297-4339. The examiner can normally be reached Monday-Thursday 7:00-5:00 MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wade Miles can be reached at (571) 270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE T JOHNSON/Examiner, Art Unit 3656