Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-30 are presented for examination.
Information Disclosure Statement
Item 35 (Olsson et al) in the “Other Documents” section of the information disclosure statement (IDS) filed 5/15/2024 fails to comply with 37 CFR 1.98(a)(2), which requires a legible copy of each cited foreign patent document; each non-patent literature publication or that portion which caused it to be listed; and all other information or that portion which caused it to be listed. It has been placed in the application file, but the information referred to therein has not been considered.
However, Examiner has listed the non-patent literature publication Olsson et al., "In-context learning and induction heads" Transformer Circuits Thread, March 8, 2022, 71 pages, in the attached PTO-892 form and attached a legible copy of the publication to this Office Action.
Claim Objections
Claims 4 is/are objected to because of the following informalities:
Claim(s) 4 “(iii) an example output sequence comprising“
should be “(ii) an example output sequence comprising“.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-17, 20-30, is/are rejected under 35 U.S.C. 103 as being unpatentable over Paxton (US 20230297074 A1), in view of Huang et al “InnerMonologue: Embodied Reasoning through Planning with Language Models” dated 12 Jul 2022 and retrieved from arXiv:2207.05608v1 and Sermanet (US 20230182296 A1).
Huang was cited in the IDS dated 5/15/2024.
Regarding claim 1, Paxton teaches a method for generating instructions for an agent interacting with an environment to perform a task, the method comprising (Paxton [20, 160, 161] instructions cause system device(s) to instruct an agent to interact with environment):
receiving a natural language description of the task to be performed by the agent (Paxton [31, 48, 50] user input (speech natural language) instructions for task may be received);
processing, using a language model neural network, an input sequence derived from the natural language description of the task to generate an output …. sequence that comprises … descriptions of each of a sequence of sub-goals to be achieved by the agent while performing the task (Paxton [31, 33, 48-51] Figs 5A and 5B, sequence for user input may be processed using natural language understanding (NLU) module to generate sequence of goals and actions to achieve task);
at each of one or more time steps (Paxton [33, 34, 49, 52] Figs 5A and 5B, process may be repeated until entire sequence of tasks is completed based on analysis of condition(s) at present time, processes 500 and 550 perform sequence of tasks based on user instructions and current environment image information):
receiving a current observation image characterizing a current state of the environment at the time step (Paxton [49,51] image of current environment (observation) is received);
identifying a current sub-goal in the sequence of sub-goals being performed by the agent at the time step (Paxton [48-52] current sub-goal/ intermediate goal is identified based on planner);
generating, using …. vision and language model neural network(s)… and from the current observation image, an observation embedding of the current observation (Paxton [30- 32, 36, 38, 48-52] NLU and vision models (which can be neural networks) generate embeddings about current state using current environment image);
determining whether the agent has successfully achieved the current sub-goal in the sequence as of the time step (Paxton [25, 32, 34, 50, 52, 48-52] determination is made whether current planner sub-goal/ intermediate goal (end-condition) in the path has been performed); and
in response to determining that the agent has successfully achieved the current sub-goal in the sequence as of the time step, instructing the agent to perform a next sub-goal that follows the current sub-goal in the sequence (Paxton [25, 32, 34, 50, 52, 48-52] if current end-condition is achieved but sequence of tasks has not been completed, then process continues to identify next sub-goal/ intermediate goal and instruct robot to perform the task for next sub-goal).
Paxton does not specifically teach output text sequence that comprises natural language descriptions of each of a sequence of sub-goals to be achieved by the agent while performing the task, generating, using a vision-language model (VLM) neural network …
However Huang teaches processing, using a language model …, an input sequence derived from the natural language description of the task to generate an output text sequence that comprises natural language descriptions of each of a sequence of sub-goals to be achieved by the agent while performing the task (Huang Secs 4.2 and 4.4 and Listing 2 (Pg 21), sequence for user instruction is converted to natural language description(s) for sequence of goals, demonstrates many impressive reasoning and replanning behaviors beyond the examples given in the prompt, and also inherits many of the appealing properties from its versatility and general-purpose language understanding).
It would have been obvious to one of an ordinary skill in the art before the effective filing date of the claimed invention, to have incorporated the concept taught by Huang of processing, using a language model …, an input sequence derived from the natural language description of the task to generate an output text sequence that comprises natural language descriptions of each of a sequence of sub-goals to be achieved by the agent while performing the task, into the invention suggested by Paxton; since both inventions are directed towards generating and performing sub-goals for user instructions based on current observations and success of previous sub-tasks, and incorporating the teaching of Huang into the invention suggested by Paxton would provide the added advantage of many impressive reasoning and replanning behaviors beyond the examples given in the prompt, and also inherits many of the appealing properties from its versatility and general-purpose language understanding, and the combination would perform with a reasonable expectation of success (Huang Secs 4.2 and 4.4 and Listing 2 (Pg 21)).
Paxton and Huang does not specifically teach using a vision-language model (VLM) neural network.
However Sermanet teaches generating, using a vision-language model (VLM) neural network and from the current observation image, an observation embedding of the current observation (Sermanet Abstract [33, 34, 49, 50, 51 66] Goal-conditioned policy network (which can be a neural network) uses natural language and current image observation (VLM) to generate embeddings for current observation, allows mapping from image goals and instructions respectively to the same latent visuo-lingual goal space, robot performs row (sequence) of tasks based on instructions and current observation image).
It would have been obvious to one of an ordinary skill in the art before the effective filing date of the claimed invention, to have incorporated the concept taught by Sermanet of generating, using a vision-language model (VLM) neural network and from the current observation image, an observation embedding of the current observation, into the invention suggested by Paxton and Huang; since both inventions are directed towards a robot performing a sequence of tasks based on instructions and current observation image, and incorporating the teaching of Sermanet into the invention suggested by Paxton and Huang would provide the added advantage of allowing mapping from image goals and instructions respectively to the same latent visuo-lingual goal space, and the combination would perform with a reasonable expectation of success (Sermanet Abstract [33, 34, 49, 50, 51 66]).
Regarding claim 20, Paxton teaches a method of training a language-conditioned … neural network that is configured to process an input comprising (i) data characterizing a state of an environment and (ii) data representing a natural language description of a task to generate an … output that defines an action to be performed by an agent interacting with the environment to perform the task, the method comprising (Paxton [33, 20, 160, 161] instructions cause system device(s) to train a neural network (predicate predictor) to instruct an agent to interact with environment, neural network may be trained based on task NLU and environment state observation to generate actions to be performed by robot):
processing, using a language model neural network, an input sequence derived from the natural language description of the task to generate an output … sequence that comprises … descriptions of each of a sequence of sub-goals to be achieved by the agent while performing the task (Paxton [31, 33, 48-51] Figs 5A and 5B, sequence for user input may be processed using natural language understanding (NLU) module to generate sequence of goals and actions to achieve task);
controlling the agent using the language-conditioned … neural network to perform a task episode of the task and to generate a trajectory that includes experience data for each of a sequence of time steps during the task episode that comprises a respective observation image for the time step (Paxton [34, 37, 49, 52] model(s) can determine path(s) (trajectory) of tasks to be performed based on the NLU and image analysis, and monitor current state based on sequence of tasks (episode) and current state observation based on image, then process continues to identify next sub-goal/ intermediate goal and instruct robot to perform the task for next sub-goal, Paxton [33, 34, 49, 52] steps may be performed for different time steps- repeated until entire sequence of tasks is completed based on analysis of condition(s) at present time);
determining, using … vision and language model neural network(s)…, whether any of the sub-goals in the sequence were successfully achieved by the agent at any of the sequence of time steps (Paxton [30- 32, 36, 38, 48-52] NLU and vision models (which can be neural networks) generate embeddings about current state using current environment image, Paxton [25, 32, 34, 50, 52, 48-52] determination is made whether current planner sub-goal/ intermediate goal (end-condition) in the path has been performed);; and
in response to determining that a given sub-goal was achieved at a given time step of the plurality of time steps, adding a trajectory that includes experience data for time steps preceding the given time step in the task episode to … a list of completed sub-goals …(Paxton [25, 32, 34, 49, 50, 52, 48-52] if current end-condition is achieved but sequence of tasks has not been completed, then process continues to identify next sub-goal/ intermediate goal, process keeps track of completed and not yet completed sub-goals).
Paxton does not specifically teach a language-conditioned policy neural network; an output text sequence that comprises natural language descriptions of each of a sequence of sub-goals to be achieved; controlling the agent using the language-conditioned policy neural network; determining, using a vision-language model (VLM) neural network; adding a trajectory that includes experience data for time steps preceding the given time step in the task episode to a replay memory for use in training the policy neural network.
However Huang teaches processing, using a language model neural network, an input sequence derived from the natural language description of the task to generate an output text sequence that comprises natural language descriptions of each of a sequence of sub-goals to be achieved by the agent while performing the task (Huang Secs 4.2 and 4.4 and Listing 2 (Pg 21), sequence for user instruction is converted to natural language description(s) for sequence of goals, demonstrates many impressive reasoning and replanning behaviors beyond the examples given in the prompt, and also inherits many of the appealing properties from its versatility and general-purpose language understanding).
It would have been obvious to one of an ordinary skill in the art before the effective filing date of the claimed invention, to have incorporated the concept taught by Huang of processing, using a language model neural network, an input sequence derived from the natural language description of the task to generate an output text sequence that comprises natural language descriptions of each of a sequence of sub-goals to be achieved by the agent while performing the task, into the invention suggested by Paxton; since both inventions are directed towards generating and performing sub-goals for user instructions based on current observations and success of previous sub-tasks, and incorporating the teaching of Huang into the invention suggested by Paxton would provide the added advantage of many impressive reasoning and replanning behaviors beyond the examples given in the prompt, and also inherits many of the appealing properties from its versatility and general-purpose language understanding, and the combination would perform with a reasonable expectation of success (Huang Secs 4.2 and 4.4 and Listing 2 (Pg 21)).
Paxton and Huang does not specifically teach a language-conditioned policy neural network; controlling the agent using the language-conditioned policy neural network; determining, using a vision-language model (VLM) neural network; adding a trajectory that includes experience data for time steps preceding the given time step in the task episode to a replay memory for use in training the policy neural network.
However Sermanet teaches training a language-conditioned policy neural network that is configured to process an input comprising (i) data characterizing a state of an environment and (ii) data representing a natural language description of a task to generate a policy output that defines an action to be performed by an agent interacting with the environment to perform the task (Sermanet [3, 49, 50, 53, 54] language conditioned policy neural network trained using task description and current state image observations);
controlling the agent using the language-conditioned policy neural network to perform a task episode of the task and to generate a trajectory that includes experience data for each of a sequence of time steps during the task episode that comprises a respective observation image for the time step (Sermanet [34, 54, 66, 71, 72] for each current observation in loop of observations, robot performs next action in trajectory based on next task for instructions and current observation image);
determining, using a vision-language model (VLM) neural network, whether any of the sub-goals in the sequence were successfully achieved by the agent at any of the sequence of time steps (Sermanet [33, 34, 49, 50, 51, 54, 66] Goal-conditioned policy network (which can be a neural network) uses natural language and current image observation (VLM) to generate embeddings for current observation, allows mapping from image goals and instructions respectively to the same latent visuo-lingual goal space, determine which goals in trajectory have been reached (success), and determine next state/action based on current reached state in trajectory);
in response to determining that a given sub-goal was achieved at a given time step of the plurality of time steps, adding a trajectory that includes experience data for time steps preceding the given time step in the task episode to a replay memory for use in training the policy neural network (Sermanet [34, 53, 64, 71, 78 ] based on goals reached in trajectory(ies) for task, training instance including current observation data is stored in memory to be used for training, Sermanet [33] allows for a single trained policy that can perform many complex robotic manipulation skills in a row, directly from images, and specified entirely with natural language).
It would have been obvious to one of an ordinary skill in the art before the effective filing date of the claimed invention, to have incorporated the concept taught by Sermanet of training a language-conditioned policy neural network that is configured to process an input comprising (i) data characterizing a state of an environment and (ii) data representing a natural language description of a task to generate a policy output that defines an action to be performed by an agent interacting with the environment to perform the task; controlling the agent using the language-conditioned policy neural network to perform a task episode of the task and to generate a trajectory that includes experience data for each of a sequence of time steps during the task episode that comprises a respective observation image for the time step; determining, using a vision-language model (VLM) neural network, whether any of the sub-goals in the sequence were successfully achieved by the agent at any of the sequence of time steps; in response to determining that a given sub-goal was achieved at a given time step of the plurality of time steps, adding a trajectory that includes experience data for time steps preceding the given time step in the task episode to a replay memory for use in training the policy neural network, into the invention suggested by Paxton and Huang; since both inventions are directed towards a robot performing a sequence of tasks based on instructions and current observation image, and incorporating the teaching of Sermanet into the invention suggested by Paxton and Huang would provide the added advantage of allowing for a single trained policy that can perform many complex robotic manipulation skills in a row, directly from images, and specified entirely with natural language, and the combination would perform with a reasonable expectation of success (Sermanet [3, 49, 50, 53, 54, 34, 54, 66, 71, 72, 33, 34, 51, 64, 71, 33]).
Regarding claim 2, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 1 above.
Paxton does not specifically teach wherein instructing the agent to perform a next sub-goal that follows the current sub-goal in the sequence comprises: determining, from the natural language description of the next sub-goal, one or more actions to be performed by the agent; and causing the agent to perform the one or more actions
However Huang teaches wherein instructing the agent to perform a next sub-goal that follows the current sub-goal in the sequence comprises: determining, from the natural language description of the next sub-goal, one or more actions to be performed by the agent; and causing the agent to perform the one or more actions (Huang Secs 4.2 and 4.4 and Listing 2 (Pg 21), natural language description(s) for next sub- goals is used to determine next action to be taken by robot and then robot performs the action).
Regarding claim 3, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 2 above.
Paxton does not specifically teach wherein determining, from the natural language description of the next sub-goal, one or more actions to be performed by the agent comprises: processing an input comprising (i) data characterizing the current state of the environment and (ii) data representing the natural language description of the next sub-goal using a language-conditioned policy neural network to generate a policy output that defines an action to be performed by the agent; and selecting an action using the policy output
However Sermanet teaches wherein determining, from the natural language description of the next sub-goal, one or more actions to be performed by the agent comprises: processing an input comprising (i) data characterizing the current state of the environment and (ii) data representing the natural language description of the next sub-goal using a language-conditioned policy neural network to generate a policy output that defines an action to be performed by the agent; and selecting an action using the policy output (Sermanet [68, 86-97] actions are determined based on current state in trajectory(ies) based on observation(s), trajectory(ies) may be based on natural language instructions (series of goals) and observation states, determined actions are performed by robot).
Regarding claim 4, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 1 above.
Paxton does not specifically teach wherein the input sequence comprises one or more prompt sequences, each prompt sequence comprising (i) an example natural language task description and (iii) an example output sequence comprising example natural language descriptions of example sub-goals
However Huang teaches wherein the input sequence comprises one or more prompt sequences, each prompt sequence comprising (i) an example natural language task description and (iii) an example output sequence comprising example natural language descriptions of example sub-goals (Huang Secs 3.2, 3.3, 4.4, Listing 2, input may include real world examples which include prompt sequences with input task descriptions and output sequence of goals).
Regarding claim 5, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 4 above.
Paxton does not specifically teach within each example output sequence, the example natural language descriptions of example sub-goals are arranged according to a particular syntax, and wherein the method further comprises: identifying the natural language descriptions in the output sequence by parsing the output sequence according to the particular syntax
However Huang teaches within each example output sequence, the example natural language descriptions of example sub-goals are arranged according to a particular syntax, and wherein the method further comprises: identifying the natural language descriptions in the output sequence by parsing the output sequence according to the particular syntax (Huang Secs Fig. 3, 3.2, 3.3, 4.4, Listing 2 prompts (including examples) have formats specifying how to parse natural language input by user and known tasks).
Regarding claim 6, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 1 above.
Paxton does not specifically teach wherein the VLM neural network has been pre-trained on a first training data set of images and corresponding text descriptions and
fine-tuned on a second training data set that comprises images of the environment and corresponding text descriptions
However Huang teaches wherein the VLM neural network has been pre-trained on a first training data set of images and corresponding text descriptions and
fine-tuned on a second training data set that comprises images of the environment and corresponding text descriptions (Huang Secs 3.2, 3.3, 4.4, Listing 2, Fusing Vision, Language, and Control in Robotics (Pg 2-3), Environment Feedback: Success Detection (Pg 17-18) VLM may be pretrained using first dataset and then fine-tuned using second dataset, dataset may include images and text information).
Regarding claim 7, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 1 above.
Paxton further teaches wherein the agent is a mechanical agent and the environment is a real-world environment (Paxton [22,24] agent may be a robot that moves objects (mechanical) that works in a real environment).
Regarding claim 8, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 7 above.
Paxton further teaches wherein the agent is a robot (Paxton [22,24] agent may be a robot that moves objects (mechanical) that works in a real environment).
Regarding claim 9, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 1 above.
Paxton further teaches wherein the environment is a real-world environment of a service facility comprising a plurality of items of electronic equipment and the agent is an electronic agent configured to control operation of the service facility (Paxton [24, 48, 49, 111, 113] agent may be in a real environment and be used to control electronic devices in a facility that provides services).
Regarding claim 10, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 1 above.
Paxton further teaches wherein the environment is a real-world manufacturing environment for manufacturing a product and the agent comprises an electronic agent configured to control a manufacturing unit or a machine that operates to manufacture the product (Paxton [22, 24, 25] agent may be a robot used in for manufacturing a product in a real manufacturing facility).
Regarding claim 11, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 3 above.
Claim 3 further teaches language-conditioned policy neural network.
Paxton further teaches wherein the environment is a simulation of a real-world environment, wherein the natural language description of the task is received during training of the language-conditioned … neural network, and wherein the method further comprises:
after the training, controlling a real-world agent in the real-world environment using the … neural network (Paxton [24, 30-34] environment may be simulation or real environment, neural network may be trained using natural language instruction of task, trained neural network may be used to control agent to perform tasks in real or simulated environment).
Regarding claim 12, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 3 above.
Claim 3 further teaches language-conditioned policy neural network.
Paxton further teaches wherein the environment is a simulation of a real-world environment, wherein the natural language description of the task is received during training of the language-conditioned … neural network, and wherein the method further comprises:
after the training, providing data specifying the … neural network for use in controlling a real-world agent in the real-world environment
(Paxton [24, 30-34, 112, 113] environment may be simulation or real environment, neural network may be trained using natural language instruction of task, trained neural network may be used to control agent to perform tasks in real or simulated environment, trained neural network objects may be provided for using the trained neural network).
Regarding claim 13, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 1 above.
Paxton further teaches wherein determining whether the agent has successfully achieved the current sub-goal in the sequence as of the time step comprises: determining, from the observation embedding of the current observation, whether the agent has successfully achieved the current sub-goal in the sequence as of the time step (Paxton [30- 32, 36, 38, 48-52] NLU and vision models (which can be neural networks) generate embeddings about current state using current environment image, Paxton [25, 32, 34, 50, 52, 48-52] determination is made whether current planner sub-goal/ intermediate goal (end-condition) in the path has been performed).
Regarding claim 14, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 13 above.
Claim 13 further teaches using a vision-language model (VLM) neural network.
Paxton further teaches generating a respective text embedding for each of the sub-goals in the sequence by processing the natural language description of the sub-goal using …. vision and language model neural network…, wherein determining whether the agent has successfully achieved the current sub-goal in the sequence as of the time step comprises: determining, from the observation embedding of the current observation and the text embedding of the current sub-goal, whether the agent has successfully achieved the current sub-goal in the sequence as of the time step (Paxton [30- 32, 36, 38, 48-52] NLU and vision models (which can be neural networks) generate embeddings about current state using current environment image, Paxton [30- 32, 36, 38, 48-52] NLU and vision models (which can be neural networks) generate embeddings about current state using current environment image, Paxton [25, 32, 34, 50, 52, 48-52] determination is made whether current planner sub-goal/ intermediate goal (end-condition) in the path has been performed);
Regarding claim 15, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 14 above.
Paxton does not specifically teach wherein determining, from the observation embedding of the current observation and the text embedding of the current sub-goal, whether the agent has successfully achieved the current sub-goal in the sequence as of the time step comprises: computing a similarity score between the observation embedding of the current observation and the text embedding of the current sub-goal; and
determining that the agent has successfully achieved the current sub-goal when the similarity score satisfies a threshold
However Huang teaches wherein determining, from the observation embedding of the current observation and the text embedding of the current sub-goal, whether the agent has successfully achieved the current sub-goal in the sequence as of the time step comprises: computing a similarity score between the observation embedding of the current observation and the text embedding of the current sub-goal; and determining that the agent has successfully achieved the current sub-goal when the similarity score satisfies a threshold (Huang Sec A.3 “Environment Feedback: Success Detection” similarity score between goal and observation embeddings is determined, success is detected if the score meets threshold, similarity score can be dot product).
Regarding claim 16, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 15 above.
Paxton does not specifically teach wherein the similarity score is a dot product and the threshold is satisfied when the dot product exceeds the threshold
However Huang teaches wherein the similarity score is a dot product and the threshold is satisfied when the dot product exceeds the threshold (Huang Sec A.3 “Environment Feedback: Success Detection” similarity score between goal and observation embeddings is determined, success is detected if the score meets threshold, similarity score can be dot product).
Regarding claim 17, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 1 above.
Paxton further teaches wherein instructing the agent to perform a next sub-goal that follows the current sub-goal in the sequence comprises: providing, to the agent, information about how to perform the task that is generated using at least the description of the next sub-goal (Paxton [49, 52] agent is controlled to perform task for next sub-goal by using actions for the sub-goal that the robot is made to perform).
Regarding claim 21, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 20 above.
Paxton does not specifically teach selecting one or more trajectories from the replay memory; and training the policy neural network on the one or more trajectories through imitation learning
However Sermanet teaches selecting one or more trajectories from the replay memory; and training the policy neural network on the one or more trajectories through imitation learning (Sermanet [34, 63, 64, 78] trajectories may be selected from stored instances and used to train policy neural network through imitation learning).
Regarding claim 22, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 20 above.
Paxton does not specifically teach receiving a reward that indicates whether the task was successfully performed during the task episode; and when the reward indicates that the task was successfully performed, adding the trajectory to the replay memory
However Sermanet teaches receiving a reward that indicates whether the task was successfully performed during the task episode; and when the reward indicates that the task was successfully performed, adding the trajectory to the replay memory (Sermanet [34, 53, 63, 64, 71, 78 ] based on goals reached in trajectory(ies) for task(s) (success) and loss for the instance, training instance including current observation data is stored in memory to be used for training).
Regarding claim 23, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 20 above.
Claim 20 further teaches using a vision-language model (VLM) neural network.
Paxton further teaches determining, using …. vision and language model neural network…, whether any of the sub-goals in the sequence were successfully achieved by the agent at any of the plurality of time steps comprises: generating a respective text embedding for each of the sub-goals in the sequence by processing the natural language description of the sub-goal using a …. vision and language model neural network…, and for each time step: generating, using the …. vision and language model neural network…and from the observation image for the time step, an observation embedding of the observation image for the time step; and for each sub-goal, determining, from the observation embedding of the observation image for the time step and the text embedding of the sub-goal, whether the agent has successfully achieved the sub-goal as of the time step (Paxton [30- 32, 36, 38, 48-52] NLU and vision models (which can be neural networks) generate embeddings about current state using current environment image, Paxton [30- 32, 36, 38, 48-52] NLU and vision models (which can be neural networks) generate embeddings about current state using current environment image, Paxton [25, 32, 34, 50, 52, 48-52] determination is made whether current planner sub-goal/ intermediate goal (end-condition) in the path has been performed);
Regarding claim 24 Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 20 above.
Paxton does not specifically teach obtaining a plurality of previous task trajectories for a previous task; for each previous task trajectory: determining, using the VLM neural network, whether any of the sub-goals in the sequence were successfully achieved by the agent at any of the plurality of time steps in the previous task trajectory; and in response to determining that a given sub-goal was achieved at a given time step in the previous task trajectory, adding a trajectory that includes experience data for time steps preceding the given time step in the previous task trajectory to the replay memory for use in training the policy neural network
However Sermanet teaches obtaining a plurality of previous task trajectories for a previous task; for each previous task trajectory: determining, using the VLM neural network, whether any of the sub-goals in the sequence were successfully achieved by the agent at any of the plurality of time steps in the previous task trajectory; and in response to determining that a given sub-goal was achieved at a given time step in the previous task trajectory, adding a trajectory that includes experience data for time steps preceding the given time step in the previous task trajectory to the replay memory for use in training the policy neural network (Sermanet [34, 63, 64, 78] trajectories may be selected from stored instances and used to train policy neural network through imitation learning, Sermanet [33, 34, 49, 50, 51, 54, 66] Goal-conditioned policy network (which can be a neural network) uses natural language and current image observation (VLM) to generate embeddings for current observation, allows mapping from image goals and instructions respectively to the same latent visuo-lingual goal space, determine which goals in trajectory have been reached (success), and determine next state/action based on current reached state in trajectory).
Regarding claim 25, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 24 above.
Paxton does not specifically teach wherein each previous task trajectory was generated while the policy neural network was conditioned on data representing a description of a respective different task
However Sermanet teaches wherein each previous task trajectory was generated while the policy neural network was conditioned on data representing a description of a respective different task (Sermanet [34, 63, 64, 78] Algorithm 1, neural network may be used and trained using different tasks (task id)).
Regarding claim 26 Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 20 above.
Paxton further teaches wherein the agent is a mechanical agent and the environment is a real-world environment (Paxton [22,24] agent may be a robot that moves objects (mechanical) that works in a real environment).
Regarding claim 27, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 26 above.
Paxton further teaches wherein the agent is a robot (Paxton [22,24] agent may be a robot that moves objects (mechanical) that works in a real environment).
Regarding claim 28, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 20 above.
Claim 20 further teaches language-conditioned policy neural network.
Paxton further teaches wherein the environment is a simulation of a real-world environment, wherein the method further comprises: after the training, controlling a real-world agent in the real-world environment using the … neural network (Paxton [24, 30-34] environment may be simulation or real environment, neural network may be trained using natural language instruction of task, trained neural network may be used to control agent to perform tasks in real or simulated environment).
Regarding claim 29 Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 20 above.
Claim 20 further teaches language-conditioned policy neural network.
Paxton further teaches wherein the environment is a simulation of a real-world environment and wherein the method further comprises: after the training, providing data specifying the policy neural network for use in controlling a real-world agent in the real-world environment (Paxton [24, 30-34, 112, 113] environment may be simulation or real environment, neural network may be trained using natural language instruction of task, trained neural network may be used to control agent to perform tasks in real or simulated environment, trained neural network objects may be provided for using the trained neural network).
Claim 30 is directed towards a system executing instructions similar in scope to the instructions performed by the method of claim 1, and is rejected under the same rationale. Paxton further teaches a system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for generating instructions for an agent interacting with an environment to perform a task, the operations comprising (Paxton [20, 160, 161] instructions cause system device(s) to instruct an agent to interact with environment)
Claim(s) 18, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Paxton (US 20230297074 A1), Huang et al and Sermanet (US 20230182296 A1), and further in view of Nott (US 20230101223 A1).
Regarding claim 18, Paxton, Huang and Sermanet teach(es) the invention as claimed in claim 1 above.
Paxton further teaches wherein the agent comprises a user; obtaining information defining the task from the …agent…; and using the …agent…to provide the information about how to perform the task to the user (Paxton [26, 31, 48, 50] user input (speech natural language) instructions for task may be received, (Paxton [49, 52] agent is controlled to perform task for next sub-goal based on the task definition, task us performed by using actions for the sub-goal that the robot is made to perform).).
Paxton does not specifically teach wherein the agent comprises a user of a digital assistant
However Nott teaches wherein the agent comprises a user of a digital assistant (Nott [4, 18, 35, 52] robot may be a digital assistant that performs a sequence of tasks based on current state, allows for a fully automated personal digital assistant).
It would have been obvious to one of an ordinary skill in the art before the effective filing date of the claimed invention, to have incorporated the concept taught by Nott of wherein the agent comprises a user of a digital assistant, into the invention suggested by Paxton, Huang and Sermanet; since both inventions are directed towards agent(s) performing a sequence of tasks based on current state, and incorporating the teaching of Nott into the invention suggested by Paxton, Huang and Sermanet would provide the added advantage of allowing for a fully automated personal digital assistant, and the combination would perform with a reasonable expectation of success (Nott [4, 18, 35, 52]).
Regarding claim 19, Paxton, Huang, Sermanet and Nott teach(es) the invention as claimed in claim 18 above.
Claim 19 further teaches a digital assistant.
Paxton further teaches receiving, at the …agent…, a request from the user for assistance; determining, in response to the request, that the user should perform the task; and
outputting, from the …agent…to the user, an indication of the task to be performed; wherein the current observation is a visual or audio observation or both of the user performing the task captured by the …agent… (Paxton [28, 49] request from user may be processed, request may include audio and visual information, instructions may be provided for user).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SANCHITA ROY whose telephone number is (571)272-5310. The examiner can normally be reached Monday-Friday 12-8.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SANCHITA ROY
Primary Examiner
Art Unit 2146
/SANCHITA ROY/Primary Examiner, Art Unit 2146
