DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Application filed on April 19, 2024. Claims 1-22 are pending in the case. Claims 1, 11, 16, 21, and 22 are the independent claims.
This action is non-final.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 22 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 22 recites “the reward function.” This limitation lacks antecedent basis.
Claim Rejections – 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 21 and 22 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wenhao Yu and Nimrod Gileadi and Chuyuan Fu and Sean Kirmani and Kuang-Huei Lee and Montse Gonzalez Arenas and Hao-Tien Lewis Chiang and Tom Erez and Leonard Hasenclever and Jan Humplik and Brian Ichter and Ted Xiao and Peng Xu and Andy Zeng and Tingnan Zhang and Nicolas Heess and Dorsa Sadigh and Jie Tan and Yuval Tassa and Fei Xia. Language to Rewards for Robotic Skills Synthesis. ArXive.org. June 14, 2023. arXiv:2306.08647. retrieved from the internet: https://arxiv.org/pdf/2306.08647. (hereinafter Yu).
With respect to claim 21, Yu teaches a computer-implemented method for training an autonomous machine to perform a target task in a target environment, comprising:
generating a reward signature by combining a natural language description of the target task and a natural language description of the target environment (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, where solving a Markov decision process consists in finding a policy that maximizes rewards; page 4 fourth-sixth paragraphs, continuing on page 5, describing motion description and reward coding, along with Fig. 2, Motion Descriptor LLM interprets and expands user input into natural language description of desired robot motion following predefined template; template describes common environments of robots, and motion descriptor completes the provided template; translating generated motion description into the reward function using second LLM; formulating the problem of language to reward function as a code-writing task to benefit from the LLM’s knowledge of coding and code structure; second LLM named Reward Coder; designing a prompt for instructing the LLM to generate reward specifying code, the prompt consisting of a description of reward APIs the LLM can call to specify different parameters of the reward function, an example response that we expect the Reward Coder to produce, and constraints and rules the Reward Coder needs to follow; as shown in the Reward Coder Prompt of Fig. 2, the prompt for the Reward Coder LLM includes natural language descriptions of a target task and requests executable source code to use for training a policy for the autonomous machine to perform the target task; page 18, iv) Motion Descriptor prompt for Dexterous Manipulator, which includes a description of objects in the environment; i.e. a natural language prompt for a robot to perform a task is received by a user, and this is further transformed into a natural language description of the desired motion by further combining additional information including information about objects in the environment, and this natural language description including desired motion and environment information is utilized in conjunction with a second prompt requesting a Reward Coder LLM to executable reward code for use in the context of a Markov Decision Process (i.e. to train a policy for the autonomous machine/robot to perform the target task));
generating a reward function by prompting a large language model with the reward signature (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process; finding sequence of actions that maximizes expected accumulated reward; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, where solving a Markov decision process consists in finding a policy that maximizes rewards; page 4 final paragraph, continuing on page 5, prompting Reward Coder LLM to generate reward specifying code, as shown in Fig. 2; see also example outputs of the LLMs in Appendix A.8, pages 21-29; i.e. as shown in Fig. 2 and the example outputs of Appendix A.8, the Reward Coder LLM is prompted to generate the reward specifying code and generates a function as executable source code for use in the context of a Markov Decision process (i.e. to train the policy, including by providing the corresponding reward to be provided based on reaching the goal));
computing a state description using a model of the target environment and an embedding of the natural language task description (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process commonly used to formulate robot control problems and including a state space S and initial state distribution po; finding sequence of actions that maximizes expected accumulated reward; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, including state description S including the environment and position of the robot relative to the environment; see also page 5, section 3.3, indicating that the robot sends updated robot states the MJPC planner to initiate a next planning cycle, page 7, section 4.6, indicating that in a simulation there is ground-truth state for objects in the scene and that this can be estimated in a real world environment using 3D bounding boxes as state estimations for corresponding objects in the simulation, and Fig. 6, showing the robot sending this state estimation (i.e. showing positions of objects in the environment relative to the robot); i.e. the system is implemented in the context of a Markov Decision Process formulating robot control problems and therefore includes a state space/estimation/description indicating positions of the robot other objects in the environment relative to the robot; Examiner notes that paragraph 0055 of the specification of the instant application appears to indicate that use of such a Markov Decision Process (including a state description/environment and position of robot relative to the environment) is common); and
training a policy for the autonomous machine to perform the target task using the reward function and the state description (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process commonly used to formulate robot control problems and including a state space S and initial state distribution po; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, including state description S including the environment and position of the robot relative to the environment, where solving a Markov decision process consists in finding a policy that maximizes rewards, and is an implementation of a reinforcement learning process; page 5 section 3.3, motion controller mapping the reward function generated by reward translator to low-level robot actions that maximize the accumulated reward, including using reinforcement learning; i.e. the output reward function code and state description information is utilized in the context of a Markov Decision process (i.e. to train the policy for the robot to complete the task, including by providing the corresponding reward to be provided based on reaching the goal)).
With respect to claim 22, Yu teaches a computer-implemented method for training an autonomous machine to perform a target goal in a target environment, comprising:
generating a goal position signature by combining a natural language description of the target goal and a natural language description of the target environment (e.g. page 4 fourth-sixth paragraphs, continuing on page 5, describing motion description, along with Fig. 2, Motion Descriptor LLM interprets and expands user input into natural language description of desired robot motion following predefined template; template describes common environments of robots, and motion descriptor completes the provided template; translating generated motion description into the reward function using second LLM; formulating the problem of language to reward function as a code-writing task to benefit from the LLM’s knowledge of coding and code structure; second LLM named Reward Coder; designing a prompt for instructing the LLM to generate reward specifying code, the prompt consisting of a description of reward APIs the LLM can call to specify different parameters of the reward function, an example response that we expect the Reward Coder to produce, and constraints and rules the Reward Coder needs to follow; as shown in the Reward Coder Prompt of Fig. 2, the prompt for the Reward Coder LLM includes natural language descriptions of a target task and requests executable source code to use for training a policy for the autonomous machine to perform the target task; ; page 18, iv) Motion Descriptor prompt for Dexterous Manipulator, which includes a description of objects in the environment; i.e. a natural language prompt for a robot to perform a task/achieve a desired goal position is received by a user, along with a prompt/template including natural language information about the target environment, analogous to a goal position signature which combines natural language descriptions of a target goal and a target environment);
generating a goal position function by prompting a large language model with the goal position signature (e.g. page 4, fourth and fifth paragraphs, Motion Descriptor interpreting and expanding into natural language description of desired robot motion following predefined template/prompt; page 4 final paragraph, continuing on page 5, prompting Reward Coder LLM to generate reward specifying code, as shown in Fig. 2; see also example outputs of the LLMs in Appendix A.8, pages 21-29; i.e. as shown in Fig. 2 and the example outputs of Appendix A.8, the generated code includes functions for setting goal/target positions of different parts of the robot (such as positions of the robot torso and feet, robot joint angles, etc.), executing set poses to reach a target pose, and for positions of objects in the environment such as faucet positions, etc.);
computing a state description using a model of the target environment and a goal position derived from the goal position function (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process commonly used to formulate robot control problems and including a state space S and initial state distribution po; finding sequence of actions that maximizes expected accumulated reward; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, including state description S including the environment and position of the robot relative to the environment; see also page 5, section 3.3, indicating that the robot sends updated robot states the MJPC planner to initiate a next planning cycle, page 7, section 4.6, indicating that in a simulation there is ground-truth state for objects in the scene and that this can be estimated in a real world environment using 3D bounding boxes as state estimations for corresponding objects in the simulation, and Fig. 6, showing the robot sending this state estimation (i.e. showing positions of objects in the environment relative to the robot); i.e. the system is implemented in the context of a Markov Decision Process formulating robot control problems and therefore includes a state space/estimation/description indicating positions of the robot other objects in the environment relative to the robot; Examiner notes that paragraph 0055 of the specification of the instant application appears to indicate that use of such a Markov Decision Process (including a state description/environment and position of robot relative to the environment) is common); and
training a policy for the autonomous machine to reach the target goal using the goal position derived from the goal position function, the state description, and the reward function (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process commonly used to formulate robot control problems and including a state space S and initial state distribution po; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, including state description S including the environment and position of the robot relative to the environment, where solving a Markov decision process consists in finding a policy that maximizes rewards, and is an implementation of a reinforcement learning process; page 5 section 3.3, motion controller mapping the reward function generated by reward translator to low-level robot actions that maximize the accumulated reward, including using reinforcement learning; i.e. the output reward function code and state description information is utilized in the context of a Markov Decision process (i.e. to train the policy for the robot to complete the task, including by providing the corresponding reward to be provided based on reaching the goal)).
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA 35 U.S.C. 103(c) and potential pre-AIA 35 U.S.C. 102€, (f) or (g) prior art under pre-AIA 35 U.S.C. 103(a).
Claims 1-7 and 11-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yu in view of Pack, III et al. (US 20250086011 A1).
With respect to claims 1, 11, and 16, Yu teaches a computer system for training an autonomous machine to perform a target task in a target environment, the computer system comprising: one or more processors; one or more non-transitory computer-readable media storing processor- executable instructions which, when executed, cause method steps to be performed; one or more non-transitory computer-readable media for training an autonomous machine to perform a target task in a target environment, the one or more non-transitory computer-readable media storing processor-executable instructions which, when executed, cause the method steps to be performed; and a computer-implemented method for training an autonomous machine to perform a target task in a target environment, comprising:
(in claims 10 and 16 only) receiving a natural language description of the target task and a natural language description of the target environment (e.g. page 4 fourth-fifth paragraphs, describing motion description and reward coding, along with Fig. 2, Motion Descriptor LLM interprets and expands user input into natural language description of desired robot motion following predefined template; template describes common environments of robots; page 18, iv) Motion Descriptor prompt for Dexterous Manipulator, which includes a description of objects in the environment; i.e. a user provides a natural language description of a desired robot motion, and a template/prompt is further provided that includes a natural language description of the environment such as objects in the environment);
generating a prompt for a large language model at least in part by combining a/the natural language description of the target task and a/the natural language description of the target environment, wherein the prompt requests executable source code to use for training a policy for the autonomous machine to perform the target task (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, where solving a Markov decision process consists in finding a policy that maximizes rewards; page 4 fourth-sixth paragraphs, continuing on page 5, describing motion description and reward coding, along with Fig. 2, Motion Descriptor LLM interprets and expands user input into natural language description of desired robot motion following predefined template; template describes common environments of robots, and motion descriptor completes the provided template; translating generated motion description into the reward function using second LLM; formulating the problem of language to reward function as a code-writing task to benefit from the LLM’s knowledge of coding and code structure; second LLM named Reward Coder; designing a prompt for instructing the LLM to generate reward specifying code, the prompt consisting of a description of reward APIs the LLM can call to specify different parameters of the reward function, an example response that we expect the Reward Coder to produce, and constraints and rules the Reward Coder needs to follow; as shown in the Reward Coder Prompt of Fig. 2, the prompt for the Reward Coder LLM includes natural language descriptions of a target task and requests executable source code to use for training a policy for the autonomous machine to perform the target task; page 18, iv) Motion Descriptor prompt for Dexterous Manipulator, which includes a description of objects in the environment; i.e. a natural language prompt for a robot to perform a task is received by a user, and this is further transformed into a natural language description of the desired motion by further combining additional information including information about objects in the environment, and this natural language description including desired motion and environment information is utilized in conjunction with a second prompt requesting a Reward Coder LLM to executable reward code for use in the context of a Markov Decision Process (i.e. to train a policy for the autonomous machine/robot to perform the target task));
generating a function by prompting the large language model with the prompt, wherein, based on the prompt, the function comprises executable source code that, when used to train the policy, causes a reward to be provided based on whether a goal position is reached in the target environment (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process; finding sequence of actions that maximizes expected accumulated reward; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, where solving a Markov decision process consists in finding a policy that maximizes rewards; page 4 final paragraph, continuing on page 5, prompting Reward Coder LLM to generate reward specifying code, as shown in Fig. 2; see also example outputs of the LLMs in Appendix A.8, pages 21-29; i.e. as shown in Fig. 2 and the example outputs of Appendix A.8, the Reward Coder LLM is prompted to generate the reward specifying code and generates a function as executable source code for use in the context of a Markov Decision process (i.e. to train the policy, including by providing the corresponding reward to be provided based on reaching the goal));
computing a state description using a model of the target environment, wherein the state description comprises a position of the autonomous machine relative to the target environment (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process commonly used to formulate robot control problems and including a state space S and initial state distribution po; finding sequence of actions that maximizes expected accumulated reward; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, including state description S including the environment and position of the robot relative to the environment; see also page 5, section 3.3, indicating that the robot sends updated robot states the MJPC planner to initiate a next planning cycle, page 7, section 4.6, indicating that in a simulation there is ground-truth state for objects in the scene and that this can be estimated in a real world environment using 3D bounding boxes as state estimations for corresponding objects in the simulation, and Fig. 6, showing the robot sending this state estimation (i.e. showing positions of objects in the environment relative to the robot); i.e. the system is implemented in the context of a Markov Decision Process formulating robot control problems and therefore includes a state space/estimation/description indicating positions of the robot other objects in the environment relative to the robot; Examiner notes that paragraph 0055 of the specification of the instant application appears to indicate that use of such a Markov Decision Process (including a state description/environment and position of robot relative to the environment) is common); and
training the policy for the autonomous machine to perform the target task using the function and the state description (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process commonly used to formulate robot control problems and including a state space S and initial state distribution po; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, including state description S including the environment and position of the robot relative to the environment, where solving a Markov decision process consists in finding a policy that maximizes rewards, and is an implementation of a reinforcement learning process; page 5 section 3.3, motion controller mapping the reward function generated by reward translator to low-level robot actions that maximize the accumulated reward, including using reinforcement learning; i.e. the output reward function code and state description information is utilized in the context of a Markov Decision process (i.e. to train the policy for the robot to complete the task, including by providing the corresponding reward to be provided based on reaching the goal)).
Assuming arguendo that Yu does not explicitly disclose a computer system for training an autonomous machine to perform a target task in a target environment, the computer system comprising: one or more processors; one or more non-transitory computer-readable media storing processor- executable instructions which, when executed, cause method steps to be performed; one or more non-transitory computer-readable media for training an autonomous machine to perform a target task in a target environment, the one or more non-transitory computer-readable media storing processor-executable instructions which, when executed, cause the method steps to be performed, Pack teaches a computer system for training an autonomous machine to perform a target task in a target environment, the computer system comprising: one or more processors; one or more non-transitory computer-readable media storing processor- executable instructions which, when executed, cause method steps to be performed; one or more non-transitory computer-readable media for training an autonomous machine to perform a target task in a target environment, the one or more non-transitory computer-readable media storing processor-executable instructions which, when executed, cause the method steps to be performed (e.g. paragraph 0058, computer program product having computer program code containing instructions implemented in media/storage medium; instructions implemented by general purpose processor to perform described techniques).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Yu and Pack in front of him to have modified the teachings of Yu (directed to using LLMs to generate executable reward code for learning robot policies), to incorporate the teachings of Pack (directed to automation based on prompting LLMs) to include the capability to embody the system of Yu using a processor and medium storing instructions, the instructions causing the processor to implement the system. One of ordinary skill would have been motivated to perform such a modification in order to enable automation using LLMs, allowing larger jobs to be decomposed into and accomplished with composable asynchronous tasks that use LLMs and GPTs as described in Pack (paragraph 0010).
With respect to claims 2, 12, and 17, Yu in view of Pack teaches all of the limitations of claims 1, 11, and 16 as previously discussed, and Yu further teaches wherein the target environment includes an object other than the autonomous machine, wherein the prompt includes a description of the object, wherein the goal position is a target three-dimensional position of the object, and wherein the state description further comprises a current three-dimensional position of the object (e.g. page 3, section 3.1, reward function defined in context of Markov Decision Process commonly used to formulate robot control problems and including a state space S and initial state distribution po; compare with paragraph 0055 of the specification of the instant application, indicating that such problems are commonly framed as Markov Decision Processes, including state description S including the environment and position of the robot relative to the environment; see also page 5, section 3.3, indicating that the robot sends updated robot states the MJPC planner to initiate a next planning cycle, page 7, section 4.6, indicating that in a simulation there is ground-truth state for objects in the scene and that this can be estimated in a real world environment using 3D bounding boxes as state estimations for corresponding objects in the simulation, and Fig. 6, showing the robot sending this state estimation (i.e. showing positions of objects in the environment relative to the robot); Figs. 3(b)-(c), Fig. 5(b), and Fig. 6, showing that the dexterous manipulator robot’s environment includes additional objects such as a palm, faucet, a drawer, a bowl, a banana, an apple, etc.; pages 18 and 19, showing prompts for Motion Descriptor and Reward Coder for Dexterous Manipulator, including indications of the apple, banana, box, bowl, drawer handle, faucet handle, drawer center, etc. in the target environment, along with indications for a function to minimize distances between two objects (i.e. indicating a goal position (in which the positions are minimized) with respect to a current position (a position prior to the minimization of the distance, such as a default position for the palm); in addition, the baseline Code-as-Policies prompt for dexterous manipulator on pages 19-20 appears to indicate that the object positions can be defined using x, y, and z coordinates (i.e. indicating a three-dimensional position)).
With respect to claims 3, 13, and 18, Yu in view of Pack teaches all of the limitations of claims 1, 11, and 16 as previously discussed, and Yu further teaches wherein the prompt includes a function definition with parameters, a docstring describing functionality of the parameters of the function, and a request to extend the function with a body implementation of the function (e.g. see Reward Coder prompts for Quadruped on pages 15-16 and Dexterous Manipulator on pages 18-19, where each includes function definitions with parameters, docstrings describing the functionality of the parameters of the function and a request to turn the description of the robot’s motion into a corresponding program using the functions (analogous to a request to extend the functions with a body implementation of the function)).
With respect to claims 6, 14, and 19, Yu in view of Pack teaches all of the limitations of claims 1, 11, and 16 as previously discussed, and Yu further teaches wherein the prompt includes one or more examples of one or more valid functions for one or more tasks other than the target task, wherein the one or more examples are provided in source code form (e.g. see Reward Coder prompts for Quadruped on pages 15-16 and Dexterous Manipulator on pages 18-19, where each includes examples of the functions in source code form).
With respect to claims 7 and 15, and 20, Yu in view of Pack teaches all of the limitations of claims 1, 11, and 16 as previously discussed, and Yu further teaches wherein the prompt is a second prompt (e.g. as shown in Fig. 2, the Reward Coder Prompt is a second of two prompts), further comprising:
generating a first prompt for a large language model at least in part by combining the natural language description of the target task and the natural language description of the target environment, wherein the first prompt requests one or more goal positions to use in training the policy for the autonomous machine to perform the target task (e.g. page 4, fourth and fifth paragraphs describing combination of user natural language input and designed template/prompt (Motion Descriptor Prompt as shown in Fig. 2), which includes both the natural language description of the target task along with natural language information about the environment, such as a prompt for make the robot dog stand up on two feet along with the prompt/template to fill in based on this information; as shown in Motion Descriptor Prompt for Dexterous Manipulator on page 18, this prompt can also include a description of the target environment (such as descriptions of objects contained in the environment) along with definitions of how to perform tasks in terms of goal positions (such as moving manipulator’s palm close to another object’s position, rotating an object by a given number of degrees, lifting an object to a given height, opening or closing an object, etc.));
generating the goal position by prompting a large language model with the first prompt (e.g. page 4, fourth and fifth paragraphs, Motion Descriptor completes the provided template to produce a natural language description of the desired robot motion, as shown in Fig. 2, i.e. the produced description of the Motion Descriptor);
wherein the second prompt references the goal position (e.g. as shown in Fig. 2, Reward Coder Prompt includes a reference to the natural language description of the desired robot motion, i.e. “We have a description of a robot’s motion and we want you to turn that into the corresponding program…”).
With respect to claim 4, Yu in view of Pack teaches all of the limitations of claim 1 as previously discussed. Pack further teaches the method further comprising validating the function at least in part by prompting a large language model for tests to validate the function (e.g. paragraph 0012, determining from prompt, composable tasks that can be used to generate output responsive to the prompt input by the user; decomposing the prompt into any number of tasks and generating text based prompts to be input into LLM used by selected tasks; paragraph 0021-0022, prompt to LLM requesting generation of application or software component; subtask testing the computer code, including subtasks to generate and perform tests on the output; output from the LLM validated, such as performing test on the LLM output; subtask to generate tests to test output that is computer code including syntax tests for the programming language of the computer code, security tests, and other debugging tests including input and output validation; i.e. based on a user’s initial prompt to generate computer code, the system may generate additional prompts corresponding to subtasks, including a subtask to generate tests to validate the LLM-produced code).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Yu and Pack in front of him to have modified the teachings of Yu (directed to using LLMs to generate executable reward code for learning robot policies), to incorporate the teachings of Pack (directed to automation based on prompting LLMs) to include the capability to generate an additional prompt for a determined subtask of the reward code generation task (i.e. of Yu), where the determined subtask is for testing the generated code/functions and the prompt is therefore to request generation of tests to validate the generated code/functions (as taught by Pack). One of ordinary skill would have been motivated to perform such a modification in order to enable automation using LLMs, allowing larger jobs to be decomposed into and accomplished with composable asynchronous tasks that use LLMs and GPTs as described in Pack (paragraph 0010).
With respect to claim 5, Yu in view of Pack teaches all of the limitations of claim 4 as previously discussed, and Pack further teaches the method further comprising correcting the function when said validating identifies an issue at least in part by prompting a large language model for a correction, wherein prompting the large language model for the correction includes providing, to the large language model, the function and information about the issue (e.g. paragraph 0023, generated computer code fails tests run against it, and task regenerates the computer code by using the same inputs, i.e. using the same prompt input by the task into the LLM, the prompt being modified based on the test that the output failed; after computer code generated by LLM fails a test, prompt input modified to indicate which test the computer code failed, allowing the LLM to generate computer code that does not fail that test).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Yu and Pack in front of him to have modified the teachings of Yu (directed to using LLMs to generate executable reward code for learning robot policies), to incorporate the teachings of Pack (directed to automation based on prompting LLMs) to include the capability to, when validation of the generated function/computer code identifies an issue, such as via a failed test, generate an additional prompt requesting that the code/function be regenerated, the prompt including information indicating which test the code failed (i.e. information identifying the function/code and indicating an issue with the function/code, such as that it failed a test). One of ordinary skill would have been motivated to perform such a modification in order to enable automation using LLMs, allowing larger jobs to be decomposed into and accomplished with composable asynchronous tasks that use LLMs and GPTs as described in Pack (paragraph 0010).
Claims 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Yu in view of Pack, further in view of Kunz (US 20250258659 A1).
With respect to claim 8, Yu in view of Pack teaches all of the limitations of claim 6 as previously discussed, and Yu further teaches wherein different examples are used to generate different functions for at least two different target tasks including said target task (e.g. as shown in the Reward Coder Prompt for Quadruped and Reward Coder Prompt for Dexterous Manipulator on pages 15-17 and 18-19, different examples can be included in the prompt for generating different functions for different tasks, such as examples of how to move and position the robot, or examples of the robot manipulating other objects in the environment).
Yu and Pack do not explicitly disclose searching an existing code repository to find the one or more examples based at least in part on the natural language description of the target task. However, Kunz teaches searching an existing code repository to find the one or more examples based at least in part on the natural language description of the target task (e.g. paragraph 0014, given input code, identifying code artifacts such as functions, variables, etc., and then searching repository of code files for declarations, definitions, and references related to those code artifacts, these declarations, definitions, and references then passed as context to the LLM; paragraph 0037, indicating that the term “artifacts” refers to distinctly delineated snippets of code including functions, routines, subroutines, variables, constants, etc.; paragraph 0046, describing Fig. 2, step 210, command to locate code snippets stored in code repository that are relevant to identified artifacts; step 212, identifications of the relevant code snippets received; paragraph 0047, generating prompt to LLM, the prompt including instructions to generate code based on using the identified artifacts as context; i.e. where the prompt may include a code artifact related to the computational task the user wishes to perform, and relevant examples/code snippets may be found in a repository and passed back to the LLM as context for generating code in response to the prompt).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Yu, Pack, and Kunz in front of him to have modified the teachings of Yu (directed to using LLMs to generate executable reward code for learning robot policies) and Pack (directed to automation based on prompting LLMs), to incorporate the teachings of Kunz (directed to context provisioning for code generation with large language models) to include the capability to receive, as part of the input including the natural language description of the target task (as taught by Yu) additional context information, including code context information for the target task, and to search a code repository to find relevant examples/code snippets (as taught by Kunz). One of ordinary skill would have been motivated to perform such a modification in order to provide context for LLM code generation requests, resulting in more reliable code generation by the LLM as described in Kunz (paragraph 0011, 0014).
With respect to claim 9, Yu in view of Pack teaches all of the limitations of claim 1 as previously discussed, and Yu further teaches
wherein the prompt includes an example of a task-independent portion of another function (e.g. as shown in the Reward Coder Prompt for Quadruped and Reward Coder Prompt for Dexterous Manipulator on pages 15-17 and 18-19, different examples can be included in the prompt for generating different functions for different tasks, such as examples of how to move and position the robot, or examples of the robot manipulating other objects in the environment; moreover, it is noted that the provided examples include both examples of functions for performing the specified task (i.e. functions for setting torso target positions, feet position parameters, and stepping parameters in the Reward Coder Prompt for Quadruped and functions for setting object distances, setting object orientations, setting object positions, setting joints to values, etc. in the Reward Coder Prompt for Dexterous Manipulator; Examiner notes that these appear to be functions for instructing the robot to carry out the specified task and are therefore analogous to task-dependent portions of the specified functions (that is, functions that are included because they are required for the particular task)), and examples of functions which are not necessarily specific to the specified task (i.e. a function for resetting a reward, and a function for sending parameters to the robot to execute the plan; Examiner notes that these functions appear to be necessary to the execution of any task, but do not receive specific values corresponding to the user-specified task, and are therefore analogous to task-independent portions of other functions (that is, functions that would be included regardless of the particular task));
wherein the prompt requests that the large language model include an explicit reference to the task-independent portion of the other function in the function (e.g. as shown in the Reward Coder Prompt for Quadruped and Reward Coder Prompt for Dexterous Manipulator on pages 15-17 and 18-19, the prompts include additional context explicitly requesting “Don’t forget to call execute_plan at the end” and “execute_plan should be called exactly once at the end”; i.e. where the function execute_plan appears to be necessary to have the robot actually execute the plan, but does not have task-specific details, and therefore appears to be analogous to a task-independent portion of a function, and the prompts include instructions to explicitly include/reference this function in the generated code).
Yu and Pack do not explicitly disclose wherein the task-independent portion of the other function is stored in a repository with other task-independent portions of a plurality of functions and selected based at least in part on the natural language description of the target task and one or more characteristics of the task-independent portion. However, Kunz teaches wherein the task-independent portion of the other function is stored in a repository with other task-independent portions of a plurality of functions and selected based at least in part on the natural language description of the target task and one or more characteristics of the task-independent portion (e.g. paragraph 0014, given input code, identifying code artifacts such as functions, variables, etc., and then searching repository of code files for declarations, definitions, and references related to those code artifacts, these declarations, definitions, and references then passed as context to the LLM; paragraph 0037, indicating that the term “artifacts” refers to distinctly delineated snippets of code including functions, routines, subroutines, variables, constants, etc.; paragraph 0046, describing Fig. 2, step 210, command to locate code snippets stored in code repository that are relevant to identified artifacts; step 212, identifications of the relevant code snippets received; paragraph 0047, generating prompt to LLM, the prompt including instructions to generate code based on using the identified artifacts as context; i.e. where the prompt may include a code artifact related to the computational task the user wishes to perform, and relevant examples/code snippets may be found in a repository and passed back to the LLM as context for generating code in response to the prompt).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Yu, Pack, and Kunz in front of him to have modified the teachings of Yu (directed to using LLMs to generate executable reward code for learning robot policies) and Pack (directed to automation based on prompting LLMs), to incorporate the teachings of Kunz (directed to context provisioning for code generation with large language models) to include the capability to receive, as part of the input including the natural language description of the target task (as taught by Yu) additional context information, including code context information for the target task, and to search a code repository to find and select relevant examples/code snippets (as taught by Kunz). One of ordinary skill would have been motivated to perform such a modification in order to provide context for LLM code generation requests, resulting in more reliable code generation by the LLM as described in Kunz (paragraph 0011, 0014).
With respect to claim 10, Yu in view of Pack teaches all of the limitations of claim 1 as previously discussed, and Yu further teaches
wherein the prompt includes an example of a task-dependent portion of another function (e.g. as shown in the Reward Coder Prompt for Quadruped and Reward Coder Prompt for Dexterous Manipulator on pages 15-17 and 18-19, different examples can be included in the prompt for generating different functions for different tasks, such as examples of how to move and position the robot, or examples of the robot manipulating other objects in the environment; moreover, it is noted that the provided examples include both examples of functions for performing the specified task (i.e. functions for setting torso target positions, feet position parameters, and stepping parameters in the Reward Coder Prompt for Quadruped and functions for setting object distances, setting object orientations, setting object positions, setting joints to values, etc. in the Reward Coder Prompt for Dexterous Manipulator; Examiner notes that these appear to be functions for instructing the robot to carry out the specified task and are therefore analogous to task-dependent portions of the specified functions (that is, functions that are included because they are required for the particular task)));
wherein the prompt requests that the large language model use the task-dependent portion as an example without including, in the function to be generated based on the prompt, the task-dependent portion of the other function and without including, in the function to be generated based on the prompt, a reference to the task-dependent portion of the other function (e.g. as shown in the Reward Coder Prompt for Quadruped and Reward Coder Prompt for Dexterous Manipulator on pages 15-17 and 18-19, the prompts include additional context explicitly placing additional limitations on the code generation, including “do not invent new functions or classes. the only allowed functions you can call are the ones listed above. do not leave unimplemented code blocks in your response. do not calculate the position or direction of any object (except for the ones provided above)…you do not need to make the robot do extra things not mentioned in the plan such as stopping the robot..” etc.; i.e. where these additional instructions, taken in conjunction with provided code examples of the prompts, appear to indicate that the task-dependent portions should be used as an example without including (for generating responsive code/functions) task-dependent portions of any other functions or references to task-dependent portions of any other functions; that is, even though the prompt may include examples of multiple different tasks, in generating code for a given task, only the example for that task is to be used as an example, and no functions or references to other functions (i.e. specific to other tasks) should be included in the generated code/function (i.e. for the Quadruped, only generate code for changing robot positions based on the provided example, or for the Manipulator, only generate code for changing object positions and orientations, etc. based on the provided example, and do not include any other functions/references to functions for other tasks, including not leaving unimplemented blocks of code in the response, not performing additional functions such as calculating positions or directions of other objects, or to make the robot perform extra actions such as stopping, etc.).
Yu and Pack do not explicitly disclose wherein the task- dependent portion of the other function is stored in a repository with other task-dependent portions of a plurality of functions and selected based at least in part on the natural language description of the target task and one or more characteristics of the task- dependent portion. However, Kunz teaches wherein the task- dependent portion of the other function is stored in a repository with other task-dependent portions of a plurality of functions and selected based at least in part on the natural language description of the target task and one or more characteristics of the task- dependent portion (e.g. paragraph 0014, given input code, identifying code artifacts such as functions, variables, etc., and then searching repository of code files for declarations, definitions, and references related to those code artifacts, these declarations, definitions, and references then passed as context to the LLM; paragraph 0037, indicating that the term “artifacts” refers to distinctly delineated snippets of code including functions, routines, subroutines, variables, constants, etc.; paragraph 0046, describing Fig. 2, step 210, command to locate code snippets stored in code repository that are relevant to identified artifacts; step 212, identifications of the relevant code snippets received; paragraph 0047, generating prompt to LLM, the prompt including instructions to generate code based on using the identified artifacts as context; i.e. where the prompt may include a code artifact related to the computational task the user wishes to perform, and relevant examples/code snippets may be found in a repository and passed back to the LLM as context for generating code in response to the prompt).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Yu, Pack, and Kunz in front of him to have modified the teachings of Yu (directed to using LLMs to generate executable reward code for learning robot policies) and Pack (directed to automation based on prompting LLMs), to incorporate the teachings of Kunz (directed to context provisioning for code generation with large language models) to include the capability to receive, as part of the input including the natural language description of the target task (as taught by Yu) additional context information, including code context information for the target task, and to search a code repository to find and select relevant examples/code snippets (as taught by Kunz). One of ordinary skill would have been motivated to perform such a modification in order to provide context for LLM code generation requests, resulting in more reliable code generation by the LLM as described in Kunz (paragraph 0011, 0014).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain,” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting in re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (GCPA 1968)). Further, a reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co, v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert, denied, 493 U.S. 975 (1989). See also Upsher-Smith Labs. v. Pamlab, LLC, 412 F,3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir, 2005): Celeritas Technologies Ltd. v. Rockwell International Corp., 150 F.3d 1354, 1361, 47 USPQ2d 1516, 1522-23 (Fed. Cir. 1998).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEREMY L STANLEY whose telephone number is (469)295-9105. The examiner can normally be reached on Monday-Friday from 9:00 AM to 5:00 PM CST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar, can be reached at telephone number (571) 270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR for authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/JEREMY L STANLEY/
Primary Examiner, Art Unit 2127
Prosecution Insights