DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is made non-final.
Claims 1-20 are pending. Claims 1, 12 and 18 are independent claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1:
Step 1: This part of the eligibility analysis evaluates whether the claim falls within any statutory category. See MPEP 2106.03. Claim 1 recites: A method comprising… Claim 1 is directed to a method. (Step 1: YES).
Step 2A prong 1: Does the claim recite a judicial exception? Claim 1 recites: updating at least one first value of at least one parameter or at least one hyperparameter associated with one or more first machine learning models included in the plurality of machine learning models based at least on at least one second value of the at least one parameter or the at least one hyperparameter associated with one or more second machine learning models included in the plurality of machine learning models… Updating a parameter based on another parameter is a mental process, i.e., copying the value (and possibly involves mathematical calculations). These steps can be performed mentally or are mathematical calculations (Step 2A prong 1: YES).
Step 2A prong 2: Does the claim recite additional elements? Do those additional elements, considered individually and in combination, integrate the judicial exception into a practical application? Claim 1 recites: performing one or more operations to train a plurality of machine learning models to control at least a portion of a robot to perform a task… and subsequent to the updating, performing one or more additional operations to train the plurality of machine learning models to control at least the portion of the robot to perform the task. Performing one or more operations to train machine learning (ML) models is recited at a high level of generality and is an attempt to use the ML models by merely applying the abstract idea (i.e., perform the mental processes/math) without placing any limits on how they operate. Further, the claim omits any details as to how the neural network model solves a technical problem and instead recites only the idea of a solution or outcome. See MPEP 2106.05(f). Thus, the limitation represents no more than mere instructions to implement the abstract idea which is equivalent to adding the words “apply it” to the recited judicial exception. Specifying that the models are trained to control a robot to perform a task is an additional element(s) specifying a field of use without significantly more (Step 2A prong 2: NO).
Step 2B: These elements are recited at such a high level of generality that they fail to integrate the abstract idea into a practical application, since they provide nothing more than mere instructions to implement an abstract idea on a generic computer (MPEP 2106.05(f)) or limit the field of use without significantly more (MPEP 2106.05(h)). These limitations, taken either alone or in combination, fail to provide an inventive concept (Step 2B: NO). Thus, the claim is not patent eligible.
Regarding claims 2-11, they recite limitations which further narrow the abstract idea by specifying more details of the mental and mathematical process that occurs (Claim 2, identifying model performance in a percentage range is a mental process; Claim 3, similarly to claim 2, identifying model performance in a percentage range is a mental process; Claim 4, replacing parameters is a mental process; Claim 5, using reinforcement learning operations as the training operations is still recited at a high level of generality; Claim 6, specifying that the models are trained to perform tasks like reaching an object, picking the object up, etc., is an attempt to limit the field of use of the invention; Claim 7, using meta-learning based training operations is still recited at a high level of generality and provides nothing more than mere instructions to implement an abstract idea on a generic computer; Claim 8, training additional machine learning models is still recited at a high level of generality and provides nothing more than mere instructions to implement an abstract idea on a generic computer (MPEP 2106.05(f)); Claim 9, selecting a model based on performance is a mental process, and performing operations to perform a task using the model is insignificant extra-solution activity of data outputting without significantly more; Claim 10, specifying that the models are trained to perform tasks like regrasping an object, throwing an object, etc., is an attempt to limit the field of use of the invention; Claim 11, specifying that the processor is part of a variety of systems is an attempt to limit the field of use of the invention).Regarding claim 12:
Step 1: This part of the eligibility analysis evaluates whether the claim falls within any statutory category. See MPEP 2106.03. Claim 12 recites: A method comprising… Claim 12 is directed to a method. (Step 1: YES).
Step 2A prong 1: Does the claim recite a judicial exception? Claim 12 recites: updating at least one first value of at least one parameter or at least one hyperparameter associated with one or more second machine learning models included in the plurality of machine learning models based at least on at least one second value of the at least one parameter or the at least one hyperparameter associated with one or more third machine learning models included in the plurality of machine learning models (updating a parameter based on another parameter is a mental process, i.e., copying the value)… and selecting the first machine learning model from the plurality of machine learning models (selecting a model is a mental process). These steps can be performed mentally or are mathematical calculations (Step 2A prong 1: YES).
Step 2A prong 2: Does the claim recite additional elements? Do those additional elements, considered individually and in combination, integrate the judicial exception into a practical application? Claim 12 recites: receiving sensor data associated with a robot, generating an action based at least on the sensor data and a first machine learning model; and controlling at least a portion of the robot to perform a task based on the action, wherein the first machine learning model was trained by: performing one or more operations to train a plurality of machine learning models to control at least the portion of the robot to perform the task… subsequent to the updating, performing one or more additional operations to train the plurality of machine learning models to control at least the portion of the robot to perform the task… Performing one or more operations to train ML models is recited at a high level of generality and is an attempt to use the ML models by merely applying the abstract idea (i.e., perform the mental processes/math) without placing any limits on how they operate. Further, the claim omits any details as to how the neural network model solves a technical problem and instead recites only the idea of a solution or outcome. See MPEP 2106.05(f). Thus, the limitation represents no more than mere instructions to implement the abstract idea which is equivalent to adding the words “apply it” to the recited judicial exception. Specifying that the models are trained to control a robot to perform a task is an additional element(s) specifying a field of use without significantly more. Receiving sensor data associated with a robot is insignificant extra-solution activity of data gathering. Generating an action based on input data and a model, and controlling a robot based on that action is insignificant extra-solution activity of data outputting (Step 2A prong 2: NO).
Step 2B: These elements are recited at such a high level of generality that they fail to integrate the abstract idea into a practical application, since they provide nothing more than mere instructions to implement an abstract idea on a generic computer (MPEP 2106.05(f)), limit the field of use without significantly more (MPEP 2106.05(h)), or only amount to data gathering or outputting without significantly more (MPEP 2106.05(g)). These limitations, taken either alone or in combination, fail to provide an inventive concept (Step 2B: NO). Thus, the claim is not patent eligible.
Regarding claims 13-17, they recite limitations which further narrow the abstract idea by specifying more details of the mental and mathematical process that occurs (Claim 13, determining model performance in a percentage range is a mental process, Claim 14, using reinforcement learning operations as the training operations is still recited at a high level of generality and provides nothing more than mere instructions to implement an abstract idea on a generic computer (MPEP 2106.05(f)); Claim 15, performing simulations in parallel on GPUs is still recited at a high level of generality, i.e., applying the models on generic computing components; Claim 16, training operations based on meta-optimization of a reward associated with reaching an object, picking it up, etc., is limiting the field of use without significantly more, and performing general reinforcement learning (i.e., reward-based) operations provides nothing more than mere instructions to implement an abstract idea on a generic computer (MPEP 2106.05(f)); Claim 17, training additional machine learning models is still recited at a high level of generality).
Regarding claim 18:
Step 1: This part of the eligibility analysis evaluates whether the claim falls within any statutory category. See MPEP 2106.03. Claim 18 recites: A system comprising one or more processors… Claim 18 is directed to an apparatus. (Step 1: YES).
Step 2A prong 1: Does the claim recite a judicial exception? Claim 18 recites: using a machine learning model trained based at least on one or more population-based training operations and one or more reinforcement learning operations. Population-based training and reinforcement learning operations comprise mathematical calculations, for example, to calculate parameter updates. These steps are mathematical calculations (Step 2A prong 1: YES).
Step 2A prong 2: Does the claim recite additional elements? Do those additional elements, considered individually and in combination, integrate the judicial exception into a practical application? Claim 18 recites: to control at least a portion of a robot. Specifying that the model is used is used to control at least a portion of a robot to perform a task is an additional element(s) specifying a field of use (i.e., robot control) without significantly more (Step 2A prong 2: NO).
Step 2B: These elements are recited at such a high level of generality that they fail to integrate the abstract idea into a practical application, since they only limit the field of use without significantly more (MPEP 2106.05(h)). These limitations, taken either alone or in combination, fail to provide an inventive concept (Step 2B: NO). Thus, the claim is not patent eligible.
Regarding claims 19 and 20, they recite limitations which further narrow the abstract idea by specifying more details of the mental and mathematical process that occurs (Claim 19, training multiple models is recited at a high level of generality, i.e., applying the abstract idea, without placing any limits on how the neural network model operates and omitting details as to how the neural network model solves a technical problem and instead recites only the idea of a solution or outcome (see MPEP 2106.05(f)); Claim 20, specifying that the processors control the robot to perform tasks like reaching an object, picking the object up, etc., is an attempt to limit the field of use of the invention without significantly more).
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 2, 4, 5, 7-9, 11-14, 17 and 18-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Jaderberg et al. (US 20210004676 A1), herein Jaderberg.
Regarding claim 1, Jaderberg teaches: A method comprising: performing one or more operations to train a plurality of machine learning models to control at least a portion of a robot to perform a task (¶40, The network may be used to control agent interacting in an environment, for example, a robot operating in a warehouse or an autonomous vehicle operating in the real world); updating at least one first value of at least one parameter or at least one hyperparameter associated with one or more first machine learning models included in the plurality of machine learning models based at least on at least one second value of the at least one parameter or the at least one hyperparameter associated with one or more second machine learning models included in the plurality of machine learning models (¶116, If the candidate neural network is ranked below a certain threshold (e.g., bottom 20% of all candidate neural networks), the system 100 samples a candidate neural network ranked above a certain threshold (e.g., top 20% of all candidate neural networks) and sets the new network parameters and new hyperparameters of the candidate neural network to the values of the sampled candidate neural network); and subsequent to the updating, performing one or more additional operations to train the plurality of machine learning models to control at least the portion of the robot to perform the task (¶96, the system repeats the process 200 for the candidate neural network – the referenced “process 200” that is repeated includes training – see fig. 2).
Regarding claim 2, Jaderberg teaches: The method of claim 1, wherein the one or more first machine learning models include a predefined percentage of worst performing machine learning models in the plurality of machine learning models, and the one or more second machine learning models include a predefined percentage of best performing machine learning models in the plurality of machine learning model (¶116, If the candidate neural network is ranked below a certain threshold (e.g., bottom 20% of all candidate neural networks), the system 100 samples a candidate neural network ranked above a certain threshold (e.g., top 20% of all candidate neural networks) and sets the new network parameters and new hyperparameters of the candidate neural network to the values of the sampled candidate neural network).
Regarding claim 4, Jaderberg teaches: The method of claim 1, wherein the updating the at least one first value of the at least one parameter or the at least one hyperparameter associated with the one or more first machine learning models comprises replacing the at least one value of the at least one parameter or the at least one hyperparameter associated with the one or more first machine learning models with the at least one second value of the at least one parameter or the at least one hyperparameter associated with the one or more second machine learning models (¶116, If the candidate neural network is ranked below a certain threshold (e.g., bottom 20% of all candidate neural networks), the system 100 samples a candidate neural network ranked above a certain threshold (e.g., top 20% of all candidate neural networks) and sets the new network parameters and new hyperparameters of the candidate neural network to the values of the sampled candidate neural network).
Regarding claim 5, Jaderberg teaches: The method of claim 1, wherein the one or more operations to train the plurality of machine learning models comprise one or more reinforcement learning operations (¶111, In one example, a reinforcement learning task is specified as the machine learning task for the system. Each candidate neural network receives inputs and generates outputs that conform to a deep reinforcement learning task).
Regarding claim 7, Jaderberg teaches: The method of claim 1, wherein the one or more operations to train the plurality of machine learning models are based at least on meta-optimization of an objective (¶69, the population based neural network training system 100 additionally performs meta-optimization, where the hyperparameters and network parameters for each candidate neural network 120A-N are additionally adapted according to the performance of the entire population).
Regarding claim 8, Jaderberg teaches: The method of claim 1, wherein the performing the one or more operations to train the plurality of machine learning models comprises: performing one or more operations to train a third machine learning model included in the plurality of machine learning models; and performing one or more operations to train a fourth machine learning model included in the plurality of machine learning models, wherein the one or more operations to train the third machine learning model begin at a different time than the one or more operations to train the fourth machine learning model (¶96, the system repeats the process 200 for the candidate neural network – an iterative process of training and evaluating different candidate models involves training different models at different time points).
Regarding claim 9, Jaderberg teaches: The method of claim 1, further comprising, subsequent to the performing the one or more additional operations: selecting a third machine learning model included in the plurality of machine learning models based on a performance of the third machine learning model (¶21, selecting the trained values of the network parameters from the parameter values in the maintained data based on the maintained quality measures for the candidate neural networks after the training operations have repeatedly been performed – and – ¶23, The method may further comprise providing the trained values of the network parameters for use in processing new inputs to the neural network); and performing one or more operations to control at least the portion of the robot to perform the task using the third machine learning model (¶40, The network may be used to control agent interacting in an environment, for example, a robot operating in a warehouse or an autonomous vehicle operating in the real world).
Regarding claim 11, Jaderberg teaches: The method of claim 1, wherein the method is performed by a processor comprised in at least one of: an infotainment system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device (¶49, the system 100 can output (e.g., by outputting to a user device or by storing in memory accessible to the system) the trained values of the network parameters of the trained neural network 150 for later use in processing inputs using the trained neural network 150); a system implemented using the robot (¶40, The network may be used to control agent interacting in an environment, for example, a robot operating in a warehouse or an autonomous vehicle operating in the real world); a system for generating or presenting virtual reality, augmented reality, or mixed reality content; a system for performing conversational AI operations; a system implementing one or more large language models (LLMs); a system for generating synthetic data (¶126, In some implementations, a generative adversarial network (GAN) task is specified as the machine learning task for the population based neural network training system); a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.
Regarding claim 12, Jaderberg teaches: A method comprising: receiving sensor data associated with a robot; generating an action based at least on the sensor data and a first machine learning model; and controlling at least a portion of the robot to perform a task based on the action (¶15, As another example, if the input to the neural network is data characterizing the state of an environment being interacted with by an agent, e.g., a robot or other mechanical agent, the output generated by the neural network can be a policy output that defines a control input for the agent. For example, the output can include or define a respective probability for each action in a set of possible actions to be performed by the agent or a respective Q value, i.e., a return estimate, for each action in the set of possible actions), wherein the first machine learning model was trained by: performing one or more operations to train a plurality of machine learning models to control at least the portion of the robot to perform the task, updating at least one first value of at least one parameter or at least one hyperparameter associated with one or more second machine learning models included in the plurality of machine learning models based at least on at least one second value of the at least one parameter or the at least one hyperparameter associated with one or more third machine learning models included in the plurality of machine learning models (¶116, If the candidate neural network is ranked below a certain threshold (e.g., bottom 20% of all candidate neural networks), the system 100 samples a candidate neural network ranked above a certain threshold (e.g., top 20% of all candidate neural networks) and sets the new network parameters and new hyperparameters of the candidate neural network to the values of the sampled candidate neural network), subsequent to the updating, performing one or more additional operations to train the plurality of machine learning models to control at least the portion of the robot to perform the task (¶40, The network may be used to control agent interacting in an environment, for example, a robot operating in a warehouse or an autonomous vehicle operating in the real world), and selecting the first machine learning model from the plurality of machine learning models (¶21, selecting the trained values of the network parameters from the parameter values in the maintained data based on the maintained quality measures for the candidate neural networks after the training operations have repeatedly been performed – and – ¶23, The method may further comprise providing the trained values of the network parameters for use in processing new inputs to the neural network).
Regarding claim 13, Jaderberg teaches: The method of claim 12, wherein the one or more second machine learning models include a predefined percentage of worst performing machine learning models in the plurality of machine learning models, and the one or more third machine learning models include a predefined percentage of best performing machine learning models in the plurality of machine learning models (¶116, If the candidate neural network is ranked below a certain threshold (e.g., bottom 20% of all candidate neural networks), the system 100 samples a candidate neural network ranked above a certain threshold (e.g., top 20% of all candidate neural networks) and sets the new network parameters and new hyperparameters of the candidate neural network to the values of the sampled candidate neural network).
Regarding claim 14, Jaderberg teaches: The method of claim 12, wherein the one or more operations to train the plurality of machine learning models comprise one or more reinforcement learning operations (¶111, In one example, a reinforcement learning task is specified as the machine learning task for the system. Each candidate neural network receives inputs and generates outputs that conform to a deep reinforcement learning task).
Regarding claim 17, Jaderberg teaches: The method of claim 12, wherein the performing the one or more operations to train the plurality of machine learning models comprises: performing one or more operations to train a fourth machine learning model included in the plurality of machine learning models; and performing one or more operations to train a fifth machine learning model included in the plurality of machine learning models, wherein the one or more operations to train the fourth machine learning model begin at a different time than the one or more operations to train the fifth machine learning model (¶96, the system repeats the process 200 for the candidate neural network – an iterative process of training and evaluating different candidate models involves training different models at different time points).
Regarding claim 18, Jaderberg teaches: A system comprising: one or more processors to control at least a portion of a robot using a machine learning model trained based at least on one or more population-based training operations (¶45, FIG. 1 shows an example population based neural network training system) and one or more reinforcement learning operations (¶111, In one example, a reinforcement learning task is specified as the machine learning task for the system. Each candidate neural network receives inputs and generates outputs that conform to a deep reinforcement learning task).
Regarding claim 19, Jaderberg teaches: The system of claim 18, wherein the machine learning was trained by performing operations that comprise updating at least one first value of at least one first parameter or at least one first hyperparameter associated with one or more first machine learning models included in a plurality of machine learning models based at least on at least one second value of at least one second parameter or at least one second hyperparameter associated with one or more second machine learning models included in the plurality of machine learning models (¶116, If the candidate neural network is ranked below a certain threshold (e.g., bottom 20% of all candidate neural networks), the system 100 samples a candidate neural network ranked above a certain threshold (e.g., top 20% of all candidate neural networks) and sets the new network parameters and new hyperparameters of the candidate neural network to the values of the sampled candidate neural network).
Regarding claim 20, Jaderberg teaches: The system of claim 18, wherein the one or more processors control at least the portion of the robot to at least one of reach an object, pick up the object, manipulate the object, move the object, or move within an environment (¶40, The network may be used to control agent interacting in an environment, for example, a robot operating in a warehouse or an autonomous vehicle operating in the real world).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jaderberg in view of Liang et al. (“Regularized Evolutionary Population-Based Training”, 2020), herein Liang.
Regarding claim 3, Jaderberg teaches: The method of claim 1… a predefined percentage of… a predefined percentage of (¶116, If the candidate neural network is ranked below a certain threshold (e.g., bottom 20% of all candidate neural networks), the system 100 samples a candidate neural network ranked above a certain threshold (e.g., top 20% of all candidate neural networks))…
Jaderberg fails to explicitly teach: wherein the one or more first machine learning models include… machine learning models in the plurality of machine learning models whose performance is neither worst nor best in the plurality of machine learning models, and the one or more first (Examiner note: is this meant to read “second”?) machine learning models include… best performing machine learning models in the plurality of machine learning models. The interpretation of this is that the first group contains models that are the best and not the best, i.e., 30th to the 99th percentile, but it does not prevent models with lower performance levels from being included in the first group.
However, in the same field of endeavor, Liang teaches: wherein the one or more first machine learning models include… machine learning models in the plurality of machine learning models whose performance is neither worst nor best in the plurality of machine learning models, and the one or more first machine learning models include… best performing machine learning models in the plurality of machine learning models (pg. 4, Section 3.2, ¶4, For each 𝑀𝑔𝑖, a uniform mutation operator 𝛾 is applied by introducing multiplicative Gaussian noise independently to each variable in h𝑔𝑖. The mutation operator can randomly and independently reinitialize every variable as well. This approach allows for the exploration of novel combinations of hyperparameters. After mutation, a uniform crossover operator 𝜉 is applied, where each variable in h𝑔𝑖 is randomly swapped (50% probability) with the same variable from another individual in M𝑔, resulting in the creation of h𝑔𝑖 – i.e., good models are “bred” with other good models, so high performing models update each other).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include high and average performing models in the group of models to be updated as disclosed by Liang in the method disclosed by Jaderberg to find promising new parameter sets efficiently (pg. 1, Section 1, ¶4, to discover promising combinations of hyperparameters for DNN training. In particular, EPBT uses selection, mutation, and crossover operators adapted from genetic algorithms [53] to find good solutions).
Claim(s) 6, 10 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jaderberg in view of Vogelsong et al. (US 10926408 B1), herein Vogelsong.
Regarding claim 6, Jaderberg teaches: The method of claim 1, wherein the one or more operations to train the plurality of machine learning models are based at least on a reward (¶111, Each candidate neural network receives inputs and generates outputs that conform to a deep reinforcement learning task. In reinforcement learning, the aim is to find a policy to maximize expected episodic return within an environment… ¶113, the quality measure of the candidate neural network is the mean value of the pre-determined number of previous episodic rewards).
Jaderberg fails to teach: associated with at least one of reaching an object, picking up the object, or bringing the object to a location.
However, in the same field of endeavor, Vogelsong teaches: associated with at least one of reaching an object, picking up the object, or bringing the object to a location (Col. 6, line 42, FIGS. 1A and 1B depict throwing an object as one example of a robotically-performed task for which a control policy can be generated using machine learning techniques as described herein… Other example real-world tasks include… transferring physical objects to and from storage structures… lifting objects – and – Col. 9, line 37, a reward function or other function suitable for programmatic evaluation of task performance success. This function can be used to guide policy updates via reinforcement learning).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use rewards based on lifting an object or bringing an object to a location as disclosed by Vogelsong in the method disclosed by Jaderberg to create adaptable, effective robot control policies that may be difficult or time consuming for a human to come up with (Col. 2, line 53, By using targeted updates to machine-learned policies to control robotic task performance, the present technology is able to achieve levels of robustness, accuracy, and flexibility not available by traditional methods. As an example, a machine learned robotic control policy may yield the capability to perform tasks that a human cannot figure out or imagine, for example an autopilot control policy that can recover from stall).
Regarding claim 10, Jaderberg fails to teach: The method of claim 1, wherein the task includes at least one of regrasping an object, throwing an object, or reorienting an object with one or two arms of the robot.
However, in the same field of endeavor, Vogelsong teaches: wherein the task includes at least one of regrasping an object, throwing an object, or reorienting an object with one or two arms of the robot (Col. 6, line 42, FIGS. 1A and 1B depict throwing an object as one example of a robotically-performed task for which a control policy can be generated using machine learning techniques as described herein – and line 56, flipping a bottle).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to train the model(s) to perform tasks such as throwing or reorienting an object as disclosed by Vogelsong in the method disclosed by Jaderberg to create adaptable, effective robot control policies that may be difficult or time consuming for a human to come up with (Col. 2, line 53, By using targeted updates to machine-learned policies to control robotic task performance, the present technology is able to achieve levels of robustness, accuracy, and flexibility not available by traditional methods. As an example, a machine learned robotic control policy may yield the capability to perform tasks that a human cannot figure out or imagine, for example an autopilot control policy that can recover from stall).
Regarding claim 16, Jaderberg teaches: The method of claim 12, wherein the one or more operations to train the plurality of machine learning models are based at least on at least one of meta-optimization of an objective or optimization of a reward (¶111, Each candidate neural network receives inputs and generates outputs that conform to a deep reinforcement learning task. In reinforcement learning, the aim is to find a policy to maximize expected episodic return within an environment… ¶113, the quality measure of the candidate neural network is the mean value of the pre-determined number of previous episodic rewards. The candidate neural network with the highest mean episodic reward has the highest quality measure and is considered the “best” in terms of measured fitness).
Jaderberg fails to teach: associated with at least one of reaching an object, picking up the object, or bringing the object to a location.
However, in the same field of endeavor, Vogelsong teaches: associated with at least one of reaching an object, picking up the object, or bringing the object to a location (Col. 6, line 42, FIGS. 1A and 1B depict throwing an object as one example of a robotically-performed task for which a control policy can be generated using machine learning techniques as described herein… Other example real-world tasks include… transferring physical objects to and from storage structures… lifting objects).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use rewards based on lifting an object or bringing an object to a location to create adaptable, effective robot control policies that may be difficult or time consuming for a human to come up with (Col. 2, line 53, By using targeted updates to machine-learned policies to control robotic task performance, the present technology is able to achieve levels of robustness, accuracy, and flexibility not available by traditional methods. As an example, a machine learned robotic control policy may yield the capability to perform tasks that a human cannot figure out or imagine, for example an autopilot control policy that can recover from stall).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jaderberg in view of Bischoff et al. (US 20210122038 A1), herein Bischoff.
Regarding claim 15, Jaderberg fails to teach: The method of claim 12, wherein the performing the one or more operations to train the plurality of machine learning models comprises performing one or more operations to simulate the robot in a plurality of simulations, and the plurality of simulations are performed in parallel via one or more graphics processing units (GPUs).
However, in the same field of endeavor, Bischoff teaches: wherein the performing the one or more operations to train the plurality of machine learning models comprises performing one or more operations to simulate the robot (¶16, A technical system may be in particular an autonomous technical system, such as for example an autonomous robot. Ascertaining a combination of action steps may be understood in particular as meaning planning a sequence of actions…) in a plurality of simulations, and the plurality of simulations are performed in parallel via one or more graphics processing units (GPUs) (¶38-39, In a further advantageous embodiment of the method, various combinations of action steps of the technical system may be simulated temporally in parallel on more than one computing unit… may be carried out in parallel, such as for example on graphics processors (graphics processing units, GPUs for short)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use a GPU to run robot simulations in parallel as disclosed by Bischoff in the method disclosed by Jaderberg to improve training speed (¶39, This allows in particular a quick and efficient calculation of a combination of favorable action steps).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Lilley (US 20190180181 A1), which discloses ranking models into performance based percentage ranges.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HARRISON CHAN YOUNG KIM whose telephone number is (571)272-0713. The examiner can normally be reached Monday - Thursday 9:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, CESAR PAULA can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HARRISON C KIM/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145