Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on February 17th, 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 1 and 4-5 are objected to because of the following informalities:
Claim 1 Lines 17-18: “outputted as action” should be revised to “outputted as an action”.
Claim 4 Line 2: “and the both policies” should be revised to “and both of the two policies”.
Claim 4 Line 2: “as basis” should be revised to “as a basis”.
Claim 5 Line 3: “are terminate” should be revised to “are terminated”.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, 5-7, 9-10 and 12-13 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Analysis of the claims in view of MPEP § 2106.04 are provided below.
Regarding Claim 1: A computer-implemented method of determining actions for controlling a robot, comprising:
receiving a first and second input, wherein the first input is a sentence describing a task of the robot, wherein the second input is a sensor output characterizing a state of an environment of the robot;
feeding the first and second input into a first and second machine learning model respectively, wherein the first and second machine learning models are configured to determine tokens for their respective inputs;
concatenating the determined tokens of the first and second machine learning models;
feeding the concatenated tokens into a third machine learning model, wherein the third machine learning model comprises two policies that are configured to output a skill action and a moving action respectively, wherein the skill action characterizes a categorization of different high-level action categories of the robot and the moving action is an explicit movement proposal for the robot; and
deciding based on the skill action whether the moving action is outputted as action or a more precise movement proposal for the robot than the moving action as the action is determined according to the high-level action category of the skill action from an external source.
Step 1: Statutory Category – Yes
The claim recites a process, which falls within one of the four statutory categories. MPEP § 2106.03.
Step 2A Prong One Evaluation: Judicial Exception – Yes
The Office submits that the foregoing bolded limitation(s) constitute judicial exceptions in terms of “mental processes” because under broadest reasonable interpretation, the claim covers performance using mental processes.
The claim recites limitations of receiving two inputs that are further fed into multiple layers of data processing models to result in a decision made regarding a type of action of the robot. These limitations as drafted, are simple processes that under their broadest reasonable interpretation, covers performance of the limitations in the mind without further reciting significant structure or application to classify as an inventive concept. Nothing in the claim precludes the elements of the limitations from being performed in the mind. For example, a person could receive two inputs consisting of a task description of a robot and a state of an environment of the robot respectively, purely through mental thoughts and observations. A person can also reasonably process the initial two inputs in multiple layers of complexity through a series of simple cognitive operations, and define the outputs of each layer in their mind. These are all processes that a person can execute reasonably in their mind, including a decision being made after the mental processing steps of the initial inputs is complete. Thus, this step recites a mental process.
Each of the limitations identified above falls within at least one of collecting and storing information, both of which fall within the bucket of “Mental Processes” of abstract ideas.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Evaluation: Practical Application – No
Claim 1 is evaluated whether as a whole it integrates the recited judicial exception into a practical application. As noted in MPEP § 2106.04, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer or generic component to implement an abstract idea, adding insignificant extra-solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application”.
In the present case, the additional limitations beyond the above-noted abstract ideas are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”).
The additional elements recited in Claim 1 do not integrate the judicial exception(s) into a practical application, and are merely using generic components (or even lacking generic components) to implement an abstract idea. Examiner notes the various “machine learning models” claimed are stated as generic models without providing additional function or structure to the method that would preclude the additional limitations from being performed in the mind. The involvement of these elements is nothing more than generally linking the abstract idea into a particular field of use. The claim is directed to an abstract idea.
Step 2B Evaluation: Inventive Concept – No
Claim 1 is evaluated as to whether the claims as a whole amount to significantly more than the recited exception (i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim).
As discussed with respect to Step 2A Prong Two, the additional elements are merely generally linking the abstract idea into a particular field of use. The same analysis applies here in 2B.
For these reasons, there is no inventive concept in the claim, and thus it is ineligible.
Regarding Claims 2, 5-7, 9 and 13, these claims generally only further limit the abstract idea by introducing additional steps that can be performed mentally. While some of these claims include additional elements not previously discussed, they are discussed in a substantially similar manner as the additional elements in Claim 1, such as additional sensory inputs, additional tokens, types of robots as additional elements, and generic computer programs. Therefore, similar analysis can be used that would arrive at the same conclusion that these claims are still ineligible.
Regarding Claim 12, other than falling under a different statutory category, the claim is merely directed towards a system with no supportive structure, other than claiming the functionality of the method in claim 1. Therefore, similar analysis can apply and these claims are also ineligible.
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because it is directed to “A computer program” which falls under “products that do not have a physical or tangible form, such as […] a computer program per se”. MPEP § 2106.03.
This rejection can be overcome by revising “A computer program […]” to “A non-transitory computer readable medium comprising a program […]”.
Examiner notes that physical operation of the robot as controlled by component(s) of the system performing the generic data gathering and processing steps is suggested as part of intended use language in at least Claims 1. In the event the claims are amended to positively recite the robot’s physical operation as implemented using the information acquired and decisions made, it will likely overcome the 101 rejection(s) of Claims 1-2, 5-7, 9 and 13 noted above.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1 (along with claims 2-14 due to dependency) is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The limitation of “deciding based on the skill action whether the moving action is outputted as action or a more precise movement proposal for the robot that the moving action […] from an external source” in claim 1 is a highly confusing limitation necessitating clarification regarding the decision process, thus rendering the claim indefinite. The Examiner does not understand how the decision is being made because of the lack of clarity in the claim as written, and unsure of what exactly is making the decision. Examiner further notes the inclusion of an “external source” seems irrelevant to the subject matter of the claim, but it is unclear whether or not this is the case as the rest of the limitation is also indefinite.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-6 and 8-14 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Toshev et al. (US Patent Pub. No. 2021/0397195 A1), herein “Toshev”.
Regarding Claim 1, Toshev discloses a computer-implemented method of determining actions for controlling a robot (See 0095, “[…] method of navigating a mobile robot in an environment […] controlling one or more actuators of the mobile robot based on the corresponding low-level action output to cause the mobile robot to implement the corresponding low-level action.”), comprising:
receiving a first and second input, wherein the first input is a sentence describing a task of the robot, wherein the second input is a sensor output characterizing a state of an environment of the robot (See 0075-0076, “[…] identifies a target label for a navigation target in the environment. The target label can be a semantically meaningful one hot vector, a word embedding of a semantic descriptor of a navigation target, a target label that is an image embedding of an image of a navigation target, and/or other target label that provides semantic meaning for the navigation target. The target label can be generated based on user interface input and/or based on output from a higher-level task planner that identifies the navigation target. For example, a target label for a “trash can” can be generated based on spoken user interface input of “navigate to the trash can”. For instance, the target label can be based on an image of a “trash can” identified based on the spoken user interface input and/or based on a word embedding of “trash can” […] obtains current observation data based on output from robot component(s). For example, the current observation data can include a current image captured by a camera of the robot, and optionally a current proximity sensor reading of a proximity sensor of the robot.”);
feeding the first and second input into a first and second machine learning model respectively, wherein the first and second machine learning models are configured to determine tokens for their respective inputs (See 0004, “[…] training and/or using both a high-level policy model and a low-level policy model for mobile robot navigation […] high-level policy model and the low-level policy model can each be a machine learning model, such as a neural network model.” See also 0024, “[…] a target label that is a word embedding of a semantic descriptor of a navigation target, a target label that is an image embedding of an image of a navigation target […]” See also 0027, “[…] image embedder can be a neural network model that is used to process an image and generate a condensed (relative to the pixel size) embedding of the image […]” Examiner notes the word and image embeddings are the same as tokens for the first and second input respectively);
concatenating the determined tokens of the first and second machine learning models (See 0027, “[…] value function can be implemented utilizing a recurrent neural network (RNN) taking as input the concatenated and transformed embeddings of the observation x, target label g, and the proximity bit p […]”);
feeding the concatenated tokens into a third machine learning model, wherein the third machine learning model comprises two policies that are configured to output a skill action and a moving action respectively, wherein the skill action characterizes a categorization of different high-level action categories of the robot and the moving action is an explicit movement proposal for the robot (See 0005, “[…] high-level policy model is used to generate, based on a target label for a navigation target and based on current robot observation(s) (e.g., observation data), high-level output that indicates which of a plurality of discrete high-level actions should be implemented […] low-level action output defines a low-level action that defines robot movement more granularly than does the high-level action. As one non-limiting example, the low-level action can define a corresponding angular velocity and a corresponding linear velocity for each of one or more wheels of a mobile robot. The low-level action output can then be utilized to control one or more actuators of the mobile robot to implement the corresponding low-level action.” See also 0007, “[…] high-level and low-level policies can be cooperatively utilized to achieve efficient mobile robot navigation […]”); and
deciding based on the skill action whether the moving action is outputted as action or a more precise movement proposal for the robot than the moving action as the action is determined according to the high-level action category of the skill action from an external source (See 0079, “[…] the system determines whether the high-level action can be implemented without utilization of a low-level policy model. For example, action(s) such as “turn left” or “turn right” may optionally be implemented without utilization of the low-level policy model, while other action(s) such as “forward” require utilization of the low-level policy model.”).
Regarding Claim 2, Toshev further discloses the method according to claim 1, wherein the external source comprises a set of specialized skills for the different high-level action categories, wherein the specialized skills are methods configured to provide a movement proposal for the respective high-level action category based on a state of the current environment of the robot, wherein the specialized skills are provided with additional sensory input of a current state of the robot and of the state the environment (See 0005 as referenced above. See also 0031, “[…] images represent states of the robot in the world and can be organized in a graph, whose edges represent actions moving the robot from one state to another.” See also 0050-0051, “[…] high-level engine 134 can process observation data 101 and a target label 102 utilizing the high-level policy model 154 to generate a high-level action 103. The observation data 101 can include, for example, a current observation from the vision component 111 (and optionally a current observation from vision component 112 and/or other sensor(s)) […] additional observation data 104 can be, for example, a current observation from the vision component […] process can be continued, relying each time on new current observation data 101 and new current additional observation data 104, until a navigation target is reached. Through continual performance […]” See also 0097, “[…] controlling one or more actuators of the mobile robot based on a corresponding default low-level action defined for the corresponding particular high-level action […]” Examiner notes the plurality of discrete high-level actions is a set of specialized skills of the robot, and different high-level actions require different parameters within the low-level engine to determine which policy model to enable for physical navigation. Examiner further notes the high-level policies include additional observation data, and retrieves data from sensors continually).
Regarding Claim 3, Toshev further discloses the method according to claim 1, wherein the first machine learning model is a pre-trained Large Language Model, and the second machine learning model is a pre-trained vision encoder (See 0004, “[…] the high-level policy model is a recurrent neural network (RNN) model and/or the low-level policy model is a feed forward neural network model, such as a convolutional neural network (CNN) model.” See also 0024, “[…] word embedding of a semantic descriptor […]” See also 0096, “[…] generated by processing the current image using an image embedding model […]” Examiner notes the policy models respectively function as a language model and vision encoder, since the former outputs word embeddings based on the first input and an image embedding through an image embedding model based on sensor output from the camera).
Regarding Claim 4, Toshev further discloses the method according to claim 1, wherein the third machine learning model is a transformer model and the both policies share the transformer model as basis and differ by a regression head for outputting the moving action and a classification head for outputting the skill action (See 0027, “[…] taking as input the concatenated and transformed embeddings […] a single layer long short term memory (LSTM) network model, or other memory network model (e.g., gated recurrent unit (GRU)). The image embedder can be a neural network model that is used to process an image and generate a condensed (relative to the pixel size) embedding of the image […]” See also 0029, “[…] low-level action space
PNG
media_image1.png
38
33
media_image1.png
Greyscale
low can be continuous, and can optionally be defined by the kinematics of the robot.” See also 0042, “[…] continuous Deep Q-Learning (DDPG) is used in training the low-level policy model. For example, the policy can be to execute a “forward” action without colliding with objects.” See also 0096, “[…] value assigned based on a location of the navigation target in the environment, a classification of an object, or an embedding of an image of the object. At each of the plurality of iterations, the method can further include: determining that the corresponding particular high-level action is one that is capable […]” Examiner notes the continuous output for training the low-level model is the same as a regression head for outputting a low-level moving action, while the high-level policy model is based on additional parameters including classification of objects).
Regarding Claim 5, Toshev further discloses the method according to claim 1, wherein the skill action comprises a list of different high-level action categories, wherein the high-level actions categories are terminate, moving according to the moving action and different predefined specialized skills (See 0005 as referenced above and, “[…] high-level actions can include “go forward”, “turn right”, and “turn left”. The low-level policy model is used to generate, based on current robot observation(s) (that can optionally differ from those utilized in generating the high-level output) and optionally based on a high-level action selected based on the high-level output, low-level action output.” See also 0062, “[…] the system determines not to continue with the current supervised episode (e.g., the navigation target has been reached) […]” See also 0080, “[…] the system determines the high-level action can be implemented without utilization of the low-level policy model, the system proceeds to block 412 and selects a low-level action for the high-level action. For example, if the high-level action is “turn right”, a default low level action for “turn right” can be selected.”).
Regarding Claim 6, Toshev further discloses the method according to claim 1, wherein during the concatenation of the tokens, additional read-out tokens are added (See 0096, “[…] generating each of the corresponding high-level action outputs, the corresponding additional observation data can also be processed, along with the corresponding current observation data and the target label, using the trained high-level policy model.”).
Regarding Claim 8, Toshev further discloses the method according to claim 1, wherein depending on the action a control signal for the robot is determined, wherein the robot is controlled to carry out the action by the control signal (See 0085, “[…] translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.”).
Regarding Claim 9, Toshev further discloses the method according to claim 1, wherein the robot is a manufacturing machine or an assembly robot (See 0001, “[…] various mobile robots require robust navigation in dynamic environments.” Examiner notes the robot being either a manufacturing machine or assembly robot is not given patentable weight due to the invention being claimed as a process and the prior art disclosing said process).
Regarding Claim 10, Toshev further discloses a computer program that is configured to cause a computer to carry out the method according to claim 1 with all of its steps if the computer program is carried out by a processor (See 0091-0092, “Storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules […] software modules are generally executed by processor 614 alone or in combination with other processors […]”).
Regarding Claim 11, Toshev further discloses a machine-readable storage medium on which the computer program according to claim 10 is stored (See 0013, “[…] include at least one transitory or non-transitory computer readable storage medium storing instructions executable by one or more processor(s) […]”).
Regarding Claim 12, Toshev further discloses a system that is configured to carry out the method according to claim 1 (See 0013, “[…] include a system of one or more computers and/or one or more robots that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein.”).
Regarding Claim 13, Toshev further discloses the method according to claim 1, wherein the robot is an assembly robot (See 0001 as referenced above).
Regarding Claim 14, Toshev further discloses the method according to claim 1, wherein the sensor output is an image (See 0011, “[…] real-world observations (that can include RGB image observations and/or other higher fidelity observations) […]” See also 0055, “[…] real observation data can include a real RGB image from a deployment environment […]”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 7 is rejected under 35 U.S.C. 103 as being obvious over Toshev et al. (US Patent Pub. No. 2021/0397195 A1), in view of Zhu et al. (US Patent Pub. No. 2023/0280726 A1), herein “Zhu”, filed March 1st, 2022.
Regarding Claim 7, Toshev does not explicitly disclose the method according to claim 1, wherein a new specialized skill is added to the external source, wherein the different high-level action categories of the skill actions is expanded by an additional category for the new specialized skill, wherein the policy of the third machine learning model for the skill action is retrained by finetuning.
Zhu, in a similar field of endeavor, teaches a new specialized skill is added to the external source, wherein the different high-level action categories of the skill actions is expanded by an additional category for the new specialized skill, wherein the policy of the third machine learning model for the skill action is retrained by finetuning (See 0008, “[…] manipulation task may be broken down into a plurality of sequential sub-tasks (policies). These policies may be fine-tuned so that a terminal state distribution of a given policy matches an initial state distribution of another policy that immediately follows the given policy within the plurality of policies. The fine-tuned plurality of policies may then be chained together and implemented within a manipulation environment.” See also 0019-0021, “[…] the updated feature state may then be analyzed to determine additional manipulations necessary to achieve a desired terminal state distribution for the policy […] initiating a first policy of the fine-tuned plurality of policies, and initiating additional policies in response to a completion of previous policy such that the fine-tuned plurality of policies is sequentially implemented.”).
In view of Zhu’s teachings, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include, with the hierarchical robot control system using layers of trained learning models as disclosed by Toshev, policy fine-tuning techniques and additional robot manipulation logic, with a reasonable expectation of success, since fine-tuning the high-level selection policy when introducing additional robot skills or action types is a routine adaptation of a learned policy to recognize and select among an expanded set of available behaviors. Furthermore, the combination would increase the flexibility of hierarchical control and enable efficient learning of new task behaviors by applying known policy optimization techniques to known robot policy frameworks to obtain the predictable result of improved task execution.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Ding et al. (US Patent Pub. No. 2017/0320210 A1), which is directed towards a robot and robot method executing sets of autonomous control tasks through control instructions and environmental sensor data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Bryant Tang whose telephone number is (571)270-0145. The examiner can normally be reached M-F 8-5 CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached at (571)272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRYANT TANG/Examiner, Art Unit 3658
/JASON HOLLOWAY/Primary Examiner, Art Unit 3658
Prosecution Insights