Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to RCE
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/07/2026 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5-8, 10, 15, 17, 18, 19, 20, 21 are rejected under 35 U.S.C. 103 as being unpatentable by Chen (US20220184806) in view of Holson (US20230015796) and Aparicio (US20240335941) and SolowJow (US20230330858).
Regarding claim 1, Chen teaches a processing system for controlling a device using machine learning models, comprising:
at least one memory having executable instructions stored thereon; and one or more processors configured to execute the executable instructions to cause the processing system to ([0020], [0027]-[0028]):
access data characterizing a physical environment in which the device is operating ([0029] disclosing images of environment);
generate a first set of affordable actions based on processing the data via ([0030]-[0033] disclosing a set of potential actions “affordable” based on the current state of the environment, at least [0037] disclosing locations for the desired goal and the desired pickup);
each affordable action of the first set of affordable actions corresponds to a respective set of action parameters ([0027]-[0044] disclosing the set of affordable actions include a set of parameters including at least a location of an object in the environment);
generate, via a second set of machine learning([0030]-[0033] disclosing based on the set of affordable actions to select via a prediction model an action from the set of actions); and
cause the device to execute the first selected action (0032] disclosing an actuator performing the action).
While Chen does not explicitly disclose the machine learning model to generate candidate actions.
Holson teaches the neural network to generate candidate actions and select from the actions ([0040]-[0043], [0087]-[0090] disclosing machine learning to generate an operation to be carried and position to be selected based on a task and selecting the task based on confidence value ).
it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Chen to incorporate the teaching of Holson of machine learning model in order to generate the output that defines the operation to be carried out at the interaction point or environmental conditions as taught by Holson [0041], it would be obvious to combine or substitute the neural network with the model of Chen in order to generate tasks such as a pickup task for a specific object such as a cup or another physical object based on pre-trained data thus improving efficiency and accuracy of actions tailored to objects [0041].
Chen as modified by Holson does not teach wherein to generate the first set of affordable actions, the one or more processors are configured to cause the processing system to generate a set of output affordance maps, wherein each output affordance map of the set of output affordance maps corresponds to a respective affordable action of the first set of affordable actions, An affordance map of the set of output affordance maps comprises a pixel encoding indicating a region of a first object in the physical environment. at least one set of action parameters corresponds to a force to be applied to a second object in the physical environment
Aparicio teaches wherein to generate the first set of affordable actions, the one or more processors are configured to cause the processing system to generate a set of output affordance maps, wherein each output affordance map of the set of output affordance maps corresponds to a respective affordable action of the first set of affordable actions, ([0005], [0020]-[0030] disclosing affordance maps that defines actions to be performed, the respective grasps at pixels with their success metric).
An affordance map of the set of output affordance maps comprises a pixel encoding indicating a region of a first object in the physical environment (at least [0011]-[0030] disclosing the labeling of a region to be grasped on the physical object at a pixel, i.e., pixel encoding of the region of a physical object, in fact Aparicio teaches the affordance is determined for a first object, second object and plurality of objects and the control to grasp at least a first and second object).
It would have been obvious to incorporate the teaching of Aparicio in order to define actions that are feasible based on the affordance map as taught by Aparicio [0005]. The combination of Aparicio teaching of an affordance map is obvious yielding predictable results in order to determine graspable metric and probabilities associated with the affordance grasps improving grasping objects.
Chen as modified by Holson and Aparicio does not teach at least one set of action parameters corresponds to a force to be applied to a second object in the physical environment.
Solowjow teaches at least one set of action parameters corresponds to a force to be applied to a second object in the physical environment ([0030]-[0036] disclosing the at least one set of parameters for affordable actions is a force applied to a second object used for insertion).
It would have been obvious to one of ordinary skill in the art to have combined the teaching of Solowjow of set of action parameters include a force applied to an object to the teaching of affordable actions as taught by Chen as modified by Holson and Aparicio yielding predictable results in order to avoid damaging the second piece recognizing a force limit to be applied thus improving robotic safety.
Regarding claim 5, Chen as modified by Holson and Aparicio and Solowjow teaches the processing system of claim 1, wherein the one or more processors are further configured to cause the processing system to:
determine that the device has executed the first selected action ([0043]-[0044]Chen disclosing the object has been removed);
access second data characterizing the physical environment after execution of the first selected action ([0043-[0044] disclosing the physical state is characterized as the object has been removed);
generate a second set of affordable actions based on processing the second data via the first set of machine learning models, wherein each respective affordable action of the second set of affordable actions indicates an action that can be performed conditioned on the device having executed the first selected action ([0043]-[0045] disclosing the second actions are determined for each state taking consideration of any previous state and action that has been performed);
generate, via the second set of machine learning models, a second selected action to be performed with a second object in the physical environment based on the task and the second set of affordable actions ([0043]-[0045] disclosing determining the action on an object, herein the second object is just interpreted as the object since there is no previous action on any first object and Aparicio in claim 1 already teaches the affordances for plurality of objects); and
cause the device to execute the second selected action (at least [0042]-[0045] disclosing the actuator performing the action).
Holson teaches the neural network to generate candidate actions ([0040]-[0043], [0087]-[0090] disclosing machine learning to generate an operation to be carried and position to be selected based on a task and selecting the task based on confidence value ).
it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Chen as modified by Holson and Aparicio to incorporate the teaching of Holson of machine learning model in order to generate the output that defines the operation to be carried out at the interaction point or environmental conditions as taught by Holson [0041], it would be obvious to combine or substitute the neural network with the model of Chen in order to generate tasks such as a pickup task for a specific object such as a cup or another physical object based on pre-trained data thus improving efficiency and accuracy of actions tailored to objects [0041].
Regarding claim 6, Chen as modified by Holson and Aparicio and Solowjow teaches the processing system of claim 5, wherein the second set of affordable actions includes at least one action that is not included in the first set of affordable actions, and wherein the first set of affordable actions includes at least one action that is not included in the second set of affordable actions (Chen [0043]-[0046] disclosing the second action is selected after the object is removed thus any action for the object is not included).
Regarding claim 7, Chen as modified by Holson and Aparicio and Solowjow teaches the processing system of claim 5, wherein:
the device comprises a robot, to cause the device to execute the first selected action, the one or more processors are configured to cause the processing system to cause the robot to grasp the second object located in the physical environment, and the second selected action comprises an action performed by the robot with the second object (Chen [0043]-[0044] disclosing an action of picking and grasping the object and placing the object as another action).
Regarding claim 8, Chen as modified by Holson and Aparicio and Solowjow teaches the processing system of claim 7, wherein the one or more processors are further configured to cause the processing system to: generate, via the second set of machine learning models, a third selected action to be performed based on the task and a third set of affordable actions, wherein each respective affordable action of the third set of affordable actions indicates an action that can be performed conditioned on the device having executed the second selected action; and cause the device to execute the third selected action (Chen [0043]-[0047] disclosing plurality of actions to be performed after each other and depend on current state and task and previous state which is interpreted to include a third action from which the next action is selected and it is interpreted that a next action is conditioned on the completion of the previous action).
Regarding claim 10, Chen as modified by Holson and Aparicio and Solowjow teaches the processing system of claim 1, wherein each respective affordable action of the first set of affordable actions further indicates a respective probability that the affordable action can be performed at the location in the physical environment, and wherein the first selected action comprises an affordable action having a highest probability of the first set of affordable actions (Chen [0038] at least disclosing the probability that the first action has the most success to achieve the desired task).
Regarding claim 15, Chen as modified by Holson and Aparicio and Solowjow teaches the processing system of claim 1, wherein a different set of action parameters corresponds to at least one of a location of a second object in the physical environment, an orientation in which to interact with the second object in the physical environment.
Specifically, Solowjow teaches wherein a different set of action parameters corresponds to at least one of a location of a second object in the physical environment, an orientation in which to interact with the second object in the physical environment ([at least [0036] disclosing the action parameters are based on the location of the second object).
It would have been obvious to one of ordinary skill in the art to have combined the teaching of Solowjow of parameter associated with the location of a second object in order to improve the insertion of a first object into the second object thus improving accuracy of insertion.
Regarding claim 17, Chen as modified by Holson and Aparicio and Solowjow teaches the processing system of claim 1, wherein to generate the first set of affordable actions, the one or more processors are further configured to cause the processing system to:
generate, via a first generative artificial intelligence model, an execution plan including a plurality of sub-actions to complete a task in the physical environment, each respective sub-action identifying a respective operation to perform on a respective object in the physical environment ([0027]-[0044] disclosing a plurality of actions and sub actions to perform on an object in an environment); and
Aparicio further teaches for each respective sub-action in the execution plan, generate a respective affordable action based on an object map identifying a location of the respective object in the physical environment and an identified actionable point associated with the respective object ([0022]-[0026] disclosing selecting an action of grasping an object in an environment based on its location and a grasping point).
It would have been obvious to one of ordinary skill in the art to have modified the teaching Chen as modified by Holson to incorporate the teaching of Aparicio of for each respective sub-action in the execution plan, generate a respective affordable action based on an object map identifying a location of the respective object in the physical environment and an identified actionable point associated with the respective object in order to securely grasp an object at a position indicated in an image and at a affordable point thus securing the grasping action as taught by Aparicio [0022]-[0026]. It is also obvious to combine the teaching of Aparicio yielding predictable results, the depth map aids in identifying the pose of an object which improves object manipulation.
Regarding claim 18, Chen as modified by Holson and Aparicio and Solowjow teaches processing system of claim 17, wherein to generate the first selected action to be performed in the physical environment, the one or more processors are configured to cause the processing system to generate, via a second generative artificial intelligence model, executable code for performing the first selected action based on a first sub-action in the execution plan (Chen [0027]-[0044] disclosing each action is performed before the other action).
Claim 19 is rejected for similar reasons as claim 1.
Claim 20 is rejected for similar reasons as claim 17, see above rejection.
Claim 21 is rejected for similar reasons as claim 15, see above rejection.
Claims 2 are rejected under 35 U.S.C. 103 as being unpatentable by Chen (US20220184806) in view of Holson (US20230015796) and Aparicio (US20240335941) and SolowJow (US20230330858) and Nagarajan (US11097418).
Regarding claim 2, Chen as modified by Holson as modified by Holson and Apricio and Solowjow teaches the processing system of claim 1, wherein the one or more sub-tasks comprises a sequence of sub-tasks, wherein the one or more processors are further configured to cause the processing system to decompose, via the second set of machine learning models, the task into the sequence of sub-tasks (Chen [0036]-[0037] disclosing predicting promising action sequence. [0038] disclosing a task includes more than one motion. [0041]-[0042] action sequence).
Nagarajan teaches a neural network decomposing a task (col. 3.-col. 4 disclosing machine learning selecting a grasp strategy including sliding a plate to an edge and then grasping, i.e., decomposing the task of grasping, at least col. 16 lines 50-60 discloses the machine learning is a neural network).
it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teaching of Chen as modified by Holson and Apricio and Solowjow to incorporate the teaching of Nagarajan of a neural network decomposing a task in order to determine the best strategy such as decomposing the grasp into a pregrasp and then a grasp based on classification of an object thus improving robotic action and facilitating an action that otherwise would be difficult as taught by Nagarajan col. 4. It is also obvious to try the method of using the neural network to predict the motion instead of an algorithm with reasonable expectation of success and improving the action for objects by tailoring a decomposed action for them based on classification.
Claims 3, 4, 9 are rejected under 35 U.S.C. 103 as being unpatentable by Chen (US20220184806) in view of Holson (US20230015796) and Aparicio (US20240335941) and SolowJow (US20230330858) and Su (US20210122039).
Regarding claim 3, Chen as modified by Holson and Aparicio and Solowjow does not teach the processing system of claim 1, wherein:. to generate, via the second set of machine learning models, the first selected action, the one or more processors are configured to cause the processing system to: select a first affordable action included in the first set of affordable actions; and modify the set of action parameters associated with the first affordable action to generate a set of modified action parameters. the device executes the first selected action based on the set of modified action parameters
Su teaches to generate, via the second set of machine learning models, the first selected action, the one or more processors are configured to cause the processing system to: select a first affordable action included in the first set of affordable actions; and modify the set of action parameters associated with the first affordable action to generate a set of modified action parameters ([0129 disclosing the machine learning. [0157]-[0159] disclosing selecting a first way to hold the object such as from sides or top, orientation “parameter” and deciding to reorienting the object to avoid slip and or using force and or reducing the force which are adjusted parameters of an action, also the adjusted orientation or grasp is an adjusted parameter).
the device executes the first selected action based on the set of modified action parameters ([0157]-[0159] disclosing changing an orientation, see figure 6c showing the change of orientation and the points of grasp).
It would have been obvious to one of ordinary skill in the art to have modified the teaching of Chen as modified by Holson and Aparicio and Solowjow to incorporate the teaching of Wen of to generate, via the second set of machine learning models, the first selected action, the one or more processors are configured to cause the processing system to: select a first affordable action included in the first set of affordable actions; and modify the set of action parameters associated with the first affordable action to generate a set of modified action parameters . The combination is obvious to one of ordinary skill in the art to modify the action in order to reduce slip thus improving an action performed by a robot such as securely holding an object yielding predictable results and helps the robot to carry the object in a current possible way due to limitations in the environment and further adjust the grasp as taught by Su [0157]-[0159].
Regarding claim 4, Chen as modified by Holson and Aparicio and Solowjow and Su teaches the processing system of claim 3,
Specifically Su teaches wherein to cause the device to execute the first selected action, the one or more processors are configured to cause the processing system to convert the set of modified action parameters into one or more control signals for output to the device in the physical environment (at least [0129] disclosing the processor implements machine learning. [0158]-[0159] disclosing the robotic system carrying the maneuver based on the parameter).
The combination is obvious to one of ordinary skill in the art to modify the action in order to reduce slip thus improving an action performed by a robot such as securely holding an object yielding predictable results.
Regarding claim 9, Chen as modified by Holson as modified by Holson and Aparicio and Solowjow teaches the processing system of claim 1,
Su teaches wherein the one or more processors are configured to cause the processing system to, while the device is executing the first selected action:
monitor, via the second set of machine learning models, a state of the device; and in response to a determination, via the second set of machine learning models, that the first selected action should be modified: adjust, via the second set of machine learning models, one or more action parameters corresponding to the first selected action, or cause the device to stop performing the first selected action ([0158]-[0159] disclosing a set of grasps that can be applied that has adjusted parameter such as increasing a force and to reorient the object while monitoring the forces and moments by the computer and performing the action).
The combination is obvious to one of ordinary skill in the art to modify the action in order to reduce slip thus improving an action performed by a robot such as securely holding an object yielding predictable results.
Claims 12 are rejected under 35 U.S.C. 103 as being unpatentable by Chen (US20220184806) in view of Holson (US20230015796) and Aparicio (US20240335941) and SolowJow (US20230330858) and Garg (US20240386733).
Regarding claim 12, Chen as modified by Holson and Aparicio and Solowjow teaches the processing system of claim 1. But does not teach wherein the one or more processors are further configured to cause the processing system to decompose the set of output affordance maps into a plurality of patches.
Garg further teaches decompose the set of output affordance maps into a plurality of patches ([0018] disclosing the affordances “i.e., a map is decomposed into affordances” are given to the machine learning );
generate a plurality of embeddings based on the plurality of the patches; and output the plurality of embeddings to the second set of machine learning models ([0018] disclosing generating knowledge as embeddings)
it would have been obvious to one of ordinary skill in the art to have modified the teaching of Chen as modified by Holson and Aparicio and Solowjow to incorporate the affordances map of Aparicio decomposed into the affordances of Garg in order to facilitate the action planning as taught by Garg [0018].
Claims 13 are rejected under 35 U.S.C. 103 as being unpatentable by Chen (US20220184806) in view of Holson (US20230015796) and Aparicio (US20240335941) and SolowJow (US20230330858) and Fulda (“NPL” what can you do with a rock? Affordance extraction via word embeddings)
Regarding claim 13, Chen as modified by Holson and Aparicio and Solowjow teaches processing system of claim 1,
Fulda teaches wherein to generate the first set of affordable actions, the one or more processors are configured to cause the processing system to generate a set of word tokens, wherein each word token of the set of word tokens corresponds to a textual description of a respective affordable action of the first set of affordable actions (at least abstract disclosing affordable actions to devices based on the object as text enabling an agent to perform tasks).
It would have been obvious to incorporate the teaching of Fulda in order to facilitate robotic acquisition of actionable insights without the perquisite of offline reinforcement learning or large number of data and in addition to make the agent selection resemble human selection as taught by Fulda (abstract).
Claims 14 are rejected under 35 U.S.C. 103 as being unpatentable by Chen (US20220184806) in view of Holson (US20230015796) and Aparicio (US20240335941) and SolowJow (US20230330858) and Garg (US20240386733) and Fulda (“NPL” what can you do with a rock? Affordance extraction via word embeddings).
Regarding claim 14, Chen as modified by Holson and Aparicio and Solowjow and Fulda teaches the processing system of claim 13,
Garg teaches wherein the set of word tokens is based on a set of features extracted from the set of output affordance maps (Garg [0018] disclosing the text is based on affordances generated by machine learning).
it would have been obvious to one of ordinary skill in the art to have modified the teaching of Chen as modified by Holson and Aparicio and Fulda to incorporate the affordances map of Aparicio decomposed into the affordances of Garg in order to facilitate the action planning as taught by Garg [0018].
Claims 16 are rejected under 35 U.S.C. 103 as being unpatentable by Chen (US20220184806) in view of Holson (US20230015796) and Aparicio (US20240335941) and SolowJow (US20230330858) and Claussen (US20200254609).
Regarding claim 16, Chen as modified by Holson and Aparicio and Solowjow teaches the processing system of claim 1,
Claussen teaches wherein the first set of machine learning models comprises a set of convolutional neural networks, and wherein each convolutional neural network included in the set of convolutional neural networks corresponds to a different type of action that can be performed by the device (abstract disclosing a plurality of task specific neural networks to perform a task for the robot, see claim 12 disclosing making a full neural network task specific).
It would have been obvious to have modified the teaching of Chen as modified by Holson to incorporate the teaching of Claussen in order to reduce the load since training a single algorithm can become prohibitive as taught by Claussen [0003].
Response to Arguments
Applicant’s arguments filed on 04/07/2026 have been fully considered but they are not persuasive.
With respect to applicant’s arguments regarding amended claims 1, 19, to include a force applied to the second object. Applicant’s arguments with respect to the independent claims regarding the force parameter have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
The prior art made of record and not relied upon is considered pertinent to
applicant's disclosure. The prior art cited in PTO-892 and not mentioned above disclose related devices and methods.
NPL “Robot-enabled construction assembly with automated sequence planning based on ChatGPT: ROBOGPT” disclosing LLM to simplify robot programming into natural language sequential tasks.
US20240131698 disclosing the force sequence for object stacking.
US20240351218 disclosing the LLM machine learning generates a sequence of tasks to accomplish a task.
US20230080768 disclosing changing the sequence of tasks on the fly based on the current state of the object.
US20230104775 disclosing CNN NEURAL NETWORK decomposing tasks from videos of humans.
US20240131698 disclosing specific neural network for each task and subtask.
US20230278213 disclosing adjusting a regrasp in order to perform an assembly operation.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMAD O EL SAYAH whose telephone number is (571)270-7734. The examiner can normally be reached on M-Th 6:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached on (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMAD O EL SAYAH/Examiner, Art Unit 3658B
Prosecution Insights