of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 6, 8-9, 11, and 20-22 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Gonzalez Aguirre et al (US Pub 2020/0324409 A1), hereafter known as Gonzalez Aguirre.
For Claim 1, Gonzalez Aguirre teaches A method, comprising:
receiving, by data processing hardware of a robot, a request for manipulating a target constrained object; ([0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).
[0036] In some embodiments, the mission planner 108 may receive the one or more maps of the map and semantics generator 104 and/or an identification of ranked actions from the robot modeler 106. The mission planner 108 may determine a resulting symbolic plan with attributes and active subsets of actions based on the one or more maps of the map and semantics generator 104. The active subset of actions may be a resulting intersection of actions (e.g., verbs linked by noun names in the segmented parts) from the one or more maps from the map and semantics generator 104, actions from the ranked actions from the robot modeler 106 to implement the plan and actions (e.g., granular atomic-actions) identified from the command data 114.
Figure 1)
receiving, from at least one sensor of the robot, perception data indicative of the target constrained object; ([0025] The map and semantics generator 104 may further generate a surface map that identifies surfaces based on the sensor data and the occupancy map (e.g., classify the occupied spaces into various surfaces). For example, the surface map may be a structured point-cloud that includes a collection of 3D vertex points linked by edges on the surfaces.
[0026] The map and semantics generator 104 may further generate a semantic labelled map (e.g., connect labels to surfaces in the surface map) based on the surface map and the sensor data. For example, the map and semantics generator 104 may include a deep neural network that identifies each object in the surface map, identifies boundaries of the object, applies a label (e.g., cup, cube, bottle, table, etc.) to the object (e.g., surface segments) and assigns a unique value (e.g., an instance identifier) to the object for future reference.
[0027] The map and semantics generator 104 may further generate a part labelled semantic map (e.g., generation of semantic endowed surface regions which may be referred to as semantic patches or surface patches) based on the semantic labelled map and the sensor data. For example, the part labelled semantic map may identify the parts of each objects. As a more detailed example, if a motorcycle is identified, the parts may include a handle, frame, seat, tank and wheel. Each of the parts may be labelled in the part labelled semantic map.
[0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).
Figure 1)
receiving, by the data processing hardware, a semantic model of the target constrained object generated based on the perception data; ([0024] The map and semantics generator 104 may generate one or more maps based on the image and/or range data. For example, the map and semantics generator 104 may generate an occupancy map to represent an environment of the robot such as an occupancy map (continuous or discrete) that maps occupied spaces. In some embodiments, the map and semantics generator 104 may further map unoccupied spaces and/or unknown spaces (spaces that cannot be identified as occupied or unoccupied) and store the unoccupied spaces and/or unknown spaces in the occupied map or another map.
[0025] The map and semantics generator 104 may further generate a surface map that identifies surfaces based on the sensor data and the occupancy map (e.g., classify the occupied spaces into various surfaces). For example, the surface map may be a structured point-cloud that includes a collection of 3D vertex points linked by edges on the surfaces.
[0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).
[0031] In some embodiments, the location data 112 may provide sensor information which is used by the robot modeler 106 in conjunction with the direct and inverse kinematics to precompute a 3d body-relative reachability map. By using that reachability map and the current state of the scene, the robot modeler 106 may identify a possible set of actions (e.g., actions to physically manipulate the object). In some embodiments the map and semantics generator 104 may provide the scene to the robot modeler 106.
[0032] In some embodiments, robot modeler 106 may receive the one or more maps of map and semantics generator 104 and determine suitable grips for various objects based on the graspability map. For example, the robot modeler 106 may identify that certain grips would be ineffective (e.g., would be unable to manipulate an object, would not be able to hold a slipper object, etc.) for the objects and thus exclude such grips from being propagated to the mission planner 108 as ranked action.
[0078] The robot modeler 354 may therefore efficiently and quickly reject unsuitable grasps while simultaneously providing a mathematical analysis to drive gradient estimations for best grips during, for example, neural network training. The robot modeler 354 may further select grasps from the graspability map 368 and generate grasp actions and rankings based on direct and inverse kinematics and grasping indexes 366. Thus, the robot modeler 354 may generate actionable grasp models, and rank the models based on metrics 364.)
determining, by the data processing hardware, a location for a robotic arm of the robot to interact with the target constrained object based on the semantic model and the request; and ([0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).
[0032] In some embodiments, robot modeler 106 may receive the one or more maps of map and semantics generator 104 and determine suitable grips for various objects based on the graspability map. For example, the robot modeler 106 may identify that certain grips would be ineffective (e.g., would be unable to manipulate an object, would not be able to hold a slipper object, etc.) for the objects and thus exclude such grips from being propagated to the mission planner 108 as ranked action.
[0051] Therefore, the scene semantic spatial context generator 304 may generate a 6D kinematic frame based on the fused assertion. The 6D kinematic frame may be 6D because of 3 degrees of freedom for position and 3 degrees of freedom for orientation. The set of 6 degrees of freedom may unambiguously define a pose in space. In some embodiments, the motion may further be defined with respect to speed in each dimension namely V.sub.x, V.sub.y, V.sub.z as well as V.sub.roll, V.sub.pitch and V.sub.Yaw, which will may correspond to another set of 6 degrees of freedom. The degrees of freedom of the robotic end effector 132 may not be limited by 6 degrees of freedom, but may depend on a robot structure associated with the robotic end effector 132, motors and joint types. In some embodiments, a robot may need to have at least 6 degrees of freedom to grasp objects in a general position)
controlling, by the data processing hardware, the robotic arm to manipulate the target constrained object based on the location for the robotic arm to interact with the target constrained object. ([0037] The mission planner 108 may provide the decomposed commands and plans 126 to the end effector controller 128 (e.g., a processor on the end effector that controls actions). Additionally, the robot modeler 106 may provide the ranked action to the end effector controller 128, and the map and semantics generator 104 may provide the map and semantic data to the end effector controller 128. The end effector controller 128 controls the robotic end effector 130, 132 to implement the decomposed commands and plans that include actions that are identified by the mission planner 108 (e.g., intersections of actions). The sensor array 102 may further provide sensor data 134 to the end effector controller 128 so the end effector controller 128 may control the end effector 130 based on updated sensor data (e.g., positional data).)
For Claim 2, Gonzalez Aguirre teaches The method of claim 1, wherein the target constrained object is constrained in at least one degree of freedom (DoF) of movement. ([0027] The map and semantics generator 104 may further generate a part labelled semantic map (e.g., generation of semantic endowed surface regions which may be referred to as semantic patches or surface patches) based on the semantic labelled map and the sensor data. For example, the part labelled semantic map may identify the parts of each objects. As a more detailed example, if a motorcycle is identified, the parts may include a handle, frame, seat, tank and wheel. Each of the parts may be labelled in the part labelled semantic map.
Depending on the type of part (such as a handle) the part would be constrained in at least one degree of freedom).
For Claim 3, Gonzalez Aguirre teaches The method of claim 1, wherein the request comprises an indication of the target constrained object and an instruction for manipulating the target constrained object. ([0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).
[0036] In some embodiments, the mission planner 108 may receive the one or more maps of the map and semantics generator 104 and/or an identification of ranked actions from the robot modeler 106. The mission planner 108 may determine a resulting symbolic plan with attributes and active subsets of actions based on the one or more maps of the map and semantics generator 104. The active subset of actions may be a resulting intersection of actions (e.g., verbs linked by noun names in the segmented parts) from the one or more maps from the map and semantics generator 104, actions from the ranked actions from the robot modeler 106 to implement the plan and actions (e.g., granular atomic-actions) identified from the command data 114.
Figure 1)
For Claim 6, Gonzalez Aguirre teaches The method of claim 1, wherein receiving the semantic model comprises determining, by the data processing hardware, the semantic model by:
identifying a graspable portion of the target constrained object within the perception data and identifying a location where the graspable portion is attached to a remainder of the target constrained object; ([0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).
[0032] In some embodiments, robot modeler 106 may receive the one or more maps of map and semantics generator 104 and determine suitable grips for various objects based on the graspability map. For example, the robot modeler 106 may identify that certain grips would be ineffective (e.g., would be unable to manipulate an object, would not be able to hold a slipper object, etc.) for the objects and thus exclude such grips from being propagated to the mission planner 108 as ranked action.
[0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).)
identifying a plurality of axes of the target constrained object;
identifying an axis of rotation of the target constrained object; and/or
identifying an axis of the target constrained object that can be grasped.
For Claim 8, Gonzalez Aguirre teaches The method of claim 1, further comprising:
determining a pose of the robotic arm for grasping the target constrained object based on the semantic model. ([0054] The free and occupied map generator 316 may generate sparse dual-space map that may capture and split the occupied and unfilled (free) spaces. This mapping may allow for: i) registering diverse 3D images while exploring various interaction (e.g., grasping) scenarios for a kinematic end effector, ii) determine possible collision-free manipulator 6D poses in the environment and iii) serve as an effective scaffolding data structure to store multiresolution local surface descriptors such as volumetric (e.g., with respect to voxels) semantic labels and other attributes.
[0089] Illustrated processing block 442 identifies operational limits of a mission. Illustrated processing block 444 identifies task information. Illustrated processing block 446 determines operational bounds. Illustrated processing block 448 determines a goal of the mission. Illustrated processing block 450 generates a plan based on information provided in the above identified processing blocks.
[0035] For example, the mission planner 108 may set a maximal speed of the robotic end effector 132 along a manipulation trajectory (e.g., for social space sharing) or maintain containers with orientation limits to avoid failure of the mission (e.g., spilling liquids in a container). The case-by-case operational limits may change in each step of the plan. Thus, identifying the operation limits may filter both affordances by attribute and prioritize affordances by range matching.
[0081] Furthermore, actions may be considered macro-plans that provide operational limits corresponding to the specific domain and task and action goals 394. The mission planner 390 may set a maximal speed of an end-effector along a manipulation trajectory (for social space sharing) or keeping containers with orientation limits (e.g., smoothness or responsiveness) to avoid spilling liquids. The case-by-case operational limits may change in each step of the plan, and may be stored in the operation limits 388. Thus, providing this information in the grasp planning may enable filtering both affordances by attribute and prioritizing affordances by range matching.)
For Claim 9, Gonzalez Aguirre teaches The method of claim 8, further comprising:
resolving one or more ambiguities in the pose of the robotic arm for grasping the target constrained object based on the semantic model, one or more limits associated with joints of the robotic arm, and/or capabilities of actuators of the robotic arm, ([0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).)
wherein one or more ambiguities comprise whether a gripper of the robotic arm can interact with the target constrained object in a plurality of different poses and/or a plurality of poses of the robotic arm. ([0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).)
For Claim 11, Gonzalez Aguirre teaches The method of claim 1, further comprising:
determining a set of parameters for manipulating the target constrained object based on the location for the robotic arm to interact with the target constrained object, (Figure 5,
[0081] Furthermore, actions may be considered macro-plans that provide operational limits corresponding to the specific domain and task and action goals 394. The mission planner 390 may set a maximal speed of an end-effector along a manipulation trajectory (for social space sharing) or keeping containers with orientation limits (e.g., smoothness or responsiveness) to avoid spilling liquids. The case-by-case operational limits may change in each step of the plan, and may be stored in the operation limits 388. Thus, providing this information in the grasp planning may enable filtering both affordances by attribute and prioritizing affordances by range matching.)
wherein controlling the robotic arm to manipulate the target constrained object is further based on the set of parameters, and ([0081] Furthermore, actions may be considered macro-plans that provide operational limits corresponding to the specific domain and task and action goals 394. The mission planner 390 may set a maximal speed of an end-effector along a manipulation trajectory (for social space sharing) or keeping containers with orientation limits (e.g., smoothness or responsiveness) to avoid spilling liquids. The case-by-case operational limits may change in each step of the plan, and may be stored in the operation limits 388. Thus, providing this information in the grasp planning may enable filtering both affordances by attribute and prioritizing affordances by range matching.)
wherein the set of parameters comprises an initial direction to apply wrench to manipulate the target constrained object and/or a task type associated with the target constrained object. ([0034] The mission planner 108 may identify a task from a high level directive (e.g., clean the kitchen). For example, the high level directive may be decomposed into a sequence of granular atomic-actions which may be referred to as macro-plans. The macro-plans may not only provide actions (e.g., physical actions to physically manipulate the object) that may be undertaken to complete the task, but further provide operational limits corresponding to the specific domain and task.
[0080] The mission planner 390 may capture and unfold high-level directives from sensor data provided by the sensor array 386 (e.g., “clean the kitchen”). The mission planner 390 may decompose the directive into a fine granular sequence of physical atomic-actions or tasks (e.g., primary task, secondary task, target object part assertion, affordance list, etc.) to accomplish the high level directive. The tasks may be stored in the task information 392.
[0081] Furthermore, actions may be considered macro-plans that provide operational limits corresponding to the specific domain and task and action goals 394. The mission planner 390 may set a maximal speed of an end-effector along a manipulation trajectory (for social space sharing) or keeping containers with orientation limits (e.g., smoothness or responsiveness) to avoid spilling liquids. The case-by-case operational limits may change in each step of the plan, and may be stored in the operation limits 388. Thus, providing this information in the grasp planning may enable filtering both affordances by attribute and prioritizing affordances by range matching.
[0035] For example, the mission planner 108 may set a maximal speed of the robotic end effector 132 along a manipulation trajectory (e.g., for social space sharing) or maintain containers with orientation limits to avoid failure of the mission (e.g., spilling liquids in a container). The case-by-case operational limits may change in each step of the plan. Thus, identifying the operation limits may filter both affordances by attribute and prioritize affordances by range matching.)
For Claim 20, Gonzalez Aguirre teaches A non-transitory computer-readable medium having stored therein instructions that, when executed by data processing hardware of a robot, cause the data processing hardware to:
receive a request for manipulating a target constrained object; ; ([0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).
[0036] In some embodiments, the mission planner 108 may receive the one or more maps of the map and semantics generator 104 and/or an identification of ranked actions from the robot modeler 106. The mission planner 108 may determine a resulting symbolic plan with attributes and active subsets of actions based on the one or more maps of the map and semantics generator 104. The active subset of actions may be a resulting intersection of actions (e.g., verbs linked by noun names in the segmented parts) from the one or more maps from the map and semantics generator 104, actions from the ranked actions from the robot modeler 106 to implement the plan and actions (e.g., granular atomic-actions) identified from the command data 114.
Figure 1)
receive, from at least one sensor of the robot, perception data indicative of the target constrained object; ([0025] The map and semantics generator 104 may further generate a surface map that identifies surfaces based on the sensor data and the occupancy map (e.g., classify the occupied spaces into various surfaces). For example, the surface map may be a structured point-cloud that includes a collection of 3D vertex points linked by edges on the surfaces.
[0026] The map and semantics generator 104 may further generate a semantic labelled map (e.g., connect labels to surfaces in the surface map) based on the surface map and the sensor data. For example, the map and semantics generator 104 may include a deep neural network that identifies each object in the surface map, identifies boundaries of the object, applies a label (e.g., cup, cube, bottle, table, etc.) to the object (e.g., surface segments) and assigns a unique value (e.g., an instance identifier) to the object for future reference.
[0027] The map and semantics generator 104 may further generate a part labelled semantic map (e.g., generation of semantic endowed surface regions which may be referred to as semantic patches or surface patches) based on the semantic labelled map and the sensor data. For example, the part labelled semantic map may identify the parts of each objects. As a more detailed example, if a motorcycle is identified, the parts may include a handle, frame, seat, tank and wheel. Each of the parts may be labelled in the part labelled semantic map.
[0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).
Figure 1)
receive a semantic model of the target constrained object generated based on the perception data; ([0024] The map and semantics generator 104 may generate one or more maps based on the image and/or range data. For example, the map and semantics generator 104 may generate an occupancy map to represent an environment of the robot such as an occupancy map (continuous or discrete) that maps occupied spaces. In some embodiments, the map and semantics generator 104 may further map unoccupied spaces and/or unknown spaces (spaces that cannot be identified as occupied or unoccupied) and store the unoccupied spaces and/or unknown spaces in the occupied map or another map.
[0025] The map and semantics generator 104 may further generate a surface map that identifies surfaces based on the sensor data and the occupancy map (e.g., classify the occupied spaces into various surfaces). For example, the surface map may be a structured point-cloud that includes a collection of 3D vertex points linked by edges on the surfaces.
[0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).
[0031] In some embodiments, the location data 112 may provide sensor information which is used by the robot modeler 106 in conjunction with the direct and inverse kinematics to precompute a 3d body-relative reachability map. By using that reachability map and the current state of the scene, the robot modeler 106 may identify a possible set of actions (e.g., actions to physically manipulate the object). In some embodiments the map and semantics generator 104 may provide the scene to the robot modeler 106.
[0032] In some embodiments, robot modeler 106 may receive the one or more maps of map and semantics generator 104 and determine suitable grips for various objects based on the graspability map. For example, the robot modeler 106 may identify that certain grips would be ineffective (e.g., would be unable to manipulate an object, would not be able to hold a slipper object, etc.) for the objects and thus exclude such grips from being propagated to the mission planner 108 as ranked action.
[0078] The robot modeler 354 may therefore efficiently and quickly reject unsuitable grasps while simultaneously providing a mathematical analysis to drive gradient estimations for best grips during, for example, neural network training. The robot modeler 354 may further select grasps from the graspability map 368 and generate grasp actions and rankings based on direct and inverse kinematics and grasping indexes 366. Thus, the robot modeler 354 may generate actionable grasp models, and rank the models based on metrics 364.)
determine a location for a robotic arm of the robot to interact with the target constrained object based on the semantic model and the request; and ([0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).
[0032] In some embodiments, robot modeler 106 may receive the one or more maps of map and semantics generator 104 and determine suitable grips for various objects based on the graspability map. For example, the robot modeler 106 may identify that certain grips would be ineffective (e.g., would be unable to manipulate an object, would not be able to hold a slipper object, etc.) for the objects and thus exclude such grips from being propagated to the mission planner 108 as ranked action.
control the robotic arm to manipulate the target constrained object based on the location for the robotic arm to interact with the target constrained object. ([0037] The mission planner 108 may provide the decomposed commands and plans 126 to the end effector controller 128 (e.g., a processor on the end effector that controls actions). Additionally, the robot modeler 106 may provide the ranked action to the end effector controller 128, and the map and semantics generator 104 may provide the map and semantic data to the end effector controller 128. The end effector controller 128 controls the robotic end effector 130, 132 to implement the decomposed commands and plans that include actions that are identified by the mission planner 108 (e.g., intersections of actions). The sensor array 102 may further provide sensor data 134 to the end effector controller 128 so the end effector controller 128 may control the end effector 130 based on updated sensor data (e.g., positional data).)
For Claim 21, Gonzalez Aguirre teaches The non-transitory computer-readable medium of Claim 20, wherein the target constrained object is constrained in at least one degree of freedom (DoF) of movement. ([0027] The map and semantics generator 104 may further generate a part labelled semantic map (e.g., generation of semantic endowed surface regions which may be referred to as semantic patches or surface patches) based on the semantic labelled map and the sensor data. For example, the part labelled semantic map may identify the parts of each objects. As a more detailed example, if a motorcycle is identified, the parts may include a handle, frame, seat, tank and wheel. Each of the parts may be labelled in the part labelled semantic map.
Depending on the type of part (such as a handle) the part would be constrained in at least one degree of freedom).
For Claim 22, Gonzalez Aguirre teaches The non-transitory computer-readable medium of Claim 20, wherein the request comprises an indication of the target constrained object and an instruction for manipulating the target constrained object. ([0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).
[0036] In some embodiments, the mission planner 108 may receive the one or more maps of the map and semantics generator 104 and/or an identification of ranked actions from the robot modeler 106. The mission planner 108 may determine a resulting symbolic plan with attributes and active subsets of actions based on the one or more maps of the map and semantics generator 104. The active subset of actions may be a resulting intersection of actions (e.g., verbs linked by noun names in the segmented parts) from the one or more maps from the map and semantics generator 104, actions from the ranked actions from the robot modeler 106 to implement the plan and actions (e.g., granular atomic-actions) identified from the command data 114.
Figure 1)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez Aguirre in light of Wandzel et al (US Pub 2021/0347046 A1), hereafter known as Wandzel.
For Claim 4, Gonzalez Aguirre teaches The method of claim 1, wherein the request includes natural language, the method further comprising:
Gonzalez Aguirre does not teach parsing the natural language using a large language model to generate an indication of the target constrained object and an instruction for manipulating the target constrained object.
Wandzel, however, does teach parsing the natural language using a large language model to generate an indication of the target constrained object and an instruction for manipulating the target constrained object. ([0012] In one aspect, the present invention features a method of operating a mobile robot to conduct a multi-object search task within an environment. The mobile robot can include at least one processor that can execute computer readable instructions stored in at least one non-transitory computer readable storage medium to perform various operations including: (1) representing a multi-object search task in an Object-Oriented Partially Observable Markov Decision Process model having at least one belief pertaining to a state and at least one observation space within an environment of the robot, wherein the state is represented in terms of classes and objects and each object has at least one attribute and a semantic label; (2) receiving, at the mobile robot from a user, a language command identifying at least one target object and at least one location corresponding to the target object; (3) updating the at least one belief, associated with the at least one target object, based on the language command; (4) driving the mobile robot to the at least one observation space identified in the updated belief, (5) searching, using at least one sensor on the mobile robot while traversing the at least one observation space identified in the updated belief, for the at least one target object; and (6) notifying the user upon finding the at least one target object.)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date that the command that targets the object and commands to manipulate it come from natural language and an LLM because it would allow a user to give simple commands with little training without having to use a particular type of screen or interface device. It would be expected to be successful at giving an intent to a robotic system.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez Aguirre in light of Nogami et al (US Pub 2012/0166165 A1), hereafter known as Nogami.
For Claim 5, Gonzalez Aguirre teaches The method of claim 1, further comprising:
Gonzalez Aguirre does not teach
displaying a camera view received from a camera of the robot on a screen of a remote device;
receiving the request as an input of the remote device; and
displaying, on the screen, a simulated movement of the target constrained object.
Nogami, however, does teach displaying a camera view received from a camera of the robot on a screen of a remote device; ([0009] For example, when a simulation is performed in the robot simulation apparatus 50, as shown in FIG. 8, an action display screen 53a is displayed on the display unit 53 of the robot simulation apparatus 50. On this action display screen 53a, a virtual robot R to be simulated, a camera Ca that images a distal end of the virtual robot R in a virtual space, and a robot sensor Se that detects the position of the virtual robot R in the virtual space are displayed. The virtual robot R to be simulated is a vertical multi-joint robot or a scalar robot. Such a virtual robot R includes a robot body section Ra, which is a proximal end section, and a robot hand Rb, which is a distal end section.)
receiving the request as an input of the remote device; and ([0066] The order of processing in the action display program is explained below. As shown in FIG. 4, in the action display program, first, the processor 11 determines, referring to a program counter and the like, whether a command that should be executed is present (step S21). When no command that should be executed is present (NO in step S21), the processor 11 ends the action display program and ends the robot simulation according to the end of the action display program. On the other hand, when a command that should be executed is present (YES in step S21), the processor 11 reads out a start point of an action of the robot body section Ra and an end point of the action of the robot body section Ra from the robot data 12a and calculates an optimum track until the robot body section Ra located at the start point reaches the end point (step S22). In calculating the optimum track, the processor 11 calculates the optimum track from the start point to the end point on the basis of various conditions set in advance such as a condition that the robot body section Ra moves on a shortest track and a condition that a track of the robot body section Ra have a curvature equal to or larger than a predetermined curvature. The processor 11 stays on standby until an interrupt is generated by the interrupt generating unit 11b (NO in step S23).)
displaying, on the screen, a simulated movement of the target constrained object. ([0070] Subsequently, when the value counted by the process counter 11c reaches the set number of times K (YES in step S27), the processor 11 updates, on the display unit 13, the image of the virtual robot R on the basis of target tracks calculated in track calculation processing performed K times (step S28). The processor 11 determines whether the robot body section Ra reaches the end point. When the robot body section Ra does not reach the end point, the processor 11 stays on standby until the next interrupt is generated (NO in step S29, step S23). On the other hand, when the robot body section Ra reaches the end point, the processor 11 determines again whether a command that should be executed is present (YES in step S29, step S21).
[0009] For example, when a simulation is performed in the robot simulation apparatus 50, as shown in FIG. 8, an action display screen 53a is displayed on the display unit 53 of the robot simulation apparatus 50. On this action display screen 53a, a virtual robot R to be simulated, a camera Ca that images a distal end of the virtual robot R in a virtual space, and a robot sensor Se that detects the position of the virtual robot R in the virtual space are displayed. The virtual robot R to be simulated is a vertical multi-joint robot or a scalar robot. Such a virtual robot R includes a robot body section Ra, which is a proximal end section, and a robot hand Rb, which is a distal end section.
Figure 4)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Gonzalez Aguirre in light of Nogami such that a simulated movement is shown on a display because it would allow a user to see the expected outcome of their command before it is carried out. This would allow the user to prevent miscommunications, as well as help prevent potentially dangerous actions that were not intended.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez Aguirre in light of Buehler et al (US Pub 2013/0346348 A1) hereafter known as Buehler.
For Claim 7, Gonzalez Aguirre teaches The method of claim 1, wherein receiving the semantic model comprises determining, by the data processing hardware, the semantic model by:
applying segmentation to the perception data to identify different portions of the target constrained object; and ([0028] In some embodiments, the map and semantics generator 104 may omit portions of the object from further analysis if the portions are smaller than a predetermined size (e.g., smaller than a contact area of the end effector). In some embodiments, may decrease the resolution of surface patches for more efficient storage and access of corresponding identification data (e.g., corresponding image data may be stored as an octree) and further associate actions (e.g., a physical manipulation of the object) that may be taken with each identified object (e.g., move cup, refill cup, clean cup, etc.) and based on the labelled parts. Thus, the map and semantics generator 104 may link verbs (e.g., actions) and noun names in the segmented surfaces and/or parts)
applying a computer vision algorithm to identify where a handle is attached to a remainder of the target constrained object, and identify one or more other geometrical properties of the target constrained object. ([0027] The map and semantics generator 104 may further generate a part labelled semantic map (e.g., generation of semantic endowed surface regions which may be referred to as semantic patches or surface patches) based on the semantic labelled map and the sensor data. For example, the part labelled semantic map may identify the parts of each objects. As a more detailed example, if a motorcycle is identified, the parts may include a handle, frame, seat, tank and wheel. Each of the parts may be labelled in the part labelled semantic map.
[0101] The illustrated system 150 also includes an input output (10) module 158 implemented together with the host processor 152 and a graphics processor 160 (e.g., GPU) on a semiconductor die 162 as a system on chip (SoC). The illustrated IO module 158 communicates with, for example, a display 164 (e.g., touch screen, liquid crystal display/LCD, light emitting diode/LED display), a network controller 166 (e.g., wired and/or wireless), and mass storage 168 (e.g., hard disk drive/HDD, optical disk, solid state drive/SSD, flash memory). In some embodiments, the system 150 may further include processors and/or AI accelerators 148 dedicated to artificial intelligence (AI) and/or neural network (NN) processing. For example, the system SoC 162 may include vision processing units (VPUs) and/or other AI/NN-specific processors such as AI accelerator 148, etc. In some embodiments, any aspect of the embodiments described herein may be implemented in one or more of the processors and/or accelerators such as AI accelerator 148 dedicated to AI and/or NN processing, the graphics processor 160 and/or the host processor 152.)
Gonzalez Aguirre does not teach determine a set of principal axes of the target constrained object.
Buehler, however, does teach determine a set of principal axes of the target constrained object. ([0062] Upon selection of a visual-model type or class, the learning algorithm 504 extracts a representation 506 from the annotated image(s) of the object (i.e., from contextual data), i.e., it creates a data structure capturing certain visual characteristics of the object. Live sensor data 500 may be used to tune or verify the training process. This representation 506 becomes part of the robot's contextual knowledge 502, and serves, at a later time, as input for the detection algorithm 508. The detection algorithm 508 uses the representation 506 in conjunction with real-time sensor input 500 to identify and localize objects that visually match the representation 506. Its output 510 may, for example, take the form of a Boolean parameter indicating whether the object can be seen in the real-time image, along with real values for a set of parameters indicating the position, size, and/or orientation of the object in a coordinate space associate with the image (e.g., center coordinates x and y, a scaling factors indicative of size, and an angle .theta. indicating the orientation of a selected object axis with respect to a coordinate axis).)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Gonzalez in light of Buehler such that the principal axes are determined because it would assist in determining upwards directions as well as directions the object may be manipulated. Knowing upwards directions would be useful for containers with liquids, which Gonzalez Aguirre does teach, for example.
Claims 10, 12-14, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez Aguirre in light of Da Silva et al (US Pub 9,440,353 B1), hereafter known as Da Silva.
For Claim 10, Gonzalez Aguirre teaches The method of claim 1, further comprising:
determining a pose for the robot based on the location for the robotic arm to interact with the target constrained object, ([0081] Furthermore, actions may be considered macro-plans that provide operational limits corresponding to the specific domain and task and action goals 394. The mission planner 390 may set a maximal speed of an end-effector along a manipulation trajectory (for social space sharing) or keeping containers with orientation limits (e.g., smoothness or responsiveness) to avoid spilling liquids. The case-by-case operational limits may change in each step of the plan, and may be stored in the operation limits 388. Thus, providing this information in the grasp planning may enable filtering both affordances by attribute and prioritizing affordances by range matching.)
wherein controlling the robotic arm to manipulate the target constrained object is further based on the pose for the robot, and ([0081] Furthermore, actions may be considered macro-plans that provide operational limits corresponding to the specific domain and task and action goals 394. The mission planner 390 may set a maximal speed of an end-effector along a manipulation trajectory (for social space sharing) or keeping containers with orientation limits (e.g., smoothness or responsiveness) to avoid spilling liquids. The case-by-case operational limits may change in each step of the plan, and may be stored in the operation limits 388. Thus, providing this information in the grasp planning may enable filtering both affordances by attribute and prioritizing affordances by range matching.
[0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).)
Gonzalez Aguirre does not teach wherein the pose for the robot comprises a pose for a body of the robot and a pose for one or more legs of the robot.
Da Silva, however, does teach wherein the pose for the robot comprises a pose for a body of the robot and a pose for one or more legs of the robot. (Page 24, Column 17, Lines 7-22. (82) In FIG. 6A, the quadruped robot 602 is in a first state 600. The first state 600 includes a set of joint angles and/or actuator forces for each of the quadruped robot's legs (which may also be referred to herein as the starting “body pose”) and the manipulator arm (which may also be referred to herein as the starting “arm pose”). A control system of quadruped robot 602 may control actuators of quadruped robot 602 to put the quadruped robot 602 in the first state 600 in response to receiving feed-forward control inputs specifying the starting body pose and the starting arm pose. The received feed-forward control inputs may also specify an ending body pose and ending arm pose for the quadruped robot 602, along with a timing parameter that specifies the duration of the motion in transitioning from the starting pose to the ending pose.)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Gonzalez Aguirre in light of Da Silva so that the pose information includes a robot’s legs because many robots have legs and have them to move around. The position and pose of the legs could determine the distance of the arms and other grasping tools from the target object, as well as approach angle. Having their pose, position, and trajectory planned out would ensure that the grasping motion is successful.
For Claim 12, Gonzalez Aguirre teaches A robot comprising:
a robotic arm configured to manipulate a target constrained object; ([0051] Therefore, the scene semantic spatial context generator 304 may generate a 6D kinematic frame based on the fused assertion. The 6D kinematic frame may be 6D because of 3 degrees of freedom for position and 3 degrees of freedom for orientation. The set of 6 degrees of freedom may unambiguously define a pose in space. In some embodiments, the motion may further be defined with respect to speed in each dimension namely V.sub.x, V.sub.y, V.sub.z as well as V.sub.roll, V.sub.pitch and V.sub.Yaw, which will may correspond to another set of 6 degrees of freedom. The degrees of freedom of the robotic end effector 132 may not be limited by 6 degrees of freedom, but may depend on a robot structure associated with the robotic end effector 132, motors and joint types. In some embodiments, a robot may need to have at least 6 degrees of freedom to grasp objects in a general position)
at least one sensor configured to generate perception data; and (([0025] The map and semantics generator 104 may further generate a surface map that identifies surfaces based on the sensor data and the occupancy map (e.g., classify the occupied spaces into various surfaces). For example, the surface map may be a structured point-cloud that includes a collection of 3D vertex points linked by edges on the surfaces.
[0026] The map and semantics generator 104 may further generate a semantic labelled map (e.g., connect labels to surfaces in the surface map) based on the surface map and the sensor data. For example, the map and semantics generator 104 may include a deep neural network that identifies each object in the surface map, identifies boundaries of the object, applies a label (e.g., cup, cube, bottle, table, etc.) to the object (e.g., surface segments) and assigns a unique value (e.g., an instance identifier) to the object for future reference.
[0027] The map and semantics generator 104 may further generate a part labelled semantic map (e.g., generation of semantic endowed surface regions which may be referred to as semantic patches or surface patches) based on the semantic labelled map and the sensor data. For example, the part labelled semantic map may identify the parts of each objects. As a more detailed example, if a motorcycle is identified, the parts may include a handle, frame, seat, tank and wheel. Each of the parts may be labelled in the part labelled semantic map.
[0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).
Figure 1))
a control system in communication with the body and the robotic arm, the control system comprising data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to: ([0040] FIG. 2 shows a method 800 of controlling an end effector. The method 800 may generally be implemented in a robotic process such as, for example, the process 100 (FIG. 1), already discussed. In an embodiment, the method 800 is implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium such as random access memory (RAM), read only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc., in configurable logic such as, for example, programmable logic arrays (PLAs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), in fixed-functionality logic hardware using circuit technology such as, for example, application specific integrated circuit (ASIC), complementary metal oxide semiconductor (CMOS) or transistor-transistor logic (TTL) technology, or any combination thereof.)
receive a request for manipulating the target constrained object; (([0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).
[0036] In some embodiments, the mission planner 108 may receive the one or more maps of the map and semantics generator 104 and/or an identification of ranked actions from the robot modeler 106. The mission planner 108 may determine a resulting symbolic plan with attributes and active subsets of actions based on the one or more maps of the map and semantics generator 104. The active subset of actions may be a resulting intersection of actions (e.g., verbs linked by noun names in the segmented parts) from the one or more maps from the map and semantics generator 104, actions from the ranked actions from the robot modeler 106 to implement the plan and actions (e.g., granular atomic-actions) identified from the command data 114.
Figure 1))
receive the perception data from the at least one sensor, the perception data indicative of the target constrained object; (([0025] The map and semantics generator 104 may further generate a surface map that identifies surfaces based on the sensor data and the occupancy map (e.g., classify the occupied spaces into various surfaces). For example, the surface map may be a structured point-cloud that includes a collection of 3D vertex points linked by edges on the surfaces.
[0026] The map and semantics generator 104 may further generate a semantic labelled map (e.g., connect labels to surfaces in the surface map) based on the surface map and the sensor data. For example, the map and semantics generator 104 may include a deep neural network that identifies each object in the surface map, identifies boundaries of the object, applies a label (e.g., cup, cube, bottle, table, etc.) to the object (e.g., surface segments) and assigns a unique value (e.g., an instance identifier) to the object for future reference.
[0027] The map and semantics generator 104 may further generate a part labelled semantic map (e.g., generation of semantic endowed surface regions which may be referred to as semantic patches or surface patches) based on the semantic labelled map and the sensor data. For example, the part labelled semantic map may identify the parts of each objects. As a more detailed example, if a motorcycle is identified, the parts may include a handle, frame, seat, tank and wheel. Each of the parts may be labelled in the part labelled semantic map.
[0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).
Figure 1))
receive a semantic model of the target constrained object generated based on the perception data; (([0024] The map and semantics generator 104 may generate one or more maps based on the image and/or range data. For example, the map and semantics generator 104 may generate an occupancy map to represent an environment of the robot such as an occupancy map (continuous or discrete) that maps occupied spaces. In some embodiments, the map and semantics generator 104 may further map unoccupied spaces and/or unknown spaces (spaces that cannot be identified as occupied or unoccupied) and store the unoccupied spaces and/or unknown spaces in the occupied map or another map.
[0025] The map and semantics generator 104 may further generate a surface map that identifies surfaces based on the sensor data and the occupancy map (e.g., classify the occupied spaces into various surfaces). For example, the surface map may be a structured point-cloud that includes a collection of 3D vertex points linked by edges on the surfaces.
[0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).
[0031] In some embodiments, the location data 112 may provide sensor information which is used by the robot modeler 106 in conjunction with the direct and inverse kinematics to precompute a 3d body-relative reachability map. By using that reachability map and the current state of the scene, the robot modeler 106 may identify a possible set of actions (e.g., actions to physically manipulate the object). In some embodiments the map and semantics generator 104 may provide the scene to the robot modeler 106.
[0032] In some embodiments, robot modeler 106 may receive the one or more maps of map and semantics generator 104 and determine suitable grips for various objects based on the graspability map. For example, the robot modeler 106 may identify that certain grips would be ineffective (e.g., would be unable to manipulate an object, would not be able to hold a slipper object, etc.) for the objects and thus exclude such grips from being propagated to the mission planner 108 as ranked action.
[0078] The robot modeler 354 may therefore efficiently and quickly reject unsuitable grasps while simultaneously providing a mathematical analysis to drive gradient estimations for best grips during, for example, neural network training. The robot modeler 354 may further select grasps from the graspability map 368 and generate grasp actions and rankings based on direct and inverse kinematics and grasping indexes 366. Thus, the robot modeler 354 may generate actionable grasp models, and rank the models based on metrics 364.))
determine a location for the robotic arm to interact with the target constrained object based on the semantic model and the request; and (([0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).
[0032] In some embodiments, robot modeler 106 may receive the one or more maps of map and semantics generator 104 and determine suitable grips for various objects based on the graspability map. For example, the robot modeler 106 may identify that certain grips would be ineffective (e.g., would be unable to manipulate an object, would not be able to hold a slipper object, etc.) for the objects and thus exclude such grips from being propagated to the mission planner 108 as ranked action.)
controlling the robotic arm to manipulate the target constrained object based on the location for the robotic arm to interact with the target constrained object. (([0037] The mission planner 108 may provide the decomposed commands and plans 126 to the end effector controller 128 (e.g., a processor on the end effector that controls actions). Additionally, the robot modeler 106 may provide the ranked action to the end effector controller 128, and the map and semantics generator 104 may provide the map and semantic data to the end effector controller 128. The end effector controller 128 controls the robotic end effector 130, 132 to implement the decomposed commands and plans that include actions that are identified by the mission planner 108 (e.g., intersections of actions). The sensor array 102 may further provide sensor data 134 to the end effector controller 128 so the end effector controller 128 may control the end effector 130 based on updated sensor data (e.g., positional data).))
Gonzalez Aguirre does not teach a body;
two or more legs coupled to the body;
Da Silva, however, does teach a body; (Figure 2
Page 24, Column 17, Lines 7 to 22, (82) In FIG. 6A, the quadruped robot 602 is in a first state 600. The first state 600 includes a set of joint angles and/or actuator forces for each of the quadruped robot's legs (which may also be referred to herein as the starting “body pose”) and the manipulator arm (which may also be referred to herein as the starting “arm pose”). A control system of quadruped robot 602 may control actuators of quadruped robot 602 to put the quadruped robot 602 in the first state 600 in response to receiving feed-forward control inputs specifying the starting body pose and the starting arm pose. The received feed-forward control inputs may also specify an ending body pose and ending arm pose for the quadruped robot 602, along with a timing parameter that specifies the duration of the motion in transitioning from the starting pose to the ending pose.)
two or more legs coupled to the body; (Figure 2
Page 24, Column 17, Lines 7 to 22, (82) In FIG. 6A, the quadruped robot 602 is in a first state 600. The first state 600 includes a set of joint angles and/or actuator forces for each of the quadruped robot's legs (which may also be referred to herein as the starting “body pose”) and the manipulator arm (which may also be referred to herein as the starting “arm pose”). A control system of quadruped robot 602 may control actuators of quadruped robot 602 to put the quadruped robot 602 in the first state 600 in response to receiving feed-forward control inputs specifying the starting body pose and the starting arm pose. The received feed-forward control inputs may also specify an ending body pose and ending arm pose for the quadruped robot 602, along with a timing parameter that specifies the duration of the motion in transitioning from the starting pose to the ending pose.)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Gonzalez Aguirre in light of Da Silva so that the robot has a body and legs attached because that would allow the robot to move around a location as opposed to being a robotic arm attached to a table. This would be expected to be successful at allowing the robot to maneuver around a work space and perform tasks without the tasks having to be brought to the robot.
For Claim 13, Gonzalez Aguirre teaches The robot of claim 12, wherein the target constrained object is constrained in at least one degree of freedom (DoF) of movement. (([0027] The map and semantics generator 104 may further generate a part labelled semantic map (e.g., generation of semantic endowed surface regions which may be referred to as semantic patches or surface patches) based on the semantic labelled map and the sensor data. For example, the part labelled semantic map may identify the parts of each objects. As a more detailed example, if a motorcycle is identified, the parts may include a handle, frame, seat, tank and wheel. Each of the parts may be labelled in the part labelled semantic map.
Depending on the type of part (such as a handle) the part would be constrained in at least one degree of freedom).)
For Claim 14, Gonzalez Aguirre teaches The robot of claim 12, wherein the request comprises an indication of the target constrained object and an instruction for manipulating the target constrained object. (([0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).
[0036] In some embodiments, the mission planner 108 may receive the one or more maps of the map and semantics generator 104 and/or an identification of ranked actions from the robot modeler 106. The mission planner 108 may determine a resulting symbolic plan with attributes and active subsets of actions based on the one or more maps of the map and semantics generator 104. The active subset of actions may be a resulting intersection of actions (e.g., verbs linked by noun names in the segmented parts) from the one or more maps from the map and semantics generator 104, actions from the ranked actions from the robot modeler 106 to implement the plan and actions (e.g., granular atomic-actions) identified from the command data 114.
Figure 1))
For Claim 17, Gonzalez Aguirre teaches The robot of claim 12, wherein receiving the semantic model comprises determining the semantic model by:
identifying a graspable portion of the target constrained object within the perception data and identifying a location where the graspable portion is attached to a remainder of the target constrained object; ([0030] The robot modeler 106 may identify a current location of the robot based on the location data 112 for example. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner 108. For example, the robot modeler 106 may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector 132. The robot modeler 106 may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler 106 may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).
[0032] In some embodiments, robot modeler 106 may receive the one or more maps of map and semantics generator 104 and determine suitable grips for various objects based on the graspability map. For example, the robot modeler 106 may identify that certain grips would be ineffective (e.g., would be unable to manipulate an object, would not be able to hold a slipper object, etc.) for the objects and thus exclude such grips from being propagated to the mission planner 108 as ranked action.
[0033] The map and semantics generator 104 may provide the map and semantic data 118 to the mission planner 108 and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).)
identifying a plurality of axes of the target constrained object;
identifying an axis of rotation of the target constrained object; and/or
identifying an axis of the target constrained object that can be grasped.
For Claim 18, Gonzalez Aguirre teaches The robot of claim 12, wherein the instructions further cause the data processing hardware to:
determine a set of parameters for manipulating the target constrained object based on the location for the robotic arm to interact with the target constrained object, ((Figure 5,
[0081] Furthermore, actions may be considered macro-plans that provide operational limits corresponding to the specific domain and task and action goals 394. The mission planner 390 may set a maximal speed of an end-effector along a manipulation trajectory (for social space sharing) or keeping containers with orientation limits (e.g., smoothness or responsiveness) to avoid spilling liquids. The case-by-case operational limits may change in each step of the plan, and may be stored in the operation limits 388. Thus, providing this information in the grasp planning may enable filtering both affordances by attribute and prioritizing affordances by range matching.))
wherein controlling the robotic arm to manipulate the target constrained object is further based on the set of parameters. ([0081] Furthermore, actions may be considered macro-plans that provide operational limits corresponding to the specific domain and task and action goals 394. The mission planner 390 may set a maximal speed of an end-effector along a manipulation trajectory (for social space sharing) or keeping containers with orientation limits (e.g., smoothness or responsiveness) to avoid spilling liquids. The case-by-case operational limits may change in each step of the plan, and may be stored in the operation limits 388. Thus, providing this information in the grasp planning may enable filtering both affordances by attribute and prioritizing affordances by range matching.)
For Claim 19, Gonzalez Aguirre teaches The robot of claim 18, wherein the set of parameters comprises an initial direction to apply wrench to manipulate the target constrained object and/or a task type associated with the target constrained object. ([0034] The mission planner 108 may identify a task from a high level directive (e.g., clean the kitchen). For example, the high level directive may be decomposed into a sequence of granular atomic-actions which may be referred to as macro-plans. The macro-plans may not only provide actions (e.g., physical actions to physically manipulate the object) that may be undertaken to complete the task, but further provide operational limits corresponding to the specific domain and task.
[0080] The mission planner 390 may capture and unfold high-level directives from sensor data provided by the sensor array 386 (e.g., “clean the kitchen”). The mission planner 390 may decompose the directive into a fine granular sequence of physical atomic-actions or tasks (e.g., primary task, secondary task, target object part assertion, affordance list, etc.) to accomplish the high level directive. The tasks may be stored in the task information 392.
[0081] Furthermore, actions may be considered macro-plans that provide operational limits corresponding to the specific domain and task and action goals 394. The mission planner 390 may set a maximal speed of an end-effector along a manipulation trajectory (for social space sharing) or keeping containers with orientation limits (e.g., smoothness or responsiveness) to avoid spilling liquids. The case-by-case operational limits may change in each step of the plan, and may be stored in the operation limits 388. Thus, providing this information in the grasp planning may enable filtering both affordances by attribute and prioritizing affordances by range matching.
[0035] For example, the mission planner 108 may set a maximal speed of the robotic end effector 132 along a manipulation trajectory (e.g., for social space sharing) or maintain containers with orientation limits to avoid failure of the mission (e.g., spilling liquids in a container). The case-by-case operational limits may change in each step of the plan. Thus, identifying the operation limits may filter both affordances by attribute and prioritize affordances by range matching.))
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez Aguirre in light of Da Silva in light of Wandzel.
For Claim 15, Gonzalez Aguirre teaches The robot of claim 12, wherein the request includes natural language, and wherein the instructions further cause the data processing hardware to:
Gonzalez Aguirre does not teach parse the natural language using a large language model to generate an indication of the target constrained object and an instruction for manipulating the target constrained object.
Wandzel, however, does teach parse the natural language using a large language model to generate an indication of the target constrained object and an instruction for manipulating the target constrained object. ([0012] In one aspect, the present invention features a method of operating a mobile robot to conduct a multi-object search task within an environment. The mobile robot can include at least one processor that can execute computer readable instructions stored in at least one non-transitory computer readable storage medium to perform various operations including: (1) representing a multi-object search task in an Object-Oriented Partially Observable Markov Decision Process model having at least one belief pertaining to a state and at least one observation space within an environment of the robot, wherein the state is represented in terms of classes and objects and each object has at least one attribute and a semantic label; (2) receiving, at the mobile robot from a user, a language command identifying at least one target object and at least one location corresponding to the target object; (3) updating the at least one belief, associated with the at least one target object, based on the language command; (4) driving the mobile robot to the at least one observation space identified in the updated belief, (5) searching, using at least one sensor on the mobile robot while traversing the at least one observation space identified in the updated belief, for the at least one target object; and (6) notifying the user upon finding the at least one target object.)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date that the command that targets the object and commands to manipulate it come from natural language and an LLM because it would allow a user to give simple commands with little training without having to use a particular type of screen or interface device. It would be expected to be successful at giving an intent to a robotic system.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez Aguirre in light of Da Silva in light of Nogami.
For Claim 16, Gonzalez Aguirre teaches The robot of claim 12, further comprising:
a camera, ([0023] The sensor array 102 may include imaging sensors (e.g., a 2D camera, a 3D depth camera and 6D inertial measurement unit), auditory sensors, range sensors, location sensors and so forth. The sensor array 102 may provide data to the map and semantics generator 104, robot modeler 106 and mission planner 108. For example, the sensor array 102 may provide image data (e.g., a red, green, blue and depth (RGB-D) image data, 3D camera orientation, 3D point-cloud, etc.) and/or range data 110 to the map and semantics generator 104.)
Gonzalez Aguirre does not teach wherein the instructions further cause the data processing hardware to:
display a camera view received from the camera on a screen of a remote device;
receive the request as an input of the remote device; and
display, on the screen, a simulated movement of the target constrained object.
Nogami, however, does teach wherein the instructions further cause the data processing hardware to:
display a camera view received from the camera on a screen of a remote device; ([0009] For example, when a simulation is performed in the robot simulation apparatus 50, as shown in FIG. 8, an action display screen 53a is displayed on the display unit 53 of the robot simulation apparatus 50. On this action display screen 53a, a virtual robot R to be simulated, a camera Ca that images a distal end of the virtual robot R in a virtual space, and a robot sensor Se that detects the position of the virtual robot R in the virtual space are displayed. The virtual robot R to be simulated is a vertical multi-joint robot or a scalar robot. Such a virtual robot R includes a robot body section Ra, which is a proximal end section, and a robot hand Rb, which is a distal end section.)
receive the request as an input of the remote device; and ([0066] The order of processing in the action display program is explained below. As shown in FIG. 4, in the action display program, first, the processor 11 determines, referring to a program counter and the like, whether a command that should be executed is present (step S21). When no command that should be executed is present (NO in step S21), the processor 11 ends the action display program and ends the robot simulation according to the end of the action display program. On the other hand, when a command that should be executed is present (YES in step S21), the processor 11 reads out a start point of an action of the robot body section Ra and an end point of the action of the robot body section Ra from the robot data 12a and calculates an optimum track until the robot body section Ra located at the start point reaches the end point (step S22). In calculating the optimum track, the processor 11 calculates the optimum track from the start point to the end point on the basis of various conditions set in advance such as a condition that the robot body section Ra moves on a shortest track and a condition that a track of the robot body section Ra have a curvature equal to or larger than a predetermined curvature. The processor 11 stays on standby until an interrupt is generated by the interrupt generating unit 11b (NO in step S23).)
display, on the screen, a simulated movement of the target constrained object. ([0070] Subsequently, when the value counted by the process counter 11c reaches the set number of times K (YES in step S27), the processor 11 updates, on the display unit 13, the image of the virtual robot R on the basis of target tracks calculated in track calculation processing performed K times (step S28). The processor 11 determines whether the robot body section Ra reaches the end point. When the robot body section Ra does not reach the end point, the processor 11 stays on standby until the next interrupt is generated (NO in step S29, step S23). On the other hand, when the robot body section Ra reaches the end point, the processor 11 determines again whether a command that should be executed is present (YES in step S29, step S21).
[0009] For example, when a simulation is performed in the robot simulation apparatus 50, as shown in FIG. 8, an action display screen 53a is displayed on the display unit 53 of the robot simulation apparatus 50. On this action display screen 53a, a virtual robot R to be simulated, a camera Ca that images a distal end of the virtual robot R in a virtual space, and a robot sensor Se that detects the position of the virtual robot R in the virtual space are displayed. The virtual robot R to be simulated is a vertical multi-joint robot or a scalar robot. Such a virtual robot R includes a robot body section Ra, which is a proximal end section, and a robot hand Rb, which is a distal end section.
Figure 4)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Gonzalez Aguirre in light of Nogami such that a simulated movement is shown on a display because it would allow a user to see the expected outcome of their command before it is carried out. This would allow the user to prevent miscommunications, as well as help prevent potentially dangerous actions that were not intended.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Jackson et al (US Pub 2020/0019237 A1) relates to communicating commands to a robot with legs, arms, and a body.
Takeda et al (US Pub 2016/0132623 A1) relates to displaying simulations for robots.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRISTAN J GREINER whose telephone number is (571)272-1382. The examiner can normally be reached Mon - Fri 7:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Khoi Tran can be reached at Monday-Thursday. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.J.G./Examiner, Art Unit 3656 /KHOI H TRAN/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights