DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first
inventor to file provisions of the AIA .
Status of Claims
This is the first Office action on the merits. Claims 21-40 are currently pending and addressed below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 21-26, 28, 30-31, 34, and 39-40 are rejected under 35 U.S.C. 103 as being unpatentable over Whitman et al. US 20220179420 A1 (“Whitman”) in view of Fang CN 111552301 B (“Fang”).
For claim 21, Whitman discloses a computer-implemented method of operating a legged robot, wherein the method comprises repeatedly performing algorithmic cycles at the legged robot (See at least the Abstract of Whitman – “… A method for terrain and constraint planning a step plan includes receiving, at data processing hardware of a robot… includes a body and legs… generating, by the data processing hardware, a body-obstacle map, a ground height map, and a step-obstacle map … generating… a body path for movement of the body of the robot while maneuvering in the environment based on the body-obstacle map…”), and wherein each cycle of the algorithmic cycles comprises:
updating a state of the legged robot as well as heterogeneous representations of an environment of the legged robot based on signals from a set of sensors (See at least [0033]-[0046] of Whitman – “… The robot 10 also includes a vision system 30 with at least one imaging sensor or camera 31, each sensor or camera 31 capturing image data or sensor data of the environment 8 surrounding the robot 10 … includes a perception system 110 that receives the image or sensor data 17 from the vision system 30 and generates one or more maps 112, 114, 116 that indicate obstacles in the surrounding environment 8… the vision system 30… include one or more stereo cameras … one or more radar sensors such as a scanning light-detection and ranging (LIDAR) sensor… ranging (LADAR) sensor… other three-dimensional (3D) volumetric image sensor … the constrained step planner 320 may receive the current position and velocity of the CM of the robot 10, feet touchdown and liftoff information (e.g., timing), and swing foot position and/or velocity…”), the heterogeneous representations including a first representation (See at least [0036] of Whitman – “… Using the volumetric 3D map 200, which includes the classified voxels 210, 212, the perception system 110 generates a body-obstacle map 112…represents a two-dimensional (2D) map that annotates or illustrates “keep-out areas” or “no-body regions” for the body 11 of the robot 10… marks each location … as a location that is safe for the body 11 of the robot 10 to travel through or not safe for the body 11 of the robot 10 to travel through…”) and a second representation (See at least [0037] of Whitman – “… the perception system 110 also uses the volumetric 3D map 200 (or the ground height map 116, as discussed in more detail below) to generate a step-obstacle map 114… is similar to the body-obstacle map 112, however, the keep-out areas 213 instead represent areas that steps (i.e., the feet 19 or distal ends of the legs 12) of the robot 10 should not “touch down” at…”);
generating an initial trajectory that avoids obstacles in the first representation, wherein the initial trajectory is generated through a first model based on the first representation and the state as updated last (See at least [0033]-[0046] of Whitman – “… the step planning system 100 includes a perception system 110 that receives the image or sensor data 17 from the vision system 30 and generates one or more maps 112, 114, 116 that indicate obstacles in the surrounding environment 8. The step planning system 100 also includes a control system 300 that receives the maps 112, 114, 116 generated by the perception system 110 and generates a body path or trajectory 510… may receive the current position and velocity of the CM of the robot 10, feet touchdown and liftoff information … adjusts or refines the body path trajectory 510… may account for swaying of the body 11 while stepping through the environment 8…”);
executing a second model based on the initial trajectory, the state (See at least [0046] of Whitman – “… the constrained step planner 320 may receive the current position and velocity of the CM of the robot 10… swing foot position and/or velocity… adjusts or refines the body path trajectory 510... may account for swaying of the body 11 while stepping through the environment…”), and the second representation as updated last, to obtain a collision-free trajectory (See at least [0033]-[0046] of Whitman – “… using the body path 510, generates a step path or step plan 350… control system 300 of the step planning system 100 receives the maps (the body-obstacle map 112, the step-obstacle map 114, and the ground height map 116) from the perception system 110 and generates the step plan 350 for use by the robot 10 to navigate the environment… constrained step planner 320 receives the body trajectory 510 from the body path generator 310 as a starting point for generating the final constrained step locations … determines which gait (e.g., a slow walk, a fast walk, a trot, etc.) provides the most optimal step locations with respect to step obstacles 620 (FIG. 6) … constrained step planner 320, in some implementations, adjusts or refines the body path trajectory 510…”); and
controlling the legged robot according to commands generated based on the collision-free trajectory to cause a legged locomotion of the legged robot (See at least [0033] of Whitman – “… the step planning system 100 includes a perception system 110 that receives the image or sensor data 17 from the vision system 30 and generates one or more maps 112, 114, 116 that indicate obstacles in the surrounding environment 8… generates a body path or trajectory 510 …Using the step plan 350, the robot 10 maneuvers through the environment 8 by following the step plan 350 by placing the feet 19 or distal ends of the leg 12 at the locations indicated by the step plan 350…”).
Whitman fails to specifically disclose the first model being a model trained in reinforcement learning.
However, Fang, in the same field of endeavor teaches the first model being a model trained in reinforcement learning (See at least the Abstract of Fang – “… The invention claims a layered control method for path tracking of robot based on reinforcement learning aiming at path tracking problem of the bionic robot of the salamander, establishing a hierarchical control frame… can improve the tracking precision, eliminate the static error…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Fang teaches a system for a legged robot that uses reinforcement learning for path realization and tracking.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the first model being a model trained in reinforcement learning as taught by Fang, with a reasonable expectation of success, in order to establish a hierarchical control frame and improve tracking precision as specified in at least the Abstract of Fang.
For claim 22, Whitman discloses wherein
executing the second model causes to modify the initial trajectory to avoid a collision with an obstacle in the second representation, whereby the obtained collision-free trajectory is a modified version of the initial trajectory (See at least [0033]-[0046] of Whitman – “… using the body path 510, generates a step path or step plan 350… control system 300 of the step planning system 100 receives the maps (the body-obstacle map 112, the step-obstacle map 114, and the ground height map 116) from the perception system 110 and generates the step plan 350 for use by the robot 10 to navigate the environment… constrained step planner 320 receives the body trajectory 510 from the body path generator 310 as a starting point for generating the final constrained step locations … determines which gait (e.g., a slow walk, a fast walk, a trot, etc.) provides the most optimal step locations with respect to step obstacles 620 (FIG. 6) … constrained step planner 320, in some implementations, adjusts or refines the body path trajectory 510…”).
For claim 23, Whitman discloses wherein said heterogeneous representations are representations of different kinds and/or different dimensions (See at least [0042] of Whitman – “… the perception system 110 also generates a ground height map 116 from the 3D volumetric map 200. The ground height map 116 identifies a height of a ground surface 9 at each location near the robot 10. That is, the ground height map 116, similar to a topographical map, is a 2D map that notes the height of the ground surface 9 at each location in a horizontal plane … The ground height map 116 may be used to help generate the step-obstacle map 114 (e.g., determining when the ground surface is too high or too steep to safely traverse and therefore should be marked as a step obstacle). The perception system 110 generates the ground height map 116, for example, by determining a height of the voxel 210 classified as ground surface 9 in each column of the 3D map. The step-obstacle map 114 may in turn be generated from the ground height map 116. The perception system, optionally, processes both the body-obstacle map 112 and the step-obstacle map 114…”).
For claim 24, Whitman discloses wherein
the first model and the second model form part of a high-level policy model, whereby each of the initial trajectory and the obtained collision-free trajectory is a high-level trajectory for the legged robot (See at least [0045] of Whitman – “… the constrained step planner 320 receives the body trajectory 510 from the body path generator 310 as a starting point for generating the final constrained step locations (e.g., step plan) 350… step planner 320 includes a gait determiner 330 that first determines a gait timing 332 that provides nominal step locations for of the robot 10…”),
the commands are generated as low-level commands (See at least [0045] of Whitman – “… the step solver 340 accepts the gait timing 332 and one or more constraints 342, 342a-n. The step solver 340 applies the constraints 342 to the nominal step locations of the determined gait timing 332 and solves for an optimized step plan 350...”).
For claim 25, Whitman discloses wherein
the method further comprises, at said each cycle, generating said commands by feeding the collision-free trajectory, the state of the legged robot, and one of the heterogeneous representations as updated last, to a third model, which is a low-level policy model, for the third model to repeatedly generate the commands as said low-level commands (See at least [0045]-[0046] of Whitman – “… the constrained step planner 320 receives the body trajectory 510 from the body path generator 310 as a starting point for generating the final constrained step locations … provides the most optimal step locations with respect to step obstacles 620 (FIG. 6) presented in the step-obstacle map 114 (FIG. 6) … constrained step planner 320, in some implementations, adjusts or refines the body path trajectory 510… adjusts both the translation and the yaw of the body 11…”), and the low-level commands are generated at a frequency that is higher than a frequency at which the collision-free trajectory is obtained (See at least [0058] of Whitman – “…the step plan 350 may be regenerated at a high frequency (e.g., 300 Hz) to enable real-time navigation while the robot 10 maneuvers in the environment 8. The perception system 110 may operate at a different frequency than the control system. That is, new maps may be provided to the control system 300 at a rate that is different (e.g., slower) than the rate at which the control system 300 determines a step plan 350…” Examiner notes that the paths are generated based on the maps generated to avoid obstacles).
For claim 26, Whitman discloses wherein
the heterogeneous representations of the sensed environment further include a third representation, which is updated at said each cycle along with the first representation and the second representation (See at least [0045] of Whitman – “… the constrained step planner 320 receives the body trajectory 510 from the body path generator 310 as a starting point for generating the final constrained step locations … provides the most optimal step locations with respect to step obstacles 620 (FIG. 6) presented in the step-obstacle map 114 (FIG. 6) … constrained step planner 320, in some implementations, adjusts or refines the body path trajectory 510… adjusts both the translation and the yaw of the body 11…”), and
the commands are repeatedly generated at said each cycle by feeding the third representation as updated last to the third model, in addition to the updated state of the legged robot and the collision-free trajectory (See at least [0046] of Whitman – “… The constrained step planner 320, in some implementations, receives a variety of other information. For example, the constrained step planner 320 may receive the current position and velocity of the CM of the robot 10, feet touchdown and liftoff information (e.g., timing), and swing foot position and/or velocity. The constrained step planner 320 may also receive the body-obstacle map 112. The constrained step planner 320, in some implementations, adjusts or refines the body path trajectory 510. The adjustment may be minor. For example, the constrained step planner 320 may account for swaying of the body 11 while stepping through the environment 8 (which is not accounted for in the simplified body path trajectory 510)… adjusts translation and not yaw trajectory of the body 11 of the robot 10, and in other implementations, adjusts both the translation and the yaw of the body 11...”).
For claim 28, Whitman discloses wherein
the method further comprises repeatedly updating a root representation based on signals obtained from the set of sensors (See at least [0034] of Whitman – “… The camera(s) 31 of the vision system 30, in some implementations, include one or more stereo cameras (e.g., one or more RGBD stereo cameras). In other examples, the vision system 30 includes one or more radar sensors such as a scanning light-detection and ranging (LIDAR) sensor, or a scanning laser-detection and ranging (LADAR) sensor, a light scanner, a time-of-flight sensor, or any other three-dimensional (3D) volumetric image sensor (or any such combination of sensors). In some implementations, the vision system 30 identifies occupancies of space in the environment 8 based on the captured image or sensor data 17. The perception system 110 may use image data 17 captured by the vision system 30 to generate a 3D point cloud. The point cloud is a set of data points representing surfaces of objects in the environment 8 surrounding the robot 10. From this point cloud, the perception system 110 may generate a 3D space occupancy map 200 (FIG. 2A) based on the previously identified occupancies of space in the environment 8. In some examples, the perception system 110 generates a 3D volumetric map 200, 200a of voxels 210, 212 (FIG. 2A). Each voxel 210, 212 (i.e., cube) represents a 3D space of the environment. The size of each voxel 210, 212 is dependent upon the fidelity of the perception system 110 and the processing capabilities of the vision system 30 and data processing hardware 36. For example, the robot 10 may generate a voxel map 200 (i.e., a 3D occupancy map) of the environment 8 surrounding the robot 10 (e.g., several meters in each direction) where each voxel 210, 212 is a 3 cm cube. For each voxel, the perception system 110 may store a variety of statistics…”), and
each of the first representation, the second representation, and the third representation, is repeatedly updated based on the root representation (See at least [0035]-[0037] of Whitman – “… The perception system 110, in some implementations, classifies (using, for example, a classification algorithm, e.g., linear classifiers, decision trees, neural networks, special purpose logic, etc.) each voxel 210, 212 that contains an object as either a ground surface 9, an obstacle, or other… Using the volumetric 3D map 200, which includes the classified voxels 210, 212, the perception system 110 generates a body-obstacle map 112. The body-obstacle map 112, in some implementations, represents a two-dimensional (2D) map that annotates or illustrates “keep-out areas” or “no-body regions” for the body 11 of the robot 10 … the perception system 110 also uses the volumetric 3D map 200 (or the ground height map 116, as discussed in more detail below) to generate a step-obstacle map 114. The step-obstacle map 114, in some examples, represents a 2D plan view map that illustrates keep-out or “no-step” regions 213 for steps by the legs 12 of the robot 10. That is, the step-obstacle map 114 is similar to the body-obstacle map 112, however, the keep-out areas 213 instead represent areas that steps (i.e., the feet 19 or distal ends of the legs 12) of the robot 10 should not “touch down” at…”).
For claim 30, Whitman discloses wherein
at least two representations of said heterogeneous representations are updated at said each cycle based on signals from distinct subsets of the set of sensors, whereby said at least two of said heterogeneous representations reflect heterogeneous perceptions of the environment (See at least [0034] – “… The camera(s) 31 of the vision system 30, in some implementations, include one or more stereo cameras (e.g., one or more RGBD stereo cameras). In other examples, the vision system 30 includes one or more radar sensors such as a scanning light-detection and ranging (LIDAR) sensor, or a scanning laser-detection and ranging (LADAR) sensor, a light scanner, a time-of-flight sensor, or any other three-dimensional (3D) volumetric image sensor (or any such combination of sensors). In some implementations, the vision system 30 identifies occupancies of space in the environment 8 based on the captured image or sensor data 17… the perception system 110 may generate a 3D space occupancy map 200 (FIG. 2A) based on the previously identified occupancies of space in the environment …” and [0061] of Whitman – “… method 1100 for terrain and constraint planning a step plan. The flowchart starts at operation 1102 by receiving, at data processing hardware 36 of a robot 10, image data 17 of an environment 8 about the robot 10 from at least one image sensor 31. The image sensor 31 may include one or more of a stereo camera, a scanning light-detection and ranging (LIDAR) sensor, or a scanning laser-detection and ranging (LADAR) sensor… identifying… occupancies of space in the environment 8 based on the image data 17. At step 1106, the method 1100 includes generating, by the data processing hardware 36, a three-dimensional space occupancy map 200 based on the identification of occupancies of space in the environment 8… Each voxel 212 may be classified as either a ground surface 9, an obstacle, or other… generating, by the data processing hardware 36, a two-dimensional body-obstacle map 112 based on the three-dimensional space occupancy map 200… generating, by the data processing hardware 36, a ground height map 116 based on the three-dimensional space occupancy map 200. The ground height map 116 identifies a height of the ground surface 9 at each location near the robot 10…”).
For claim 31, Whitman discloses wherein
the set of sensors used to update said two representations include two or more sensors selected from a group consisting of: one or more depth sensors, one or more stereo cameras, one or more time-of-flight sensors, ultrasonic sensors, and one or more Lidars (See at least [0034] – “… The camera(s) 31 of the vision system 30, in some implementations, include one or more stereo cameras (e.g., one or more RGBD stereo cameras). In other examples, the vision system 30 includes one or more radar sensors such as a scanning light-detection and ranging (LIDAR) sensor, or a scanning laser-detection and ranging (LADAR) sensor, a light scanner, a time-of-flight sensor, or any other three-dimensional (3D) volumetric image sensor …”).
For claim 34, Whitman fails to specifically disclose wherein
the method further comprises a preliminary step of training the first model according to a reinforcement learning framework, based on training representations of one or more training environments that are commensurate with said first representation, as well as rewards and states of the legged robot computed in accordance with the training representations and actions of the legged robots.
However, Fang, in the same field of endeavor teaches wherein
the method further comprises a preliminary step of training the first model according to a reinforcement learning framework, based on training representations of one or more training environments that are commensurate with said first representation, as well as rewards and states of the legged robot computed in accordance with the training representations and actions of the legged robots (See at least claim 1 of Fang – “… A hierarchical control method for tracking robot path based on the basis of the reinforcement learning… the upper controller based on the reinforcement learning comprises the design of the state space, the design of the action space and the design of the reward function… the bottom controller outputs the position of 19 joint robot according to the input instruction, and executed in the simulation environment, according to the state of the robot after executing the instruction, generating the reward … the design reinforcement learning the reward function uses reward to guide the robot to learn an optimal strategy, in the task of the robot tracking path… the robot can determine the direction of advance, the reward is defined as negative value, to excite the robot to reach the target position with the minimum control step number…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Fang teaches a system for a legged robot that uses reinforcement learning for path realization and tracking.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the a preliminary step of training the first model according to a reinforcement learning framework, based on rewards and states of the legged robot computed in accordance with the training representations and actions of the legged robot as taught by Fang, with a reasonable expectation of success, in order to establish a hierarchical control frame and improve tracking precision as specified in at least the Abstract of Fang.
For claim 39, Whitman discloses a robot control system for operating a legged robot, wherein the system comprises one or more processors, which is adapted to be interfaced with a set of sensors and are configured to repeatedly perform algorithmic cycles at the legged robot (See at least the Abstract – “… A method for terrain and constraint planning a step plan includes receiving, at data processing hardware of a robot… includes a body and legs… generating, by the data processing hardware, a body-obstacle map, a ground height map, and a step-obstacle map … generating… a body path for movement of the body of the robot while maneuvering in the environment based on the body-obstacle map…” and [0072] of Whitman – “… The processes and logic flows described in this specification can be performed by one or more programmable processors…”), wherein each cycle of the algorithmic cycles comprises, in operation:
updating a state of the legged robot as well as heterogeneous representations of an environment of the legged robot based on signals from a set of sensors (See at least [0033]-[0046] of Whitman – “… The robot 10 also includes a vision system 30 with at least one imaging sensor or camera 31, each sensor or camera 31 capturing image data or sensor data of the environment 8 surrounding the robot 10 … includes a perception system 110 that receives the image or sensor data 17 from the vision system 30 and generates one or more maps 112, 114, 116 that indicate obstacles in the surrounding environment 8… the vision system 30… include one or more stereo cameras … one or more radar sensors such as a scanning light-detection and ranging (LIDAR) sensor… ranging (LADAR) sensor… other three-dimensional (3D) volumetric image sensor … the constrained step planner 320 may receive the current position and velocity of the CM of the robot 10, feet touchdown and liftoff information (e.g., timing), and swing foot position and/or velocity…”), the heterogeneous representations including a first representation (See at least [0036] of Whitman – “… Using the volumetric 3D map 200, which includes the classified voxels 210, 212, the perception system 110 generates a body-obstacle map 112…represents a two-dimensional (2D) map that annotates or illustrates “keep-out areas” or “no-body regions” for the body 11 of the robot 10… marks each location … as a location that is safe for the body 11 of the robot 10 to travel through or not safe for the body 11 of the robot 10 to travel through…”), and a second representation (See at least [0037] of Whitman – “… the perception system 110 also uses the volumetric 3D map 200 (or the ground height map 116, as discussed in more detail below) to generate a step-obstacle map 114… is similar to the body-obstacle map 112, however, the keep-out areas 213 instead represent areas that steps (i.e., the feet 19 or distal ends of the legs 12) of the robot 10 should not “touch down” at…”);
generating an initial trajectory that avoids obstacles in the first representation, wherein the initial trajectory is generated through a first model based on the first representation and the state of the legged robot as updated last (See at least [0033]-[0046] of Whitman – “… the step planning system 100 includes a perception system 110 that receives the image or sensor data 17 from the vision system 30 and generates one or more maps 112, 114, 116 that indicate obstacles in the surrounding environment 8. The step planning system 100 also includes a control system 300 that receives the maps 112, 114, 116 generated by the perception system 110 and generates a body path or trajectory 510… may receive the current position and velocity of the CM of the robot 10, feet touchdown and liftoff information … adjusts or refines the body path trajectory 510… may account for swaying of the body 11 while stepping through the environment 8…”),
executing a second model based on the initial trajectory, as well as the state of the legged robot (See at least [0046] of Whitman – “… the constrained step planner 320 may receive the current position and velocity of the CM of the robot 10… swing foot position and/or velocity… adjusts or refines the body path trajectory 510... may account for swaying of the body 11 while stepping through the environment…”) and the second representation as updated last, to obtain a collision-free trajectory (See at least [0033]-[0046] of Whitman – “… using the body path 510, generates a step path or step plan 350… control system 300 of the step planning system 100 receives the maps (the body-obstacle map 112, the step-obstacle map 114, and the ground height map 116) from the perception system 110 and generates the step plan 350 for use by the robot 10 to navigate the environment… constrained step planner 320 receives the body trajectory 510 from the body path generator 310 as a starting point for generating the final constrained step locations … determines which gait (e.g., a slow walk, a fast walk, a trot, etc.) provides the most optimal step locations with respect to step obstacles 620 (FIG. 6) … constrained step planner 320, in some implementations, adjusts or refines the body path trajectory 510…”), and
controlling the legged robot according to commands generated based on the collision-free trajectory to cause a legged locomotion of the legged robot (See at least [0033] of Whitman – “… the step planning system 100 includes a perception system 110 that receives the image or sensor data 17 from the vision system 30 and generates one or more maps 112, 114, 116 that indicate obstacles in the surrounding environment 8… generates a body path or trajectory 510 …Using the step plan 350, the robot 10 maneuvers through the environment 8 by following the step plan 350 by placing the feet 19 or distal ends of the leg 12 at the locations indicated by the step plan 350…”).
Whitman fails to specifically disclose the first model trained in reinforcement learning.
However, Fang, in the same field of endeavor teaches the first model trained in reinforcement learning (See at least the Abstract of Fang – “… The invention claims a layered control method for path tracking of robot based on reinforcement learning aiming at path tracking problem of the bionic robot of the salamander, establishing a hierarchical control frame… can improve the tracking precision, eliminate the static error…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Fang teaches a system for a legged robot that uses reinforcement learning for path realization and tracking.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the first model being a model trained in reinforcement learning as taught by Fang, with a reasonable expectation of success, in order to establish a hierarchical control frame and improve tracking precision as specified in at least the Abstract of Fang.
For claim 40, Whitman discloses a computer program product for operating a legged robot, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors of the control system to cause the latter to repeatedly perform algorithmic cycles at the legged robot (See at least the Abstract – “… A method for terrain and constraint planning a step plan includes receiving, at data processing hardware of a robot… includes a body and legs… generating, by the data processing hardware, a body-obstacle map, a ground height map, and a step-obstacle map … generating… a body path for movement of the body of the robot while maneuvering in the environment based on the body-obstacle map…” and [0072] of Whitman – “… The processes and logic flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output…”), wherein each cycle of the algorithmic cycles comprises:
updating a state of the legged robot as well as heterogeneous representations of an environment of the legged robot based on signals from a set of sensors (See at least [0033]-[0046] of Whitman – “… The robot 10 also includes a vision system 30 with at least one imaging sensor or camera 31, each sensor or camera 31 capturing image data or sensor data of the environment 8 surrounding the robot 10 … includes a perception system 110 that receives the image or sensor data 17 from the vision system 30 and generates one or more maps 112, 114, 116 that indicate obstacles in the surrounding environment 8… the vision system 30… include one or more stereo cameras … one or more radar sensors such as a scanning light-detection and ranging (LIDAR) sensor… ranging (LADAR) sensor… other three-dimensional (3D) volumetric image sensor … the constrained step planner 320 may receive the current position and velocity of the CM of the robot 10, feet touchdown and liftoff information (e.g., timing), and swing foot position and/or velocity…”), the heterogeneous representations including a first representation (See at least [0036] of Whitman – “… Using the volumetric 3D map 200, which includes the classified voxels 210, 212, the perception system 110 generates a body-obstacle map 112…represents a two-dimensional (2D) map that annotates or illustrates “keep-out areas” or “no-body regions” for the body 11 of the robot 10… marks each location … as a location that is safe for the body 11 of the robot 10 to travel through or not safe for the body 11 of the robot 10 to travel through…”) and a second representation (See at least [0037] of Whitman – “… the perception system 110 also uses the volumetric 3D map 200 (or the ground height map 116, as discussed in more detail below) to generate a step-obstacle map 114… is similar to the body-obstacle map 112, however, the keep-out areas 213 instead represent areas that steps (i.e., the feet 19 or distal ends of the legs 12) of the robot 10 should not “touch down” at…”);
generating an initial trajectory that avoids obstacles in the first representation, wherein the initial trajectory is generated through a first model based on the first representation and the state as updated last (See at least [0033]-[0046] of Whitman – “… the step planning system 100 includes a perception system 110 that receives the image or sensor data 17 from the vision system 30 and generates one or more maps 112, 114, 116 that indicate obstacles in the surrounding environment 8. The step planning system 100 also includes a control system 300 that receives the maps 112, 114, 116 generated by the perception system 110 and generates a body path or trajectory 510… may receive the current position and velocity of the CM of the robot 10, feet touchdown and liftoff information … adjusts or refines the body path trajectory 510… may account for swaying of the body 11 while stepping through the environment 8…”);
executing a second model based on the initial trajectory, the state (See at least [0046] of Whitman – “… the constrained step planner 320 may receive the current position and velocity of the CM of the robot 10… swing foot position and/or velocity… adjusts or refines the body path trajectory 510... may account for swaying of the body 11 while stepping through the environment…”), and the second representation as updated last, to obtain a collision-free trajectory (See at least [0033]-[0046] of Whitman – “… using the body path 510, generates a step path or step plan 350… control system 300 of the step planning system 100 receives the maps (the body-obstacle map 112, the step-obstacle map 114, and the ground height map 116) from the perception system 110 and generates the step plan 350 for use by the robot 10 to navigate the environment… constrained step planner 320 receives the body trajectory 510 from the body path generator 310 as a starting point for generating the final constrained step locations … determines which gait (e.g., a slow walk, a fast walk, a trot, etc.) provides the most optimal step locations with respect to step obstacles 620 (FIG. 6) … constrained step planner 320, in some implementations, adjusts or refines the body path trajectory 510…”); and
controlling the legged robot according to commands generated based on the collision-free trajectory to cause a legged locomotion of the legged robot (See at least [0033] of Whitman – “… the step planning system 100 includes a perception system 110 that receives the image or sensor data 17 from the vision system 30 and generates one or more maps 112, 114, 116 that indicate obstacles in the surrounding environment 8… generates a body path or trajectory 510 …Using the step plan 350, the robot 10 maneuvers through the environment 8 by following the step plan 350 by placing the feet 19 or distal ends of the leg 12 at the locations indicated by the step plan 350…”).
Whitman fails to specifically disclose the first model being a model trained in reinforcement learning.
However, Fang, in the same field of endeavor teaches the first model being a model trained in reinforcement learning (See at least the Abstract of Fang – “… The invention claims a layered control method for path tracking of robot based on reinforcement learning aiming at path tracking problem of the bionic robot of the salamander, establishing a hierarchical control frame… can improve the tracking precision, eliminate the static error…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Fang teaches a system for a legged robot that uses reinforcement learning for path realization and tracking.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the first model being a model trained in reinforcement learning as taught by Fang, with a reasonable expectation of success, in order to establish a hierarchical control frame and improve tracking precision as specified in at least the Abstract of Fang.
Claims 27 and 32 are rejected under 35 U.S.C. 103 as being unpatentable over Whitman in view of Fang, as applied to claim 25 above, and further in view of Whitman et al. US 20210041887 A1 (“Whitman ‘887”).
For claim 27, Whitman discloses wherein
the second representation is a representation of an obstacle map (See at least [0006] of Whitman – “… generating, by the data processing hardware, a body-obstacle map…”), and
the third representation is a representation of an elevation map (See at least [0006] of Whitman – “… generating, by the data processing hardware… a ground height map… based on the image data…”).
Whitman fails to specifically disclose the first representation is a representation of a spherical ego-centric map of the environment.
However, Whitman ‘887, in the same field of endeavor teaches the first representation is a representation of a spherical ego-centric map of the environment (See at least the Abstract of Whitman ‘887 – “… The method also includes generating a spherical depth map based on the current set of sensor data and determining that a change has occurred to an obstacle represented by the voxel map based on a comparison between the voxel map and the spherical depth map…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Whitman ‘887 teaches a system for a legged robot that generates a spherical depth map using sensor data.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the first representation is a representation of a spherical ego-centric map of the environment as taught by Whitman ‘887, with a reasonable expectation of success, in order to determine changes to obstacles detected as specified in at least the Abstract of Whitman ‘887.
For claim 32, Whitman discloses wherein
the collision-free trajectory is obtained at a first average frequency of between 5 and 100 Hertz, the low-level commands are generated at a second average frequency of between 25 and 1000 Hertz (See at least [0058] of Whitman – “…the step plan 350 may be regenerated at a high frequency (e.g., 300 Hz) to enable real-time navigation while the robot 10 maneuvers in the environment 8. The perception system 110 may operate at a different frequency than the control system. That is, new maps may be provided to the control system 300 at a rate that is different (e.g., slower) than the rate at which the control system 300 determines a step plan 350…” Examiner notes that the paths are generated based on the maps generated to avoid obstacles).
Whitman fails to specifically disclose the legged robot is controlled thanks to a motion controller operating at a third average frequency that is larger than, or equal to, the second average frequency.
However, Whitman ‘887, in the same field of endeavor teaches the legged robot is controlled thanks to a motion controller operating at a third average frequency that is larger than, or equal to, the second average frequency (See at least [0046] of Whitman ‘887 – “… the controller 172 includes a plurality of controllers 172 where each of the controllers 172 has a fixed cadence. A fixed cadence refers to a fixed timing for a step or swing phase of a leg 120… the robot 100 continuously switches/selects fixed cadence controllers 172 (e.g., re-selects a controller 170 every 3 milliseconds) as the robot 100 traverses the environment 10…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Whitman ‘887 teaches a system that switches controllers to control a step or swing of a robot leg every 3 milliseconds.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the legged robot being controlled by a motion controller operating at a third average frequency that is larger than, or equal to, the second average frequency as taught by Whitman ‘887, with a reasonable expectation of success, in order to switch cadence controllers as the robot traverses the environment as specified in at least [0046] of Whitman ‘887.
Claim 29 is rejected under 35 U.S.C. 103 as being unpatentable over Whitman in view of Fang, as applied to claim 28 above, and further in view of Olson et al. US 20210046923 A1 (“Olson”) and Urtasun et al. US 20210278852 A1 (“Urtasun”).
For claim 29, Whitman fails to specifically disclose wherein
the root representation is a representation of a sparse, 3D voxel map.
However, Olson, in the same field of endeavor teaches wherein
the root representation is a representation of a sparse, 3D voxel map (See at least [0194] of Olson – “… the voxel map may be represented in a sparse manner (e.g., providing data representing occupied voxels and disregarding unoccupied voxels) or in a dense manner (e.g., without discarding voxels)…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Olson teaches a system for an autonomous vehicle that generates a voxel map represented in a sparse manner.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the root representation being a representation of a sparse, 3D voxel map as taught by Olson, with a reasonable expectation of success, in order to provide data representing occupied voxels and disregarding unoccupied voxels as specified in at least [0194] of Olson.
Furthermore, Whitman also fails to specifically disclose the root representation is a learned map represented by an artificial neural network.
However, Urtasun, in the same field of endeavor teaches the root representation is a learned map represented by an artificial neural network (See at least [0024] of Urtasun – “… To generate the attention mask, the vehicle computing system can input the voxel grid representation of the sensor data and map data obtained from a map database to a machine-learned model. The machine-learned model can be a neural network that generates a scalar score for each spatial location in the voxel grid representation…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Urtasun teaches a system for an autonomous vehicle that generates an attention mask using a map represented by neural network.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the root representation being a learned map represented by an artificial neural network as taught by Urtasun, with a reasonable expectation of success, in order to generate a scalar score for each spatial location in the voxel grid representation as specified in at least [0024] of Urtasun.
Claim 33 is rejected under 35 U.S.C. 103 as being unpatentable over Whitman in view of Fang, as applied to claim 25 above, and further in view of Jin CN 116257085 A (“Jin”).
For claim 33, Whitman fails to specifically disclose wherein
the first model comprises an actor network including a self-attention layer and an output network, wherein the self-attention layer is connected to the output network, and the second model includes one or each of a geometric obstacle detection model and a trained model, and the third model includes one or each of a model based on model predictive control and a trained model.
However, Jin, in the same field of endeavor teaches wherein
the first model comprises an actor network including a self-attention layer and an output network, wherein the self-attention layer is connected to the output network, and the second model includes one or each of a geometric obstacle detection model and a trained model, and the third model includes one or each of a model based on model predictive control and a trained model (See at least the Abstract of Jin – “… an unmanned aerial vehicle unknown environment path planning method based on depth reinforcement learning, comprising: designing the vivid unmanned aerial vehicle obstacle avoidance environment in the simulation engine… adding self-attention module in the Actor-Critic network structure based on traditional PPO algorithm, forming improved PPO algorithm; planning the path of the unmanned aerial vehicle based on the improved PPO algorithm in the simulation environment; putting the weight obtained by training into the unmanned aerial vehicle computer, performing path planning verification of the actual environment. The improved Actor-Critic network structure provided by the invention can reduce the training time, improve the convergence speed of the network, and the network structure is stable, the parameter is small, and it is suitable for being deployed on small mobile device such as computing power computer…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Jin teaches a system that performs path planning for a vehicle using an actor-critic network structure.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the first model comprising an actor network including a self-attention layer and an output network as taught by Jin, with a reasonable expectation of success, in order to reduce the training time and improve the convergence speed of the network as specified in at least the Abstract of Jin.
Claim 35 is rejected under 35 U.S.C. 103 as being unpatentable over Whitman in view of Fang, as applied to claim 34 above, and further in view of Bohez et al. US 20250224737 A1 (“Bohez”).
For claim 35, Whitman fails to specifically disclose wherein the first model includes an actor-critic network, which comprises an actor network for inferring a desired action, the actor network including a self-attention layer and a first output network, the self-attention layer connected to an output network, and a critic network for estimating a value function, the critic network including a self-attention layer connected to a second output network, and training the first model includes jointly training the actor network and the critic network, for the actor network to learn to generate said initial trajectory at said each cycle at runtime.
However, Bohez, in the same field of endeavor teaches wherein the first model includes an actor-critic network, which comprises an actor network for inferring a desired action, the actor network including a self-attention layer and a first output network, the self-attention layer connected to an output network, and a critic network for estimating a value function, the critic network including a self-attention layer connected to a second output network, and training the first model includes jointly training the actor network and the critic network (See at least [0045] – “…the training system 190 can perform the training using an actor-critic reinforcement learning technique, e.g., Maximum a Posteriori Policy Optimization (MPO) or another appropriate technique…” and [0171] of Bohez – “…the architecture of the task policy neural network 122 is dependent on the type of information included in the observation… sensor data includes low-dimensional data… the task policy neural network 122 include an MLP to encode the low-dimensional data, e.g., the concatenation of the sensor data, the task data, and the preceding latent action vector… the task policy neural network 122 can include an MLP to encode the low-dimensional data and a convolutional neural network, a self-attention neural network or a neural network that includes both convolutional and self-attention layers to encode the higher-dimensional data…”), for the actor network to learn to generate said initial trajectory at said each cycle at runtime (See at least [0184] of Bohez – “… the system also employs a value neural network during the imitation phase and uses an actor-critic reinforcement learning technique … the value neural network can be provided with additional information characterizing the reference trajectory…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Bohez teaches a system that controls a robot using a latent action vector with an actor-critic reinforcement learning technique to determine a trajectory for a robot.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the first model including an actor-critic network, which comprises an actor network for inferring a desired action, the actor network including a self-attention layer and a first output network as taught by Bohez, with a reasonable expectation of success, in order to characterize a reference trajectory as specified in at least [0184] of Bohez.
Claims 36-37 are rejected under 35 U.S.C. 103 as being unpatentable over Whitman in view of Fang and Bohez, as applied to claim 35 above, and further in view of Myung et al. US 20250018560 A1 (“Myung”).
For claim 36, Whitman fails to specifically disclose wherein the first model is trained according to a proximal policy optimisation (PPO) algorithm.
However, Myung, in the same field of endeavor teaches wherein the first model is trained according to a proximal policy optimisation (PPO) algorithm (See at least [0060] of Myung – “… For learning implicit terrain imagination, the asymmetric actor-critic architecture is adopted… this policy may explore all possible trajectories during training, improving its robustness through generalization. In the example embodiment, the policy may be optimized using a proximal policy optimization (PPO) algorithm…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Myung teaches a legged robot system that uses proximal policy optimization.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the first model being trained according to a proximal policy optimisation (PPO) algorithm as taught by Myung, with a reasonable expectation of success, in order to explore all possible trajectories during training and optimize the policy as specified in at least [0060] of Myung.
For claim 37, Whitman fails to specifically disclose wherein the first model is gradually trained by gradually increasing a difficulty of terrain in the training environment and/or by changing a composition and/or scaling weights of the computed rewards.
However, Myung, in the same field of endeavor teaches wherein the first model is gradually trained by gradually increasing a difficulty of terrain in the training environment and/or by changing a composition and/or scaling weights of the computed rewards (See at least [0074] of Myung – “… the example embodiment, a game-inspired curriculum may be employed to ensure progressive locomotion policy learning over difficult terrains. The terrains may include smooth, rough, discretized, and stair terrains with ten levels of inclination within [0°, 22°] …”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Myung teaches a legged robot system that trains a policy over difficult terrains with different levels of inclination.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the first model being gradually trained by gradually increasing a difficulty of terrain in the training environment as taught by Myung, with a reasonable expectation of success, in order to ensure progressive locomotion policy learning over difficult terrains as specified in at least [0074] of Myung.
Claim 38 is rejected under 35 U.S.C. 103 as being unpatentable over Whitman in view of Fang, as applied to claim 34 above, and further in view of Zhang et al. US 20250021109 A1 (“Zhang”).
For claim 38, Whitman discloses wherein
the first model and the second model form part of a high-level policy model, whereby each of the initial trajectory and the obtained collision-free trajectory is a high-level trajectory for the legged robot (See at least [0045] of Whitman – “… the constrained step planner 320 receives the body trajectory 510 from the body path generator 310 as a starting point for generating the final constrained step locations (e.g., step plan) 350… step planner 320 includes a gait determiner 330 that first determines a gait timing 332 that provides nominal step locations for of the robot 10…”),
the method further comprises, at said each cycle, generating said commands by feeding the collision-free trajectory, the state of the legged robot, and one of the heterogeneous representations as updated last, to a third model, which is a low-level policy model, for the third model to repeatedly generate the commands as low-level commands (See at least [0045]-[0046] of Whitman – “… the constrained step planner 320 receives the body trajectory 510 from the body path generator 310 as a starting point for generating the final constrained step locations … provides the most optimal step locations with respect to step obstacles 620 (FIG. 6) presented in the step-obstacle map 114 (FIG. 6) … constrained step planner 320, in some implementations, adjusts or refines the body path trajectory 510… adjusts both the translation and the yaw of the body 11…”) at a frequency that is higher than a frequency at which the collision-free trajectory is obtained (See at least [0058] of Whitman – “…the step plan 350 may be regenerated at a high frequency (e.g., 300 Hz) to enable real-time navigation while the robot 10 maneuvers in the environment 8. The perception system 110 may operate at a different frequency than the control system. That is, new maps may be provided to the control system 300 at a rate that is different (e.g., slower) than the rate at which the control system 300 determines a step plan 350…” Examiner notes that the paths are generated based on the maps generated to avoid obstacles).
Whitman fails to specifically disclose the third model being a trainable model, and the method further comprises, prior to repeatedly performing said algorithmic cycles at the legged robot, training the third model, in an interlaced manner with the first model.
However, Zhang, in the same field of endeavor teaches the third model being a trainable model, and the method further comprises, prior to repeatedly performing said algorithmic cycles at the legged robot, training the third model, in an interlaced manner with the first model (See at least [0019] of Zhang – “… foot sole position of each leg of the quadruped robot in the Z-axis direction is outputted by using an independent trajectory generator; and a foot sole position increment and an adjusting frequency of each leg are outputted based on the reinforcement learning policy, the foot sole position increments in the X-axis and Y-axis directions are accumulated to obtain foot sole positions in the X-axis and Y-axis directions, and the foot sole position increment in the Z-axis direction and a priori value are superposed to obtain a foot sole position in the Z-axis direction…”). Thus, Whitman discloses a system that detects a surrounding environment for a legged robot that uses models to build maps using sensor data to generate a path for the legged robot to maneuver through while avoiding obstacles, while Zhang teaches a system for controlling movement of the legs of a robot using motion environment information and reinforcement learning techniques.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the computer-implemented method, robot control system, and computer program product for operating a legged robot as disclosed in Whitman to include the feature of the third model being a trainable model, and the method further comprises, prior to repeatedly performing said algorithmic cycles at the legged robot, training the third model, in an interlaced manner with the first model as taught by Zhang, with a reasonable expectation of success, in order to adjust movement of each leg for the robot as specified in at least [0019] of Zhang.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J HERRERA whose telephone number is (571)270-5271. The examiner can normally be reached M-F 10:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, FADEY JABR can be reached at (571)272-1516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.J.H./Examiner, Art Unit 3668 /Fadey S. Jabr/Supervisory Patent Examiner, Art Unit 3668
Prosecution Insights