DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 01/02/2025 and 12/05/2024 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the IDS(s) has/have been considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-5 & 7-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu et al. (US 20220410380 A1, hereinafter Lu) in view of Vinod et al. (US 20230367336 A1, hereinafter Vinod).
Regarding claim 1, Lu teaches:
A method of controlling a robot, the method comprising:
training an agent through an actor-critic algorithm for deep reinforcement learning (DRL) (at least as in paragraph 0018, “After pre-training the actor and critic networks based on demonstration data, implementations further train the actor network and the critic network using RL and online (but potentially off-policy) episode data from robotic episodes each performed based on the actor network and/or the critic network”; at least as in paragraph 0053, “the system can perform method 200 to generate online episode data and simultaneously perform method 300 to further train the actor network and the critic network based on online episode data that is generated based on method 200”), wherein the training of the agent comprises:
identifying a robot (at least as in paragraph 0065, “iterations of method 200 can be performed across multiple real physical robots and/or across multiple simulators”; at least as in paragraph 0043, “At block 104, the system identifies one or more instances of offline robotic demonstration data”);
obtaining initial state data from the robot cluster, (at least as in paragraph 0055, “At block 202, generation of episode data begins”; at least as in paragraph 0057, “At block 206, the system processes current state data… At an initial iteration of generating episode data, the current action network and the current critic network can be as pre-trained according to method 100”; at least as in paragraph 0063, “the system can store various instances of transitions during the episode and a reward for the episode. Each transition can include state data”);
sending, by an actor, an action to the robot cluster based on the initial state data (at least as in paragraph 0057, “At block 206, the system processes current state data, using the current action network and/or the current critic network, to select the next action”; at least as in paragraph 0059, “At block 208, the system executes the next action”);
obtaining late state data from the robot cluster after the robot cluster has moved based on the action, (at least as in paragraph 0063, “At block 216, the system stores episode data from the episode. For example, the system can store various instances of transitions during the episode and a reward for the episode. Each transition can include state data, action, and next state data (i.e., next state data from the next state that resulted from the action). In some implementations, block 216 includes sub-block 216A, in which the system populates some or all of the stored episode data in a replay buffer for use in method 300 of FIG. 3. In some of those implementations, whether the system populates the stored episode data in the replay buffer can depend on whether the reward, for the episode data, is positive—indicating a successful episode (i.e., one in which the robotic task was successfully performed)”); and
inputting a reward to a critic, wherein the reward is based on the initial state data and the late state data (at least as in paragraph 0062, “If, at block 210, the system determines to perform another step in the episode, the system proceeds to block 214 and determines a reward for the episode. The reward can be determined based on a defined reward function, which will be dependent on the robotic task”).
But Lu does not explicitly teach:
comprising a plurality of robots each configured to move from a starting point to a destination…
wherein the initial state data comprises information about a first location…
wherein the late state data comprises information about a second location reached by the robot cluster.
However, Vinod, in the same field of endeavor of robot control and motion planning, specifically teaches:
comprising a plurality of robots each configured to move from a starting point to a destination (at least as in paragraph 0056, “FIG. 2A shows an environment 200 including multiple devices, wherein each device has a respective target state… The devices 201a, 201b, 201c, and 201d, are required to reach their assigned target states 203a, 203b, 203c, and 203d”)…
wherein the initial state data comprises information about a first location (at least as in paragraph 0058, “Learned functions 207a-207d produces initial trajectories 209a-209d, respectively”)…
wherein the late state data comprises information about a second location reached by the robot cluster (at least as in paragraph 0058, “The initial trajectories 209a-209d are applied to an optimization-based safety filter 211. The optimization-based safety filter 211 solves the joint-optimization problem to simultaneously determine optimal trajectories 231a-213d”; at least as in paragraph 0059, “the controller 111 controls each device based on a pre-determined number of states of their respective optimal trajectory. When the devices 201a-201d are controlled based on their respective optimal trajectory, the devices 201a-201d attain new states 215a-215d. The new states 215a-215d are fed back 217 to the learned functions 207a-207d, respectively”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Vinod's teaching of training a control system by optimizing an initial trajectory for a plurality of devices, since Vinod teaches wherein the motion planning system is easier to train, requires less computational resources, and demonstrates superior performance.
Regarding claim 2, the above combination of Lu and Vinod teaches the method of claim 1 but does not explicitly teach wherein the robot cluster comprises three robots closest to each other from among the plurality of robots.
However, Vinod, in the same field of endeavor of robot control and motion planning, specifically teaches wherein the robot cluster comprises three robots closest to each other from among the plurality of robots (at least as in paragraph 0056, wherein environment 200 includes devices 201a-d).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Vinod's teaching of training a control system by optimizing an initial trajectory for a plurality of devices, since Vinod teaches wherein the motion planning system is easier to train, requires less computational resources, and demonstrates superior performance.
Regarding claim 3, in view of the above combination of Lu and Vinod, Lu further teaches:
The method of claim 1, wherein the reward comprises a value obtained by applying the initial state data and the late state data to a reward function (at least as in paragraph 0062, “If, at block 210, the system determines to perform another step in the episode, the system proceeds to block 214 and determines a reward for the episode. The reward can be determined based on a defined reward function, which will be dependent on the robotic task”).
Regarding claim 4, the above combination of Lu and Vinod teaches the method of claim 3, but does not explicitly teach wherein the reward function comprises a goal value comprising information about whether each robot of the robot cluster arrives at the destination, a penalty value comprising information about whether each robot of the robot cluster is in a vicinity of an obstacle, and a leading value for obtaining a different value according to a positional relation between the destination and the obstacle.
However, Vinod, in the same field of endeavor of robot control and motion planning, specifically teaches:
wherein the reward function comprises a goal value comprising information about whether each robot of the robot cluster arrives at the destination, a penalty value comprising information about whether each robot of the robot cluster is in a vicinity of an obstacle, and a leading value for obtaining a different value according to a positional relation between the destination and the obstacle (at least as in paragraph 0049, “the learned function is a neural network trained using a reinforcement learning based on a reward function. The reward function penalizes the extent of violation of the constraints. The reward function returns higher values when the initial trajectory 117 satisfies the constraints as compared to when the constraints are violated (i.e., the reward function penalizes upon violation of the constraints). In an embodiment, an algorithm such as a proximal policy optimization is used to determine an initial trajectory that maximizes the reward function”; at least as in paragraph 0068, wherein the reward is based on the distance to target and penalty on control effort; at least as in paragraph 0074, “the learned function is trained to penalize the violation of only the static constraint”; at least as in paragraph 0077, “Some embodiments of the present disclosure consider a task of ensuring that an overall system having multiple devices is probabilistically collectively safe, a collection of constraints that ensures safe motion planning for the multiple devices. With a high user-specified likelihood, it is desired that all the devices, each with dynamics (1a), i) stay within a pre-determined environment bound, ii) do not collide with N.sub.O static obstacles located at c.sub.j, 1≤j≤N.sub.O denotes nominal mean positions of the obstacles for all the N.sub.O obstacles with radii γ.sub.j, 1≤j≤N.sub.O respectively, and iii) do not collide with each other given device radii r.sub.A”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Vinod's teaching of training a control system by optimizing an initial trajectory for a plurality of devices, since Vinod teaches wherein the motion planning system is easier to train, requires less computational resources, and demonstrates superior performance.
Regarding claim 5, in view of the above combination of Lu and Vinod, Lu further teaches:
The method of claim 4, wherein, based on the late state data comprising information indicating that the robot cluster has not arrived at the destination, the goal value is 0 (at least as in paragraph 0022, “the actor network can be trained based solely on successful episode data with positive rewards (i.e., episode data from successful episodes). In such an example, the quantity of unsuccessful episode data on which the actor network is trained is zero”).
Regarding claim 7, the above combination of Lu and Vinod teaches the method of claim 4 but does not explicitly teach wherein,
based on the late state data comprising information indicating that the robot cluster is closer to the destination than to the obstacle, the leading value is a first positive number, and
based on the late state data comprising information indicating that the robot cluster is closer to the obstacle than to the destination, the leading value is a second positive number that is smaller than the first positive number.
However, Vinod, in the same field of endeavor of robot control and motion planning, specifically teaches:
based on the late state data comprising information indicating that the robot cluster is closer to the destination than to the obstacle, the leading value is a first positive number (at least as in paragraph 0049, “the learned function is a neural network trained using a reinforcement learning based on a reward function. The reward function penalizes the extent of violation of the constraints. The reward function returns higher values when the initial trajectory 117 satisfies the constraints as compared to when the constraints are violated (i.e., the reward function penalizes upon violation of the constraints). In an embodiment, an algorithm such as a proximal policy optimization is used to determine an initial trajectory that maximizes the reward function”; at least as in paragraph 0068, wherein the reward is based on the distance to target and penalty on control effort; at least as in paragraph 0074, “the learned function is trained to penalize the violation of only the static constraint”; at least as in paragraph 0077, “Some embodiments of the present disclosure consider a task of ensuring that an overall system having multiple devices is probabilistically collectively safe, a collection of constraints that ensures safe motion planning for the multiple devices. With a high user-specified likelihood, it is desired that all the devices, each with dynamics (1a), i) stay within a pre-determined environment bound, ii) do not collide with N.sub.O static obstacles located at c.sub.j, 1≤j≤N.sub.O denotes nominal mean positions of the obstacles for all the N.sub.O obstacles with radii γ.sub.j, 1≤j≤N.sub.O respectively, and iii) do not collide with each other given device radii r.sub.A”), and
based on the late state data comprising information indicating that the robot cluster is closer to the obstacle than to the destination, the leading value is a second positive number that is smaller than the first positive number (at least as in paragraph 0065, “An observation vector o(k) is defined as a concatenated vector containing the state x(k), a displacement of the device's current position to the target (p (k)−q) and to the N.sub.O static obstacles (p(k)−c.sub.j), where c.sub.j denotes nominal mean positions of the obstacles for all 1≤j≤N.sub.O.”; at least as in paragraph 0068, “R(o(k),a(k))=Distanceofp(k)totargetq+Penaltyoncontrolefforta(k)+(Weightedsumof1.Math.p(k)-cj.Math.2-γj2)(3)”; at least as in paragraph 0079, “FIG. 5B illustrates a convex approximation of the second constraint—do not collide with N.sub.O static obstacles, according to some embodiments of the present disclosure”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Vinod's teaching of trained using a reinforcement learning based on a reward function, since Vinod teaches wherein the motion planning system is easier to train, requires less computational resources, and demonstrates superior performance.
Regarding claim 8, the above combination of Lu and Vinod teaches the method of claim 7 but does not explicitly teach wherein an absolute value of the first positive number increases as the robot cluster gets closer to the destination.
However, Vinod, in the same field of endeavor of robot control and motion planning, specifically teaches wherein an absolute value of the first positive number increases as the robot cluster gets closer to the destination (at least as in paragraph 0065, “Observation space: An observation vector o(k) is defined as a concatenated vector containing the state x(k), a displacement of the device's current position to the target (p (k)−q) and to the N.sub.O static obstacles (p(k)−c.sub.j)”; at least as in paragraph 0066, “Action space: An action a(k) determines a reference position as a bounded perturbation a(k) to the target position q, such that r(k)=q+a(k)”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Vinod's teaching of trained using a reinforcement learning based on a reward function, since Vinod teaches wherein the motion planning system is easier to train, requires less computational resources, and demonstrates superior performance.
Regarding claim 9, in view of the above combination of Lu and Vinod, Lu further teaches the method of claim 1, wherein the initial state data further comprises light detection and ranging (LiDAR) sensing information and location information of the robot cluster obtained before the robot cluster moves based on the action (at least as in paragraph 0039, “Techniques disclosed herein can be utilized in combination with various real and/or simulated robots, such as a telepresence robot, a wheeled robot, mobile forklift robot, a robot arm, an unmanned aerial vehicle (“UAV”), and/or a humanoid robot. The robot(s) can include various sensor component(s) and state data that is utilized in techniques disclosed herein can include sensor data that is generated by those sensor component(s) (e.g., images from a camera and/or other vision data from other vision component(s)) and/or can include state data that is derived from such sensor data (e.g., object bounding box(es) derived from vision data). As a particular example, a robot can include vision component(s) such as, for example, a monographic camera (e.g., generating 2D RGB images), a stereographic camera (e.g., generating 2.5D RGB-D images), and/or a laser scanner (e.g., LIDAR generating a 2.5D depth (D) image or point cloud)”).
Regarding claim 10, the above combination of Lu and Vinod teaches the method of claim 1 but does not explicitly teach wherein the late state data further comprises light detection and ranging (LiDAR) sensing information and location information of the robot cluster obtained after the robot cluster has moved based on the action.
However, Vinod, in the same field of endeavor of robot control and motion planning, specifically teaches wherein the late state data further comprises light detection and ranging (LiDAR) sensing information and location information of the robot cluster obtained after the robot cluster has moved based on the action (at least as in paragraph 0039, “a robot can include vision component(s) such as, for example, a monographic camera (e.g., generating 2D RGB images), a stereographic camera (e.g., generating 2.5D RGB-D images), and/or a laser scanner (e.g., LIDAR generating a 2.5D depth (D) image or point cloud)”; at least as in paragraph 0076, “The robot 420 includes a robot control system 460, one or more operational components 440a-440n, and one or more sensors 442a-442m. The sensors 442a-442m may include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and so forth”; at least as in paragraph 0045, “The state data and next state data can include, for example, environmental state data (e.g., image(s) and/or other vision data captured by vision component(s) of a robot) and/or current robot state data (e.g., that indicates a current state of component(s) of the robot)”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Vinod's teaching of trained using a reinforcement learning based on a reward function, since Vinod teaches wherein the motion planning system is easier to train, requires less computational resources, and demonstrates superior performance.
Regarding claim 11, in view of the above combination of Lu and Vinod, Lu further teaches the method of claim 1, wherein the action comprises information about a rotation speed of a wheel of each robot of the robot cluster (at least as in paragraph 0039, “Techniques disclosed herein can be utilized in combination with various real and/or simulated robots, such as a telepresence robot, a wheeled robot, mobile forklift robot, a robot arm, an unmanned aerial vehicle (“UAV”), and/or a humanoid robot”; at least as in paragraph 0045, “The robotic action can include a representation of movement of one or more robotic component(s). As one example, the robotic action can indicate, in Cartesian space, a translation and/or rotation of an end effector of a robot”).
Regarding claim 12, A method of controlling a robot via a computer device including at least one processor, the method comprising:
training, by the at least one processor, an agent through an actor-critic algorithm on a simulation for deep reinforcement learning (at least as in paragraph 0018, “After pre-training the actor and critic networks based on demonstration data, implementations further train the actor network and the critic network using RL and online (but potentially off-policy) episode data from robotic episodes each performed based on the actor network and/or the critic network”; at least as in paragraph 0053, “the system can perform method 200 to generate online episode data and simultaneously perform method 300 to further train the actor network and the critic network based on online episode data that is generated based on method 200”), wherein the training of the agent comprises:
inputting initial state data and late state data to an actor network (at least as in paragraph 0063, “At block 216, the system stores episode data from the episode. For example, the system can store various instances of transitions during the episode and a reward for the episode. Each transition can include state data, action, and next state data (i.e., next state data from the next state that resulted from the action). In some implementations, block 216 includes sub-block 216A, in which the system populates some or all of the stored episode data in a replay buffer for use in method 300 of FIG. 3. In some of those implementations, whether the system populates the stored episode data in the replay buffer can depend on whether the reward, for the episode data, is positive—indicating a successful episode (i.e., one in which the robotic task was successfully performed)”) and
inputting a reward to a critic network in the actor-critic algorithm (at least as in paragraph 0062, “If, at block 210, the system determines to perform another step in the episode, the system proceeds to block 214 and determines a reward for the episode. The reward can be determined based on a defined reward function, which will be dependent on the robotic task”),
determining, by an evaluation network of the actor network, an action of the agent (at least as in paragraph 0057, “At block 206, the system processes current state data, using the current action network and/or the current critic network, to select the next action”; at least as in paragraph 0059, “At block 208, the system executes the next action”), and
evaluating, by a value network of the critic network, a degree to which the action of the agent maximizes a preset reward (at least as in paragraph 0070, “Block 308 includes optional sub-block 308A, which can share one or more (e.g., all) aspects in common with block 108A of FIG. 1”; at least as in paragraph 0050, “as illustrated by optional sub-block 108A of block 108, the training objective can optionally utilize the Max Q-value generated at sub-block 106B”; at least as in paragraph 0048, “Instead of always using the maximum candidate action measure (e.g., Q-value) from CEM as the maximum value for training of the critic network (and optionally in the advantage function for training of the actor network) as is typical, the system can compare the actor action measure to the maximum candidate action measure—and use the greater of the two measures as the maximum value for training”),
wherein the reward comprises a value obtained by applying the initial state data and the late state data to a reward function (at least as in paragraph 0062, “If, at block 210, the system determines to perform another step in the episode, the system proceeds to block 214 and determines a reward for the episode. The reward can be determined based on a defined reward function, which will be dependent on the robotic task”).
But Lu does not explicitly teach:
wherein the initial state data comprises data about a first location of a robot cluster including a plurality of robots each configured to move from a starting point to a destination,
wherein the late state data comprises data about a second location reached by the robot cluster after the robot cluster has moved according to the action.
However, Vinod, in the same field of endeavor of robot control and motion planning, specifically teaches:
wherein the initial state data comprises data about a first location of a robot cluster including a plurality of robots each configured to move from a starting point to a destination,
(at least as in paragraph 0056, “FIG. 2A shows an environment 200 including multiple devices, wherein each device has a respective target state… The devices 201a, 201b, 201c, and 201d, are required to reach their assigned target states 203a, 203b, 203c, and 203d”; at least as in paragraph 0058, “Learned functions 207a-207d produces initial trajectories 209a-209d, respectively”)…
wherein the late state data comprises data about a second location reached by the robot cluster after the robot cluster has moved according to the action (at least as in paragraph 0058, “The initial trajectories 209a-209d are applied to an optimization-based safety filter 211. The optimization-based safety filter 211 solves the joint-optimization problem to simultaneously determine optimal trajectories 231a-213d”; at least as in paragraph 0059, “the controller 111 controls each device based on a pre-determined number of states of their respective optimal trajectory. When the devices 201a-201d are controlled based on their respective optimal trajectory, the devices 201a-201d attain new states 215a-215d. The new states 215a-215d are fed back 217 to the learned functions 207a-207d, respectively”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Vinod's teaching of training a control system by optimizing an initial trajectory for a plurality of devices, since Vinod teaches wherein the motion planning system is easier to train, requires less computational resources, and demonstrates superior performance.
Regarding claim 13, in view of the above combination of Lu and Vinod, Lu further teaches the method of claim 12, wherein the initial state data further comprises first location information of the robot cluster at the first location and first light detection and ranging (LiDAR) sensing information (at least as in paragraph 0039, “Techniques disclosed herein can be utilized in combination with various real and/or simulated robots, such as a telepresence robot, a wheeled robot, mobile forklift robot, a robot arm, an unmanned aerial vehicle (“UAV”), and/or a humanoid robot. The robot(s) can include various sensor component(s) and state data that is utilized in techniques disclosed herein can include sensor data that is generated by those sensor component(s) (e.g., images from a camera and/or other vision data from other vision component(s)) and/or can include state data that is derived from such sensor data (e.g., object bounding box(es) derived from vision data). As a particular example, a robot can include vision component(s) such as, for example, a monographic camera (e.g., generating 2D RGB images), a stereographic camera (e.g., generating 2.5D RGB-D images), and/or a laser scanner (e.g., LIDAR generating a 2.5D depth (D) image or point cloud)”), and
wherein the late state data further comprises second location information of the robot cluster at the second location and second LiDAR sensing information (at least as in paragraph 0063, “Each transition can include state data, action, and next state data (i.e., next state data from the next state that resulted from the action)”; see also paragraph 0039 above).
Regarding claim 14, in view of the above combination of Lu and Vinod, Lu further teaches:
The method of claim 13, wherein each of the first LiDAR sensing information and the second LiDAR sensing information comprises information about whether there is an object within a certain distance from the robot cluster (at least as in paragraph 0039, “The robot(s) can include various sensor component(s) and state data that is utilized in techniques disclosed herein can include sensor data that is generated by those sensor component(s) (e.g., images from a camera and/or other vision data from other vision component(s)) and/or can include state data that is derived from such sensor data (e.g., object bounding box(es) derived from vision data)”).
Regarding claim 15, the above combination of Lu and Vinod teaches the method of claim 12, but does not explicitly teach wherein the reward function comprises a goal value comprising information about whether each of robot of the robot cluster arrives at the destination, a penalty value comprising information about whether each robot of the robot cluster is in a vicinity of an obstacle, and a leading value for obtaining a different value according to a positional relation between the destination and the obstacle, and
wherein, based on the late state data comprising information indicating that the robot cluster has arrived at the destination, the goal value is 10.
However, Vinod, in the same field of endeavor of robot control and motion planning, specifically teaches:
wherein the reward function comprises a goal value comprising information about whether each of robot of the robot cluster arrives at the destination, a penalty value comprising information about whether each robot of the robot cluster is in a vicinity of an obstacle, and a leading value for obtaining a different value according to a positional relation between the destination and the obstacle, and wherein, based on the late state data comprising information indicating that the robot cluster has arrived at the destination, the goal value is 10 (at least as in paragraph 0049, “the learned function is a neural network trained using a reinforcement learning based on a reward function. The reward function penalizes the extent of violation of the constraints. The reward function returns higher values when the initial trajectory 117 satisfies the constraints as compared to when the constraints are violated (i.e., the reward function penalizes upon violation of the constraints). In an embodiment, an algorithm such as a proximal policy optimization is used to determine an initial trajectory that maximizes the reward function”; at least as in paragraph 0068, wherein the reward is based on the distance to target and penalty on control effort; at least as in paragraph 0074, “the learned function is trained to penalize the violation of only the static constraint”; at least as in paragraph 0077, “Some embodiments of the present disclosure consider a task of ensuring that an overall system having multiple devices is probabilistically collectively safe, a collection of constraints that ensures safe motion planning for the multiple devices. With a high user-specified likelihood, it is desired that all the devices, each with dynamics (1a), i) stay within a pre-determined environment bound, ii) do not collide with N.sub.O static obstacles located at c.sub.j, 1≤j≤N.sub.O denotes nominal mean positions of the obstacles for all the N.sub.O obstacles with radii γ.sub.j, 1≤j≤N.sub.O respectively, and iii) do not collide with each other given device radii r.sub.A”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Vinod's teaching of training a control system by optimizing an initial trajectory for a plurality of devices, since Vinod teaches wherein the motion planning system is easier to train, requires less computational resources, and demonstrates superior performance.
Claim(s) 6, 16-17, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu et al. (US 20220410380 A1, hereinafter Lu) in view of Vinod et al. (US 20230367336 A1, hereinafter Vinod), and further in view of Wang et al. (US 20220317695 A1, hereinafter Wang).
Regarding claim 6, the above combination of Lu and Vinod teaches the method of claim 4 but does not explicitly teach wherein, based on the late state data comprising information indicating that the robot cluster is in the vicinity of the obstacle, the penalty value is a negative number and an absolute value of the penalty value increases as a distance between the robot cluster and the obstacle decreases.
However, Wang, in the same field of endeavor of a multi-AGV motion planning system, specifically teaches wherein, based on the late state data comprising information indicating that the robot cluster is in the vicinity of the obstacle, the penalty value is a negative number and an absolute value of the penalty value increases as a distance between the robot cluster and the obstacle decreases (at least as in paragraph 0074, “The reward function describes the rewards given to the current AGV for taking the action a.sub.t in the multi-AGV motion environment, and includes at least three description types. The first description type R.sub.goal is a reward given when the current AGV reaches or approaches the target position. The second description type R.sub.agv is a penalty given when the current AGV collides with or approaches other AGVs. The third description type R.sub.rn is a penalty given when the current AGV deviates from the road network”; at least as in paragraph 0075, “when the current AGV is stationary or far away from the target position p.sub.g, a negative discount reward value −i*a is given”; at least as in paragraph 0048, “the oblique area represents the restricted area in the environment, such as obstacles, work areas, shelves, etc., which is generally drawn based on actual conditions such as maps and safety regulations of the task scene, and the AGV is not allowed to enter the restricted area… the road network is composed of paths and represents that the permitted driving route of the AGV in the environment”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Wang’s teaching of a system utilizing a reward function which penalizes based on the distance to the obstacle, since Wang teaches wherein the motion planning system improve the performance of the multi-AGV motion planning method in dynamic environments by utilizing the merits of neural network models in computing high-dimensional state space and the characteristics of reinforcement learning online control and plans a feasible route in a dense environment preventing the AGV from entering a local deadlock state.
Regarding claim 16, the above combination of Lu and Vinod teaches the method of claim 15 but does not explicitly teach
wherein, based on the late state data comprising information indicating that the robot cluster is in the vicinity of the obstacle, the penalty value is defined as rct=-0.5*e-30*sl,
where rct denotes the penalty value and sl denotes a distance between the robot cluster and the obstacle.
However, Wang, in the same field of endeavor of a multi-AGV motion planning system, specifically teaches wherein, based on the late state data comprising information indicating that the robot cluster is in the vicinity of the obstacle, the penalty value is defined as rct=-0.5*e-30*sl,
where rct denotes the penalty value and sl denotes a distance between the robot cluster and the obstacle (at least as in paragraph 0080-0084, “The second description type R.sub.agv is set in the following way: when distances between the current AGV and other AGVs are less than a first threshold condition t.sub.agv, a maximum penalty value β is given; when distances between the current AGV and other AGVs are greater than the first threshold condition t.sub.agv and less than a second threshold condition m*t.sub.agv, the discount penalty value j*β is given; when the distances between the current AGV and other AGVs are greater than the second threshold condition m*t.sub.agv, no penalty will be given; m is a preset multiple, wherein a second discount coefficient j is set based on a distance, and a discount penalty value j*β is calculated from the second discount coefficient j and the maximum penalty value β”; at least as in paragraph 0085-0088, “The third description type R.sub.rn is set in the following way: when a distance d.sub.rn between the current AGV and a current path is not less than a third threshold condition t.sub.rn, a penalty δ is given; when the distance d.sub.rn between the current AGV and the current path is less than the third threshold condition t.sub.rn, no penalty is given. At a moment t, the complete reward function R.sub.t is: R.sub.t=R.sub.goal+R.sub.agv+R.sub.rn”)
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Wang’s teaching of a system utilizing a reward function which penalizes based on the distance to the obstacle, since Wang teaches wherein the motion planning system improve the performance of the multi-AGV motion planning method in dynamic environments by utilizing the merits of neural network models in computing high-dimensional state space and the characteristics of reinforcement learning online control and plans a feasible route in a dense environment preventing the AGV from entering a local deadlock state.
Regarding claim 17, the above combination of Lu and Vinod teaches the method of claim 15 but does not explicitly teach wherein, based on the late state data comprising information indicating that the robot cluster is closer to the destination than to the obstacle, the leading value is defined as rdt=10*dt-1-dtd0,
where rdt denotes the leading value, d0 denotes a distance between the starting point and the destination, dt-1 denotes a distance between the first location and the destination, and dt denotes a distance between the second location and the destination.
However, Wang, in the same field of endeavor of a multi-AGV motion planning system, specifically teaches: wherein, based on the late state data comprising information indicating that the robot cluster is closer to the destination than to the obstacle, the leading value is defined as rdt=10*dt-1-dtd0,
where rdt denotes the leading value, d0 denotes a distance between the starting point and the destination, dt-1 denotes a distance between the first location and the destination, and dt denotes a distance between the second location and the destination (at least as in paragraph 0147-0148, “wherein the first description type is set in the following way: when the current AGV reaches the target position p.sub.g, a positive maximum reward value a is given; a first discount coefficient i is set based on a road network distance, a positive discount reward value i*a and a negative discount reward value −i*a are calculated from the first discount coefficient i and the maximum reward value a, and when the current AGV approaches the target position p.sub.g, the positive discount reward value i*a is given; when the current AGV is stationary or far away from the target position p.sub.g, a negative discount reward value −i*a is given”; at least as in paragraph at least as in paragraph 0151-0152, “the third description type is set in the following way: when a distance between the current AGV and a current path is not less than a third threshold condition t.sub.rn, a penalty δ is given; when the distance between the current AGV and the current path is less than the third threshold condition t.sub.rn, no penalty is given”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Wang’s teaching of a system utilizing a reward function which penalizes based on the distance to the obstacle, since Wang teaches wherein the motion planning system improve the performance of the multi-AGV motion planning method in dynamic environments by utilizing the merits of neural network models in computing high-dimensional state space and the characteristics of reinforcement learning online control and plans a feasible route in a dense environment preventing the AGV from entering a local deadlock state.
Regarding claim 19, Lu teaches:
A method of controlling a robot via a computer device including at least one processor, the method comprising:
training, by the at least one processor, an agent through an actor-critic algorithm on a simulation for deep reinforcement learning (at least as in paragraph 0018, “After pre-training the actor and critic networks based on demonstration data, implementations further train the actor network and the critic network using RL and online (but potentially off-policy) episode data from robotic episodes each performed based on the actor network and/or the critic network”; at least as in paragraph 0053, “the system can perform method 200 to generate online episode data and simultaneously perform method 300 to further train the actor network and the critic network based on online episode data that is generated based on method 200”), wherein the training of the agent comprises:
inputting initial state data and late state data to an actor network and inputting a reward to a critic network in the actor-critic algorithm (at least as in paragraph 0063, “At block 216, the system stores episode data from the episode. For example, the system can store various instances of transitions during the episode and a reward for the episode. Each transition can include state data, action, and next state data (i.e., next state data from the next state that resulted from the action). In some implementations, block 216 includes sub-block 216A, in which the system populates some or all of the stored episode data in a replay buffer for use in method 300 of FIG. 3. In some of those implementations, whether the system populates the stored episode data in the replay buffer can depend on whether the reward, for the episode data, is positive—indicating a successful episode (i.e., one in which the robotic task was successfully performed)”; at least as in paragraph 0062, “If, at block 210, the system determines to perform another step in the episode, the system proceeds to block 214 and determines a reward for the episode. The reward can be determined based on a defined reward function, which will be dependent on the robotic task”);
determining, by an evaluation network of the actor network, an action of the agent (at least as in paragraph 0057, “At block 206, the system processes current state data, using the current action network and/or the current critic network, to select the next action”; at least as in paragraph 0059, “At block 208, the system executes the next action”); and
evaluating, by a value network of the critic network, a degree to which the action of the agent maximizes a preset reward (at least as in paragraph 0070, “Block 308 includes optional sub-block 308A, which can share one or more (e.g., all) aspects in common with block 108A of FIG. 1”; at least as in paragraph 0050, “as illustrated by optional sub-block 108A of block 108, the training objective can optionally utilize the Max Q-value generated at sub-block 106B”; at least as in paragraph 0048, “Instead of always using the maximum candidate action measure (e.g., Q-value) from CEM as the maximum value for training of the critic network (and optionally in the advantage function for training of the actor network) as is typical, the system can compare the actor action measure to the maximum candidate action measure—and use the greater of the two measures as the maximum value for training”),
wherein the reward comprises a value obtained by applying the initial state data and the late state data to a reward function (at least as in paragraph 0062, “If, at block 210, the system determines to perform another step in the episode, the system proceeds to block 214 and determines a reward for the episode. The reward can be determined based on a defined reward function, which will be dependent on the robotic task”),
But Lu does not explicitly teach:
However, Vinod, in the same field of endeavor of robot control and motion planning, specifically teaches:
wherein the initial state data comprises data about a first location of a robot cluster including a plurality of robots each configured to move from a starting point to a destination (at least as in paragraph 0056, “FIG. 2A shows an environment 200 including multiple devices, wherein each device has a respective target state… The devices 201a, 201b, 201c, and 201d, are required to reach their assigned target states 203a, 203b, 203c, and 203d”; at least as in paragraph 0058, “Learned functions 207a-207d produces initial trajectories 209a-209d, respectively”)…
wherein the late state data comprises data about a second location reached by the robot cluster after the robot cluster has moved according to the action (at least as in paragraph 0058, “The initial trajectories 209a-209d are applied to an optimization-based safety filter 211. The optimization-based safety filter 211 solves the joint-optimization problem to simultaneously determine optimal trajectories 231a-213d”; at least as in paragraph 0059, “the controller 111 controls each device based on a pre-determined number of states of their respective optimal trajectory. When the devices 201a-201d are controlled based on their respective optimal trajectory, the devices 201a-201d attain new states 215a-215d. The new states 215a-215d are fed back 217 to the learned functions 207a-207d, respectively”)…
wherein the reward function comprises a goal value comprising information about whether each robot of the robot cluster arrives at the destination, a penalty value comprising information about whether each robot of the robot cluster is in a vicinity of an obstacle, and a leading value for obtaining a different value according to a positional relation between the destination and the obstacle,
wherein, based on the late state data comprising information indicating that the robot cluster has arrived at the destination, the goal value is 10 (at least as in paragraph 0049, “the learned function is a neural network trained using a reinforcement learning based on a reward function. The reward function penalizes the extent of violation of the constraints. The reward function returns higher values when the initial trajectory 117 satisfies the constraints as compared to when the constraints are violated (i.e., the reward function penalizes upon violation of the constraints). In an embodiment, an algorithm such as a proximal policy optimization is used to determine an initial trajectory that maximizes the reward function”; at least as in paragraph 0068, wherein the reward is based on the distance to target and penalty on control effort; at least as in paragraph 0074, “the learned function is trained to penalize the violation of only the static constraint”; at least as in paragraph 0077, “Some embodiments of the present disclosure consider a task of ensuring that an overall system having multiple devices is probabilistically collectively safe, a collection of constraints that ensures safe motion planning for the multiple devices. With a high user-specified likelihood, it is desired that all the devices, each with dynamics (1a), i) stay within a pre-determined environment bound, ii) do not collide with N.sub.O static obstacles located at c.sub.j, 1≤j≤N.sub.O denotes nominal mean positions of the obstacles for all the N.sub.O obstacles with radii γ.sub.j, 1≤j≤N.sub.O respectively, and iii) do not collide with each other given device radii r.sub.A”).
However, Wang, in the same field of endeavor of a multi-AGV motion planning system, specifically teaches wherein, based on the late state data comprising information indicating that the robot cluster is in the vicinity of the obstacle, the penalty value is defined as r_c^t=-0.5*e^(-30*s_l ),
where rct denotes the penalty value and sl denotes a distance between the robot cluster and the obstacle (at least as in paragraph 0080-0084, “The second description type R.sub.agv is set in the following way: when distances between the current AGV and other AGVs are less than a first threshold condition t.sub.agv, a maximum penalty value β is given; when distances between the current AGV and other AGVs are greater than the first threshold condition t.sub.agv and less than a second threshold condition m*t.sub.agv, the discount penalty value j*β is given; when the distances between the current AGV and other AGVs are greater than the second threshold condition m*t.sub.agv, no penalty will be given; m is a preset multiple, wherein a second discount coefficient j is set based on a distance, and a discount penalty value j*β is calculated from the second discount coefficient j and the maximum penalty value β”; at least as in paragraph 0085-0088, “The third description type R.sub.rn is set in the following way: when a distance d.sub.rn between the current AGV and a current path is not less than a third threshold condition t.sub.rn, a penalty δ is given; when the distance d.sub.rn between the current AGV and the current path is less than the third threshold condition t.sub.rn, no penalty is given. At a moment t, the complete reward function R.sub.t is: R.sub.t=R.sub.goal+R.sub.agv+R.sub.rn”)…
wherein, based on the late state data comprising information indicating that the robot cluster is closer to the destination than to the obstacle, the leading value is defined as r_d^t=10*‖d^(t-1)-d^t ‖/‖d^0 ‖ ,
where rdt denotes the leading value, d0 denotes a distance between the starting point and the destination, dt-1 denotes a distance between the first location and the destination, and dt denotes a distance between the second location and the destination (at least as in paragraph 0147-0148, “wherein the first description type is set in the following way: when the current AGV reaches the target position p.sub.g, a positive maximum reward value a is given; a first discount coefficient i is set based on a road network distance, a positive discount reward value i*a and a negative discount reward value −i*a are calculated from the first discount coefficient i and the maximum reward value a, and when the current AGV approaches the target position p.sub.g, the positive discount reward value i*a is given; when the current AGV is stationary or far away from the target position p.sub.g, a negative discount reward value −i*a is given”; at least as in paragraph at least as in paragraph 0151-0152, “the third description type is set in the following way: when a distance between the current AGV and a current path is not less than a third threshold condition t.sub.rn, a penalty δ is given; when the distance between the current AGV and the current path is less than the third threshold condition t.sub.rn, no penalty is given”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Vinod's teaching of training a control system by optimizing an initial trajectory for a plurality of devices and Wang’s teaching of a system utilizing a reward function which penalizes based on the distance to the obstacle, since Vinod teaches wherein the motion planning system is easier to train, requires less computational resources, and demonstrates superior performance and since Wang teaches wherein the motion planning system improve the performance of the multi-AGV motion planning method in dynamic environments by utilizing the merits of neural network models in computing high-dimensional state space and the characteristics of reinforcement learning online control and plans a feasible route in a dense environment preventing the AGV from entering a local deadlock state
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu et al. (US 20220410380 A1, hereinafter Lu) in view of Vinod et al. (US 20230367336 A1, hereinafter Vinod), and further in view of Kumar et al. (US 20240181639 A1, hereinafter Kumar).
Regarding claim 18, the above combination of Lu and Vinod teaches the method of claim 15 but does not explicitly teach wherein, based on the late state data comprising information indicating that the robot cluster is closer to the obstacle than to the destination, the leading value is defined as
rdt=0.0005*lmov,
where rdt denotes the leading value and lmov denotes light detection and ranging sensing information in a direction in which the robot cluster moves.
However, Kumar, in the same field of endeavor of a multi-robot control system utilizing sensors, specifically teaches wherein, based on the late state data comprising information indicating that the robot cluster is closer to the obstacle than to the destination, the leading value is defined as
rdt=0.0005*lmov,
where rdt denotes the leading value and lmov denotes light detection and ranging sensing information in a direction in which the robot cluster moves (at least as in paragraph 0102, “the last component of the reward function favors the progress towards the goal position p.sub.G(t) and is inversely proportional to the distance between the current construction robot position and the destination [Equation (12)] where R.sub.G.sup.max is the maximum achievable reward value. So, when the construction robot reaches the goal, it receives a final reward equal to R.sub.G.sup.max”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Kumar’s teaching of a system utilizing a reward function which depends on the distance to the object, since Kumar teaches wherein the control system improves the quality of acquired sensor data thus allowing the robot to learn how to adapt dynamically to its environment.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu et al. (US 20220410380 A1, hereinafter Lu) in view of Vinod et al. (US 20230367336 A1, hereinafter Vinod) and Wang et al. (US 20220317695 A1, hereinafter Wang), and further in view of Kumar et al. (US 20240181639 A1, hereinafter Kumar).
Regarding claim 20, the above combination of Lu, Vinod, and Wang teaches the method of claim 19 but does not explicitly teach
wherein, based on the late state data comprising information indicating that the robot cluster is closer to the obstacle than to the destination, the leading value is defined as
r_d^t=0.0005*l_mov,
where rdt denotes the leading value and lmov denotes light detection and ranging sensing information in a direction in which the robot cluster moves.
However, Kumar, in the same field of endeavor of a multi-robot control system utilizing sensors, specifically teaches wherein, based on the late state data comprising information indicating that the robot cluster is closer to the obstacle than to the destination, the leading value is defined as
r_d^t=0.0005*l_mov,
where rdt denotes the leading value and lmov denotes light detection and ranging sensing information in a direction in which the robot cluster moves.
(at least as in paragraph 0102, “the last component of the reward function favors the progress towards the goal position p.sub.G(t) and is inversely proportional to the distance between the current construction robot position and the destination [Equation (12)] where R.sub.G.sup.max is the maximum achievable reward value. So, when the construction robot reaches the goal, it receives a final reward equal to R.sub.G.sup.max”).
Therefore, it would have been obvious to one of the ordinary skill in the art at the effective filing date of the instant invention to modify the teachings of Lu, to include Kumar’s teaching of a system utilizing a reward function which depends on the distance to the object, since Kumar teaches wherein the control system improves the quality of acquired sensor data thus allowing the robot to learn how to adapt dynamically to its environment.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RICARDO ICHIKAWA VISCARRA whose telephone number is (571)270-0154. The examiner can normally be reached M-F 9-12 & 2-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached on (571) 270-5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RICARDO I VISCARRA/Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights