DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
In response to the Requirement for Restriction/Election mailed out on February 19, 2026, Applicant elected Invention II directed to claims 19 and 21-39. In response to this election, Applicant additionally cancelled claims 1-18 and 20.
Response to Amendment
This correspondence is additionally in response to amendments filed on March 11, 2026 following the Requirement for Restriction/Election. In filing the response for election, Applicant cancelled claims 1-18 and 20. Claims 19, 21-23, 25-32, 34, and 36-39 remain as originally filed. Claims 24, 33, and 35 are amended. Claims 40-58 are new. Examiner addresses pending claims 19 and 21-58 as filed below.
Drawings
The drawings are objected to because Figs. 3-7 appear to be blurry as such drawings are printed in gray scale. Please note MPEP 608.02.V inclusive of 37 C.F.R. 1.84 which indicates “(a)(1) Black ink. Black and white drawings are normally required. India ink, or its equivalent that secures solid black lines, must be used for drawings… (b)(1) Black and white. Photographs, including photocopies of photographs, are not ordinarily permitted in utility and design patent applications. The Office will accept photographs in utility and design patent applications, however, if photographs are the only practicable medium for illustrating the claimed invention… The photographs must be of sufficient quality so that all details in the photographs are reproducible in the printed patent.” Additionally, MPEP 608.02(b)(I) indicates, “Drawings will be accepted by the Office of Patent Application Processing (OPAP) if the drawings are readable and reproducible for publication purposes.” As such, correction to the drawings are required.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 22-38 and 40-58 objected to because of the following informalities:
Each dependent claim (claims 22-38 and 40-58) does not include a comma to separate the preamble from the body of the claim. Such separation is common in U.S. practice in order to clearly distinguish the preamble indicating the dependency from the furthering limitation(s) in the body of the claim. For example, Claim 22 would instead read “The learning method of claim 21, further comprising…”. Examiner recommends making this correction for each of claims 22-38 and 40-58 in order to improve ease of readability of dependent claims.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 19, 39-40, 42, 45-50, and 58 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by de Heuvel et al. (“Subgoal-driven navigation in dynamic environments using attention-based deep reinforcement learning”, 2023; hereinafter “de Heuvel).
Regarding claim 19, de Heuvel discloses a learning system for a navigating robot (Section III introduces a system approach to learning a robot navigation policy.), comprising:
a navigation module (See Fig. 2b.) comprising:
a first policy configured to determine actions for moving the navigating robot and navigating from a starting location to an ending location based on input from a light detection and ranging (LIDAR) sensor of the navigating robot (The subgoal agent of Fig. 2c will be considered as the first policy configured to determine actions for moving the navigating robot from a start location to an end location based on input from the LIDAR sensor.); and
a second policy configured to, based on a representation of an environment generated by the navigating robot, determine actions for moving the navigating robot from waypoint locations between the starting location and the ending location to a plurality of subgoal locations without input from the LIDAR sensor (The motion agent of Fig. 2b will be considered as the second policy which is configured to, based on the representation of the environment used to determine subgoal locations, determine actions for moving the navigating robot from the waypoint locations between the start location and end location to the plurality of subgoal locations pre-determined by the subgoal agent. As acknowledged in Section III.D, the motion agent does not have access to the LIDAR data, only to the subgoal position.); and
a representation module (See LIDAR Attention of Fig. 2d.) configured to:
selectively learn the representation during movement via the first policy based on the representation at previous times, input from the LIDAR sensor, and actions determined by the first policy at previous times (See Section III.B which acknowledges sampling over fixed time frames of 0.2s during which lidar observations are associated to implicitly determine an obstacle shape and the obstacle’s expected velocity. This renders a real-time dynamic representation of the environment during movement via the first policy. Since the first policy uses DDPG algorithms for training, the policy determines a collision-aware goal based on the previous renderings, input from the LIDAR sensor, and the associated subgoal actions at the previous times. Previous renderings are stored in the replay buffer which is part of the DDPG algorithm.); and
provide the representation to the second policy (The representation is provided to the motion agent in the form of a determined subgoal.).
Regarding claim 39, de Heuvel discloses a learning method for a navigating robot (Section III introduces a method for learning a robot navigation policy.), comprising:
by a first policy, determining actions for moving the navigating robot and navigating from a starting location to an ending location based on input from a light detection and ranging (LIDAR) sensor of the navigating robot (The subgoal agent of Fig. 2c is the first policy which determines actions for moving the navigating robot from a start location to an end location based on input from the LIDAR sensor.);
by a second policy, based on a representation of an environment generated by the navigating robot, determining actions for moving the navigating robot from waypoint locations between the starting location and the ending location to a plurality of subgoal locations without input from the LIDAR sensor (The motion agent of Fig. 2b is the second policy which, based on the representation of the environment used to determine subgoal locations, determines actions for moving the navigating robot from the waypoint locations between the start location and end location to the plurality of subgoal locations pre-determined by the subgoal agent. As acknowledged in Section III.D, the motion agent does not have access to the LIDAR data, only to the subgoal position which is determined.);
selectively learning the representation during movement via the first policy based on the representation at previous times, input from the LIDAR sensor, and actions determined by the first policy at previous times (See Section III.B which acknowledges sampling over fixed time frames of 0.2s during which lidar observations are associated to implicitly determine an obstacle shape and the obstacle’s expected velocity. This renders a real-time dynamic representation of the environment during movement via the first policy. Since the first policy uses DDPG algorithms for training, the policy determines a collision-aware goal based on the previous renderings, input from the LIDAR sensor, and the associated subgoal actions at the previous times. Previous renderings are stored in the replay buffer which is part of the DDPG algorithm.); and
providing the representation to the second policy (The representation is provided to the motion agent in the form of a determined subgoal.).
Regarding claim 40, de Heuvel discloses the learning method of claim 39 further comprising selectively learning the representation during the movement via the first policy using a neural network (As described in Section III.D, the subgoal agent which learns the representation during the movement via the first policy uses DDPG algorithms, a type of neural network.).
Regarding claim 42, de Heuvel discloses the learning system of claim 19 wherein the representation module is configured to selectively learn the representation during the movement via the first policy using a neural network (As described in Section III.D, the subgoal agent which learns the representation during the movement via the first policy uses DDPG algorithms, a type of neural network.).
Regarding claim 45, de Heuvel discloses the learning system of claim 42 wherein the neural network includes one or more self-attention mechanisms (The lidar attention weights the split sectors of lidar data and thus is considered as a self-attention mechanism.).
Regarding claim 46, de Heuvel discloses the learning system of claim 42 wherein the first policy is configured to determine an action at a time based on the representation at the time and a goal vector at that time (As identified by Section III.B, the path module takes into account the predicted subgoal position as input for the critic network. As such, when determining the action output as a combination of the path module and lidar module, both the predicted subgoal position at the time, i.e., goal vector, and the lidar-produced representation at the time are considered.).
Regarding claim 47, de Heuvel discloses the learning system of claim 46 wherein the goal vector is a Euclidean goal vector (All distances measured by the subgoal agent are best understood to be linear distances in cartesian space, and thus by determining a predicted subgoal from the current position, this goal vector would be a Euclidean goal vector. Additionally, the motion agent converts the subgoal itself into a cartesian space coordinate.).
Regarding claim 48, de Heuvel discloses the learning system of claim 42 wherein the plurality of subgoal locations include at least two subgoal locations from each of the waypoint locations (“We adapt this concept and replan the robot’s global path every three subgoal predictions” (Section III.A). Thus, the plurality of subgoals includes at least three subgoals from each of the waypoint locations identified with respect the planned A* path before replanning said A* path and sampling new waypoint locations.).
Regarding claim 49, de Heuvel discloses the learning system of claim 48 wherein each of the at least two subgoal locations of a waypoint location are within a predetermined distance range of that waypoint location (“Furthermore, we clip the lidar range at 4 m, as information about obstacles within this range around the robot is sufficient for local obstacle avoidance” (Section III.B.1). Thus, the subgoals are within 4m of the waypoint location as this is the maximum LIDAR sensing distance for the method.).
Regarding claim 50, de Heuvel discloses the learning system of claim 49 wherein the predetermined distance range is approximately 3 meters to approximately 5 meters (As identified above, the predetermined distance range is 4 meters and thus is approximately 3 meters to approximately 5 meters.).
Regarding claim 58, de Heuvel discloses the learning system of claim 19 further comprising a training module configured to train the first policy using one of reinforcement learning and imitation learning (As noted in Section III.B, the first policy is trained using DDGP which is a reinforcement learning algorithm.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 21-22, 25-30, and 38 are rejected under 35 U.S.C. 103 as being unpatentable over de Heuvel in view of Ai et al. (“R-VPCG: RGB image feature fusion-based virtual point cloud generation for 3D car detection”, 2023; hereinafter “Ai”).
Regarding claim 21, de Heuvel teaches a learning method for a navigating robot (Section III introduces a method for learning a robot navigation policy.), comprising:
by a first policy, determining actions for moving the navigating robot and navigating from a starting location to an ending location based on input from a light detection and ranging (LIDAR) sensor of the navigating robot (The subgoal agent of Fig. 2c is the first policy which determines actions for moving the navigating robot from a start location to an end location based on input from the LIDAR sensor.);
by a second policy, based on a representation of an environment generated by the navigating robot, determining actions for moving the navigating robot from waypoint locations between the starting location and the ending location to a plurality of subgoal locations without input from the LIDAR sensor (The motion agent of Fig. 2b is the second policy which, based on the representation of the environment used to determine subgoal locations, determines actions for moving the navigating robot from the waypoint locations between the start location and end location to the plurality of subgoal locations pre-determined by the subgoal agent. As acknowledged in Section III.D, the motion agent does not have access to the LIDAR data, only to the subgoal position which is determined.);
selectively learning the representation during movement via the first policy based on the representation at previous times, input from the LIDAR sensor, and actions determined by the first policy at previous times (See Section III.B which acknowledges sampling over fixed time frames of 0.2s during which lidar observations are associated to implicitly determine an obstacle shape and the obstacle’s expected velocity. This renders a real-time dynamic representation of the environment during movement via the first policy. Since the first policy uses DDPG algorithms for training, the policy determines a collision-aware goal based on the previous renderings, input from the LIDAR sensor, and the associated subgoal actions at the previous times. Previous renderings are stored in the replay buffer which is part of the DDPG algorithm.); and
providing the representation to the second policy (The representation is provided to the motion agent in the form of a determined subgoal.).
However, de Heuvel does not explicitly teach the sensor input as …images from a camera…
Ai, pertinent to the problem at hand, teaches “a novel fusion-based 3D object detection method that contains both semantic features of RGB and 3D coordinate features of point clouds” (see Section 1). Also in Section 1, Ai further describes, “As shown in Fig. 2, the proposed method build image segmentation and point cloud projection to strengthen the feature connections between RGB images and point clouds”.
Therefore, provided the above descriptions for using RGB images to further enhance features in 3D object detection, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the LIDAR sensor input of de Heuvel to additionally include input from an RGB camera as taught by Ai. One of ordinary skill in the art would have been motivated to make such a modification because the fusion methods described by Ai enhances raw point cloud data by increasing the number of non-empty voxels and in-voxel point clouds, thereby distinguishing features of relevant obstacles (see Ai, Section 1).
Regarding claim 22, de Heuvel as modified by Ai teaches the learning method of claim 21,
with de Heuvel further teaching …selectively learning the representation during the movement via the first policy using a neural network (As described in Section III.D, the subgoal agent which learns the representation during the movement via the first policy uses DDPG algorithms, a type of neural network.).
Regarding claim 25, de Heuvel as modified by Ai teaches the learning method of claim 22,
with de Heuvel further teaching wherein the neural network includes one or more self-attention mechanisms (The lidar attention weights the split sectors of lidar data and thus is considered as a self-attention mechanism.).
Regarding claim 26, de Heuvel as modified by Ai teaches the learning method of claim 21,
with de Heuvel further teaching wherein the determining by the first policy includes, by the first policy, determining an action at a time based on the representation at the time and a goal vector at that time (As identified by Section III.B, the path module takes into account the predicted subgoal position as input for the critic network. As such, when determining the action output as a combination of the path module and lidar module, both the predicted subgoal position at the time, i.e., goal vector, and the lidar-produced representation at the time are considered.).
Regarding claim 27, de Heuvel as modified by Ai teaches the learning method of claim 26,
with de Heuvel further teaching …wherein the goal vector is a Euclidean goal vector (All distances measured by the subgoal agent are best understood to be linear distances in cartesian space, and thus by determining a predicted subgoal from the current position, this goal vector would be a Euclidean goal vector. Additionally, the motion agent converts the subgoal itself into a cartesian space coordinate.).
Regarding claim 28, de Heuvel as modified by Ai teaches the learning method of claim 21,
with de Heuvel further teaching …wherein the plurality of subgoal locations include at least two subgoal locations from each of the waypoint locations (“We adapt this concept and replan the robot’s global path every three subgoal predictions” (Section III.A). Thus, the plurality of subgoals includes at least three subgoals from each of the waypoint locations identified with respect the planned A* path before replanning said A* path and sampling new waypoint locations.).
Regarding claim 29, de Heuvel as modified by Ai teaches the learning method of claim 28,
with de Heuvel further teaching …wherein each of the at least two subgoal locations of a waypoint location are within a predetermined distance range of that waypoint location (“Furthermore, we clip the lidar range at 4 m, as information about obstacles within this range around the robot is sufficient for local obstacle avoidance” (Section III.B.1). Thus, the subgoals are within 4m of the waypoint location as this is the maximum LIDAR sensing distance for the method.).
Regarding claim 30, de Heuvel as modified by Ai teaches the learning method of claim 29
with de Heuvel further teaching wherein the predetermined distance range is approximately 3 meters to approximately 5 meters (As identified above, the predetermined distance range is 4 meters and thus is approximately 3 meters to approximately 5 meters.).
Regarding claim 38, de Heuvel as modified by Ai teaches the learning method of claim 21…
with de Heuvel further teaching …training the first policy using one of reinforcement learning and imitation learning (As noted in Section III.B, the first policy is trained using DDGP which is a reinforcement learning algorithm.).
Claims 23-24 are rejected under 35 U.S.C. 103 as being unpatentable over de Heuvel in view of Ai and further in view of Gao et al. (“Improved Deep Deterministic Policy Gradient for Dynamic Obstacle Avoidance of Mobile Robot”, 2023; hereinafter “Gao”).
Regarding claim 23, de Heuvel as modified by Ai teaches the learning method of claim 22…
However, de Heuvel as modified does not explicitly teach …wherein the neural network includes a recurrent neural network.
Gao, pertinent to the problem at hand, teaches …wherein the neural network includes a recurrent neural network (“One problem of using the actor and critic framework in the DDPG algorithm is that the fully connected neural network assumes that the samples are fixed-sized input. Thus, it requires a fixed-length representation of the input. To overcome this limitation, LSTM network-based encoder is incorporated with the DDPG network, which is capable of generating an arbitrary length sequence to fixed-length representation” (Section III.A). LSTM is a type of RNN.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the DDPG algorithm of de Heuvel to include the LSTM as taught by Gao with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because the inclusion of LSTM allows a variable number of obstacles to be encoded into fixed-length representations, thus generating a continuous state-space for mobile robots and obstacles and additionally a continuous reward function and action space (see Gao, Abstract).
Regarding claim 24, de Heuvel as modified by Ai and Gao teaches the learning method of claim 23,
with Gao further teaching …wherein the recurrent neural network includes one of gated recurrent unit (GRU) memory and long-short term memory (LSTM) (As identified above, Gao uses long-short term memory.).
Claims 31-32 and 36-37 are rejected under 35 U.S.C. 103 as being unpatentable over de Heuvel in view of Ai and further in view of Jia et al. (“GRU-Attention based TD3 Network for Mobile Robot Navigation”, 2022; hereinafter “Jia”).
Regarding claim 31, de Heuvel as modified by Ai teaches the learning method of claim 21,
with de Heuvel further teaching …wherein the determining by the second policy includes, by the second policy, determining the actions for moving the navigating robot from waypoint locations between the starting location and the ending location to the plurality of subgoal locations without any sensor data based on a (As identified in Section III.D, the motion agent, i.e., second policy, is trained using a TD3 neural network. Sensor includes “without any sensor data” rather than “without any images from the camera” due to the modification as introduced by Ai in the rejection of claim 21.).
However, de Heuvel does not explicitly identify the TD3 neural network as a recurrent neural network.
Jia, pertinent to the problem at hand, teaches a GRU-Attention based TD3 Network for Mobile Robot Navigation, and as such teaches the TD3 neural network as a recurrent neural network.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the TD3 network of de Heuvel to include GRU memory as taught by Jia with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because including GRU memory will allow effective data processing with sequential characteristics, therefore mining temporal information and semantic information with the data which is processed (see Jia, Section 2.3).
Regarding claim 32, de Heuvel as modified by Ai and Jia teaches the learning method of claim 31…
with Jia further teaching …by the second policy, setting recurrent GRU memory using a GRU update function based on the representation (GRU memory is updated with a history of the last n states, i.e., update function based on the representation.), a previous instance of the recurrent GRU memory (The previous instance of the GRU memory is included in the last n states.), and previous actions from the second policy (Attention-GRU TD3 uses an actor-critic network structure and thus includes a replay buffer with previous actions from the second policy.).
Regarding claim 36, de Heuvel as modified by Ai and Jia teaches the learning method of claim 31…
However, de Heuvel as modified does not explicitly teach …by the second policy, initializing the representation upon reaching one of the waypoint locations.
Examiner ascertains that de Heuvel does however implicitly teach this limitation. In describing the algorithm for determining the subgoal locations from the waypoint, de Heuvel indicates that the subgoal agent is encouraged to navigate back to the A* path after avoiding moving obstacles based on a reward which penalizes deviations from said global path (See Section III.B.2). Thus, it would be implied that for sparse environments with few, if any, dynamic obstacles, the subgoal would trend towards the waypoints along the global A* path. At some point, this would render the waypoint as the subgoal location. Upon reaching a subgoal location, the robot is stopped until the next subgoal prediction is received (see Section III.B). Thus, while stopped at the subgoal which is the waypoint itself, the representation for generating the new subgoal would be initialized, as the unknown obstacles in the scene are not stored when considering replanning efforts. Therefore, the teachings of de Heuvel are implied and such an initialization would be obvious to one of ordinary skill in the art.
Regarding claim 37, de Heuvel as modified by Ai and Jia teaches the learning method of claim 31…
with de Heuvel further teaching …by the second policy, receiving the representation at each time step during navigation from one of the waypoint locations to one of the subgoal locations (LIDAR, which renders the representation, is sampled every .2 seconds. The timing, as indicated in Section III.B, is independent of whether the subgoal is reached. Thus, the motion agent receives the representation at each time step during navigation to the subgoal location.).
Claims 33-35 are rejected under 35 U.S.C. 103 as being unpatentable over de Heuvel in view of Ai and further in view of Ma et al. (“Goal-conditioned Behavioral Cloning with Prioritized Sampling”, 2021; hereinafter “Ma”).
Regarding claim 33, de Heuvel as modified by Ai teaches the learning method of claim 22…
However, de Heuvel as modified does not teach …jointly training the second policy and the neural network using behavior cloning.
Ma, pertinent to the problem at hand, teaches a method for goal-conditioned behavior cloning. This method requires that action choosing not only be dependent on states, but also on goals (Section II.C). Further, Section II.C of Ma acknowledges “This procedure does not need additional interactions with environment, which makes it lighter than other on-line RL algorithm. It is also convenient to combine this loss function with other policy update algorithms such as DDPG…”
Therefore, provided that de Heuvel teaches the first policy neural network as a DDPG neural network (de Heuvel, Section III.D) which results in a goal, i.e., the subgoal, and that the second policy produces the state-action pair for reaching the goal, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have combined the methods of de Heuvel to include a goal-conditioned behavior cloning method as taught by Ma with a reasonable expectation of success. The introduction of goal-conditioned behavior cloning to jointly train the second policy (state-action pair generator) and the neural network (first policy directed to goal setting) would be a mere combination of prior art elements according to known methods to yield predictable results (see MPEP 2143.I(A)). One of ordinary skill in the art would have additionally been motivated to make such a modification because the goal-conditioned behavior cloning may be used as a pre-training step before an agent interacts in real-world environments, and also proves to contribute higher scores in more difficult environments (see Ma, Section I).
Regarding claim 34, de Heuvel as modified by Ai and Ma teaches the learning method of claim 33…
with Ma further teaching …jointly training the second policy and a neural network of the first policy based on minimizing an error between (a) actions predicted by the first policy during movement and (b) ground truth actions for moving (See Equation (1) which shows the loss function for goal-conditioned behavior cloning which is directed to minimize an error between the goal, i.e., actions predicted by the first policy during movement, and an action a which is representative of the ground truth action for moving the robot.).
Regarding claim 35, de Heuvel as modified by Ai and Ma teaches the learning method of claim 34…
with Ma further teaching …jointly training the second policy and the neural network using a cross entropy loss (“The overall training procedure is straightforward, first sampling a batch of transitions according to their priorities, then taking current state and goal as input, and trying to predict the action taken by the expert transition at that time step. The overall process is shown in Fig. 1. We use cross entropy here as loss function, but other alternatives like KL divergence will also work” (Section III.B). Thus, the goal-conditioned behavior cloning method uses a cross entropy loss function.).
Claims 41 and 43-44 are rejected under 35 U.S.C. 103 as being unpatentable over de Heuvel in view of Gao et al. (“Improved Deep Deterministic Policy Gradient for Dynamic Obstacle Avoidance of Mobile Robot”, 2023; hereinafter “Gao”).
Regarding claim 41, de Heuvel teaches the learning method of claim 40…
However, de Heuvel does not explicitly teach …wherein the neural network includes a recurrent neural network.
Gao, pertinent to the problem at hand, teaches …wherein the neural network includes a recurrent neural network (“One problem of using the actor and critic framework in the DDPG algorithm is that the fully connected neural network assumes that the samples are fixed-sized input. Thus, it requires a fixed-length representation of the input. To overcome this limitation, LSTM network-based encoder is incorporated with the DDPG network, which is capable of generating an arbitrary length sequence to fixed-length representation” (Section III.A). LSTM is a type of RNN.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the DDPG algorithm of de Heuvel to include the LSTM as taught by Gao with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because the inclusion of LSTM allows a variable number of obstacles to be encoded into fixed-length representations, thus generating a continuous state-space for mobile robots and obstacles and additionally a continuous reward function and action space (see Gao, Abstract).
Regarding claim 43, de Heuvel teaches the learning system of claim 42…
However, de Heuvel does not explicitly teach …wherein the neural network includes a recurrent neural network.
Gao, pertinent to the problem at hand, teaches …wherein the neural network includes a recurrent neural network (“One problem of using the actor and critic framework in the DDPG algorithm is that the fully connected neural network assumes that the samples are fixed-sized input. Thus, it requires a fixed-length representation of the input. To overcome this limitation, LSTM network-based encoder is incorporated with the DDPG network, which is capable of generating an arbitrary length sequence to fixed-length representation” (Section III.A). LSTM is a type of RNN.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the DDPG algorithm of de Heuvel to include the LSTM as taught by Gao with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because the inclusion of LSTM allows a variable number of obstacles to be encoded into fixed-length representations, thus generating a continuous state-space for mobile robots and obstacles and additionally a continuous reward function and action space (see Gao, Abstract).
Regarding claim 44, de Heuvel as modified by Gao teaches the learning system of claim 43
with Gao further teaching wherein the recurrent neural network includes one of gated recurrent unit (GRU) memory and long-short term memory (LSTM) (As identified above, Gao uses long-short term memory.).
Claims 51-52 and 56-57 are rejected under 35 U.S.C. 103 as being unpatentable over de Heuvel in view of Jia et al. (“GRU-Attention based TD3 Network for Mobile Robot Navigation”, 2022; hereinafter “Jia”).
Regarding claim 51, de Heuvel teaches the learning system of claim 42 wherein the second policy is configured to determine the actions for moving the navigating robot from waypoint locations between the starting location and the ending location to the plurality of subgoal locations without any input from the LIDAR sensor and using a (As identified in Section III.D, the motion agent, i.e., second policy, is trained using a TD3 neural network. Sensor includes “without any sensor data” rather than “without any images from the camera” due to the modification as introduced by Ai in the rejection of claim 21.).
However, de Heuvel does not explicitly identify the TD3 neural network as a recurrent neural network.
Jia, pertinent to the problem at hand, teaches a GRU-Attention based TD3 Network for Mobile Robot Navigation, and as such teaches the TD3 neural network as a recurrent neural network.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the TD3 network of de Heuvel to include GRU memory as taught by Jia with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because including GRU memory will allow effective data processing with sequential characteristics, therefore mining temporal information and semantic information with the data which is processed (see Jia, Section 2.3).
Regarding claim 52, de Heuvel as modified by Jia teaches the learning system of claim 51
with Jia further teaching wherein the second policy is further configured to set recurrent gated recurrent unit (GRU) memory using a GRU update function based on the representation (GRU memory is updated with a history of the last n states, i.e., update function based on the representation.), a previous instance of the recurrent GRU memory (The previous instance of the GRU memory is included in the last n states.), and previous actions from the second policy (Attention-GRU TD3 uses an actor-critic network structure and thus includes a replay buffer with previous actions from the second policy.).
Regarding claim 56, de Heuvel as modified by Jia teaches the learning system of claim 51…
However, de Heuvel as modified does not explicitly teach …wherein the second policy is further configured to initialize the representation upon reaching one of the waypoint locations.
Examiner ascertains that de Heuvel does however implicitly teach this limitation. In describing the algorithm for determining the subgoal locations from the waypoint, de Heuvel indicates that the subgoal agent is encouraged to navigate back to the A* path after avoiding moving obstacles based on a reward which penalizes deviations from said global path (See Section III.B.2). Thus, it would be implied that for sparse environments with few, if any, dynamic obstacles, the subgoal would trend towards the waypoints along the global A* path. At some point, this would render the waypoint as the subgoal location. Upon reaching a subgoal location, the robot is stopped until the next subgoal prediction is received (see Section III.B). Thus, while stopped at the subgoal which is the waypoint itself, the representation for generating the new subgoal would be initialized, as the unknown obstacles in the scene are not stored when considering replanning efforts. Therefore, the teachings of de Heuvel are implied and such an initialization would be obvious to one of ordinary skill in the art.
Regarding claim 57, de Heuvel as modified by Jia teaches the learning system of claim 51
with de Heuvel further teaching wherein the second policy is further configured to receive the representation at each time step during navigation from one of the waypoint locations to one of the subgoal locations (LIDAR, which renders the representation, is sampled every .2 seconds. The timing, as indicated in Section III.B, is independent of whether the subgoal is reached. Thus, the motion agent receives the representation at each time step during navigation to the subgoal location.).
Claims 53-55 are rejected under 35 U.S.C. 103 as being unpatentable over de Heuvel in view of Ma et al. (“Goal-conditioned Behavioral Cloning with Prioritized Sampling”, 2021; hereinafter “Ma”).
Regarding claim 53, de Heuvel teaches the learning system of claim 42…
However, de Heuvel does not explicitly teach …a training module configured to jointly train the second policy and the neural network using behavior cloning.
Ma, pertinent to the problem at hand, teaches a method for goal-conditioned behavior cloning. This method requires that action choosing not only be dependent on states, but also on goals (Section II.C). Further, Section II.C of Ma acknowledges “This procedure does not need additional interactions with environment, which makes it lighter than other on-line RL algorithm. It is also convenient to combine this loss function with other policy update algorithms such as DDPG…”
Therefore, provided that de Heuvel teaches the first policy neural network as a DDPG neural network (de Heuvel, Section III.D) which results in a goal, i.e., the subgoal, and that the second policy produces the state-action pair for reaching the goal, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have combined the methods of de Heuvel to include a goal-conditioned behavior cloning method as taught by Ma with a reasonable expectation of success. The introduction of goal-conditioned behavior cloning to jointly train the second policy (state-action pair generator) and the neural network (first policy directed to goal setting) would be a mere combination of prior art elements according to known methods to yield predictable results (see MPEP 2143.I(A)). One of ordinary skill in the art would have additionally been motivated to make such a modification because the goal-conditioned behavior cloning may be used as a pre-training step before an agent interacts in real-world environments, and also proves to contribute higher scores in more difficult environments (see Ma, Section I).
Regarding claim 54, de Heuvel as modified by Ma teaches the learning system of claim 53
with Ma further teaching wherein the training module is configured to jointly train the second policy and a neural network of the first policy based on minimizing an error between (a) actions predicted by the first policy during movement and (b) ground truth actions for moving (See Equation (1) which shows the loss function for goal-conditioned behavior cloning which is directed to minimize an error between the goal, i.e., actions predicted by the first policy during movement, and an action a which is representative of the ground truth action for moving the robot.).
Regarding claim 55, de Heuvel as modified by Ma teaches the learning system of claim 54
with Ma further teaching wherein the training module is configured to jointly train the second policy and the neural network using a cross entropy loss (“The overall training procedure is straightforward, first sampling a batch of transitions according to their priorities, then taking current state and goal as input, and trying to predict the action taken by the expert transition at that time step. The overall process is shown in Fig. 1. We use cross entropy here as loss function, but other alternatives like KL divergence will also work” (Section III.B). Thus, the goal-conditioned behavior cloning method uses a cross entropy loss function.).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Brito et al. (“Where to go Next: Learning a Subgoal Recommendation Policy for Navigation in Dynamic Environments”, 2021) teaches another relevant subgoal generation method.
Tampuu et al. (“LiDAR-as-Camera for End-to-End Driving”, 2023) teaches LiDAR and camera relationships which are slightly different from the teachings of Ai but remain relevant.
US 2019/0384302 A1 teaches a relevant navigation method for occluded regions.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY L MOLNAR whose telephone number is (571)272-2276. The examiner can normally be reached 9 A.M. to 4 P.M. EST Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jonathan (Wade) Miles can be reached at (571) 270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.L.M./Examiner, Art Unit 3656
/WADE MILES/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights