DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 9 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 9 appears to be an independent claim because it alters the preamble by reciting “A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform”. Yet the claim also cites another claim by reciting “the method of claim 1,” which implies dependency or else lacks antecedent basis if claim 9 is independent. The conflicting structure and referencing another claim make the claims unclear regarding their dependency and thus indefinite.
The examiner recommends cutting and pasting parts of claim 1 into claim 9 thereby making claim 9 clearly independent. For examination purposes, claim 9 will be interpreted as follows:
A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform a method comprising:
generating a plurality of trajectories for a driving situation of the moving object based on either one or both of a speed and a steering of the moving object;
selecting candidate trajectories based on a presence of an obstacle among the plurality of trajectories;
outputting hyperparameters related to driving of the moving object by inputting data related to the driving situation to a machine learning model;
selecting a target trajectory from the candidate trajectories based on the hyperparameters; and
controlling the steering and the speed such that the moving object moves along the target trajectory,
wherein the hyperparameters comprise a first hyperparameter for the speed of the moving object, a second hyperparameter for a degree to which the moving object is able to avoid an obstacle, and a third hyperparameter for a global path to a destination, and
wherein the hyperparameters vary while the moving object travels.
Claim 20 is indefinite. The claim recites in part: “selecting candidate trajectories based on a presence of an obstacle among a plurality of trajectories generated for a driving situation of a moving object based on either one or both of a speed and a steering of the moving object”. It is not definite what the phrase “based on either one or both of a speed and a steering of the moving object” modifies. Does it modify the “selecting candidate trajectories”? Or does it modify “trajectories generated for a driving situation of a moving object”?
Claim 1 can be used as a guide. The claim recites in part: “generating a plurality of trajectories for a driving situation of the moving object based on either one or both of a speed and a steering of the moving object;” and “selecting candidate trajectories based on a presence of an obstacle among the plurality of trajectories”.
Based on claim 1 claim 20 will be interpreted as follows:
selecting candidate trajectories based on a presence of an obstacle among a plurality of trajectories, wherein the plurality of trajectories were generated for a driving situation of a moving object based on either one or both of a speed and a steering of the moving object.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 5, 6, 9, 12-14, 16, 17, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Caldwell et al. (US2021/0046924).
Regarding claim 1, Caldwell discloses:
A processor-implemented method comprising (see the method of Fig. 9 performed on the processor of Fig. 7, item 716):
generating a plurality of trajectories for a driving situation of the moving object based on either one or both of a speed and a steering of the moving object (see Fig. 9, step 902);
selecting candidate trajectories based on a presence of an obstacle among the plurality of trajectories (see steps 904-910, in these steps nearby obstacles are considered. With a YES out of 908 the system will eliminate trajectories that are not safe. Then the system will consider the remaining trajectories as candidate trajectories. See Fig. 1 for various trajectories of the host vehicle 102.);
outputting hyperparameters related to driving of the moving object by inputting data related to the driving situation to a machine learning model (What are “hyperparameters” in the present disclosure? Paragraph 0079 teaches that there can be “a first hyperparameter for a speed” and “a second hyperparameter for a degree to which a moving object may avoid an obstacle”. These can be scaled or graded and added to come up with an overall score S. With that in mind, see Caldwell, Fig. 2 in which various possible trajectories, first action 204 through fourth action 210, are graded by costs in various categories. This is just one example. Each of the trajectories shown in Fig. 1 could be adjusted as shown in Fig. 2 to determine an overall cost, as discussed in paragraph 0047.);
selecting a target trajectory from the candidate trajectories based on the hyperparameters (see Fig. 1, step 912 and 914. See also Fig. 8 for a YES out of 812.); and
controlling the steering and the speed such that the moving object moves along the target trajectory (see Fig. 8, step 814),
wherein the hyperparameters comprise
a first hyperparameter for the speed of the moving object (see Fig. 2 for various costs. See paragraph 0030 for the comfort cost being related to “speed”. The comfort cost includes abrupt accelerations as well. The comfort cost is a way to grade sub-actions. For example, as described in paragraph 0047 and Fig. 2, there can be “speed variations” of a trajectory, and the differences can show up in the comfort cost of the sub-action.),
a second hyperparameter for a degree to which the moving object is able to avoid an obstacle (see Fig. 2, for grading trajectories based on “safety”. See Caldwell Fig. 2 and paragraphs 0083-0090. In the top illustration related to first action 204 the host vehicle 202 may collide with vehicle 224. Therefore, the “safety” cost is high at 16. In fact, it is higher than the others. That trajectory is obviously not that ideal overall.), and
a third hyperparameter for a global path to a destination (see Fig. 2 and paragraph 0013 in which “progress (e.g., movement toward destination)” is a factor. See paragraph 0124 for route planning. See paragraph 0072 for ranking the various costs, including the progress cost.), and
wherein the hyperparameters vary while the moving object travels (see paragraph 0022 for performing the analysis at 10 Hz.).
Regarding claim 2, Caldwell discloses the method of claim 1.
Caldwell further discloses:
The method of claim 1, wherein
the selecting of the candidate trajectories comprises selecting the candidate trajectories using remaining trajectories excluding trajectories in which the presence of an obstacle is determined within a threshold radius around the moving object, from the plurality of trajectories (see steps 904-910, in these steps nearby obstacles are considered. With a YES out of 908 the system will eliminate trajectories that are not safe, which according to paragraph 0013, are trajectories that including “avoiding a collision between the vehicle and the object”. See paragraph 0025 for the teaching that a “collision may be based on threshold values associated with the distance [between the host vehicle and obstacle],” which can be “within 5 feet…etc.” A collision can be determined based on “threshold distance”. Then the system will consider the remaining trajectories as candidate trajectories. See Fig. 1 for various trajectories of the host vehicle 102.).
Regarding claim 3, Caldwell discloses the method of claim 1.
Caldwell further discloses:
The method of claim 1, wherein
the selecting of the target trajectory from the candidate trajectories comprises determining a score for each of the candidate trajectories using the hyperparameters (see Fig. 2 for the “Total” cost for four different trajectories.), and
selecting a candidate trajectory having a highest score as the target trajectory (See paragraph 0077 for the trajectory selected being the one with the “lowest cost” which is synonymous with the “high score.” See also paragraph 0090.).
Regarding claim 5, Caldwell discloses the method of claim 1.
Caldwell further discloses:
The method of claim 1, wherein,
in response to the driving situation being a driving situation in which an obstacle is not present within a front threshold distance of the moving object and a curvature of a driving lane, on which the moving object travels, being within a threshold curvature, the first hyperparameter is determined to be greatest among the hyperparameters (in present claim 1, the first hyperparameter is “for the speed of the moving object”. In the present published specification (which has the same numbering as the present filed specification), paragraph 0097 states that when a “curvature of the driving lane…is less than or equal to a threshold curvature,” which can reasonably be when the route ahead is straight,” and when there are no obstacles “within a threshold distance of the moving object,” which can reasonably be when there are no vehicles ahead of the host vehicle, “the driving situation of the moving object 500 may be a situation in which the moving object 500 is able to move at a high speed.” Therefore, “the machine learning model may output the first hyperparameter as a higher value compared to the other parameters, so that the moving object 500 is able to move fast.” In contrast, as made clear in paragraph 0098, if the road is straight but there is an obstacle “within a front threshold distance” then the “machine learning model may output the second hyperparameter as a higher value than other parameters, so that the moving object 500 may avoid the obstacle 520.” Thus, the host vehicle will be controlled to avoid the obstacle.
What exactly “hyperparameters” are in the disclosure is somewhat opaque but there is enough description. According to paragraph 0079, there can be “a first hyperparameter for a speed” and “a second hyperparameter for a degree to which a moving object may avoid an obstacle”. These can be summed to form a score S, as in equation 1. Note that claim 1 recites “selecting a target trajectory from the candidate trajectories based on the hyperparameters”. Paragraph 0082 recites something similar. Therefore, if hyperparameters are not the control commands themselves, they are some score, grade, or category or combination of those, such as a score for speed, by which the system selects the target or optimal trajectory. In one broad reasonable interpretation, claims 5-7, which are similar in formulation, essentially say that depending on the driving situation, one of these scores, grades, or categories (i.e., one of the hyperparameters) is going to swamp the rest in terms of determining which of the candidate trajectories is selected as the target trajectory.
In the present claim, if there is a straight road and no obstacles ahead, the host vehicle can move rapidly. In contrast, as claim 7 states, even if there is a straight road, if there is an obstacle within a front threshold of the host vehicle, obstacle avoidance maneuvers, reasonably including braking or remaining stopped, will swamp the considerations of what the best trajectory to choose is. This seems somewhat intuitive. If there are no obstacles, the host vehicle can drive freely, if there are obstacles the host vehicle will have to avoid them, which may entail losing a bit of time.
With all that in mind, see Caldwell, Caldwell Fig. 2 and paragraphs 0083-0090. In the top illustration related to first action 204 the host vehicle 202 may collide with vehicle 224. Therefore, the “safety” cost is high at 16. In fact, it is higher than the others. That trajectory is obviously not that ideal overall. This satisfies the limitations of claim 6 in which the parameter related to collision avoidance or safety swamps the parameters used to select the target trajectory, with the threshold distance being taught in paragraph 0056 (“threshold values associated with the distance”). The straight road is seen in Fig. 2. Note also that, according to Caldwell, paragraph 0054, turning onto a “substantially perpendicular” road is taught. The paragraph also teaches that an object can be a “threshold distance” from another vehicle. Depending on the route the host vehicle is going to take, that other vehicle may or may not be relevant.
What if there was no vehicle 224 present but everything else the same? Then the safety cost would obviously be 0 or something else very low. In that case, acceleration could be commanded. That case satisfies the limitations of present claim 5, with the threshold distance being taught in paragraph 0056 (“threshold values associated with the distance”).).
Regarding claim 6, Caldwell discloses the method of claim 1.
Caldwell further discloses:
The method of claim 1, wherein,
in response to the driving situation being a driving situation in which an obstacle is present within a front threshold distance of the moving object and a curvature of a driving lane, on which the moving object travels, exceeding a threshold curvature, the second hyperparameter is determined to be greatest among the hyperparameters (in the present claim 1, the second hyperparameter is “for a degree to which the moving object is able to avoid an obstacle”. The present claim can therefore broadly and reasonably be interpreted to mean that, even on a straight road, when an obstacle is present within a front threshold distance of the moving object, avoiding that obstacle is the highest priority.
With that in mind, see Caldwell, Caldwell Fig. 2 and paragraphs 0083-0090. In the top illustration related to first action 204 the host vehicle 202 may collide with vehicle 224. Therefore, the “safety” cost is high at 16. In fact, it is higher than the others. That trajectory is obviously not that ideal overall. This satisfies the limitations of claim 6 in which the parameter related to collision avoidance or safety swamps the parameters used to select the target trajectory, with the threshold distance being taught in paragraph 0056 (“threshold values associated with the distance”). The straight road is seen in Fig. 2. Note also that, according to Caldwell, paragraph 0054, turning onto a “substantially perpendicular” road is taught. The paragraph also teaches that an object can be a “threshold distance” from another vehicle. Depending on the route the host vehicle is going to take, that other vehicle may or may not be relevant.).
Regarding claim 9, Caldwell discloses:
A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform (see paragraph 0193) a method comprising:
generating a plurality of trajectories for a driving situation of the moving object based on either one or both of a speed and a steering of the moving object (for the remainder of the rejection, please see the analogous bullet points in the rejection of claim 1, which is substantially similar.);
selecting candidate trajectories based on a presence of an obstacle among the plurality of trajectories;
outputting hyperparameters related to driving of the moving object by inputting data related to the driving situation to a machine learning model;
selecting a target trajectory from the candidate trajectories based on the hyperparameters; and
controlling the steering and the speed such that the moving object moves along the target trajectory,
wherein the hyperparameters comprise a first hyperparameter for the speed of the moving object, a second hyperparameter for a degree to which the moving object is able to avoid an obstacle, and a third hyperparameter for a global path to a destination, and
wherein the hyperparameters vary while the moving object travels.
Regarding claim 12, Caldwell discloses:
An electronic device comprising: one or more processors configured to (see Fig. 7 including processor(s) 716):
generating a plurality of trajectories for a driving situation of the moving object based on either one or both of a speed and a steering of the moving object (see Fig. 9, step 902);
selecting candidate trajectories based on a presence of an obstacle among the plurality of trajectories (see steps 904-910, in these steps nearby obstacles are considered. With a YES out of 908 the system will eliminate trajectories that are not safe. Then the system will consider the remaining trajectories as candidate trajectories. See Fig. 1 for various trajectories of the host vehicle 102.);
output hyperparameters related to driving of the moving object by inputting data related to the driving situation to a machine learning model (see Caldwell, Fig. 2 in which various possible trajectories, first action 204 through fourth action 210, are graded by costs in various categories. This is just one example. Each of the trajectories shown in Fig. 1 could be adjusted as shown in Fig. 2 to determine an overall cost, as discussed in paragraph 0047.);
selecting a target trajectory from the candidate trajectories based on the hyperparameters (see Fig. 1, step 912 and 914. See also Fig. 8 for a YES out of 812.); and
control the steering and the speed such that the moving object moves along the target trajectory (see Fig. 8, step 814), and
wherein the hyperparameters comprise
a first hyperparameter for the speed of the moving object (see Fig. 2 for various costs. See paragraph 0030 for the comfort cost being related to “speed”. The comfort cost includes abrupt accelerations as well. The comfort cost is a way to grade sub-actions. For example, as described in paragraph 0047 and Fig. 2, there can be “speed variations” of a trajectory, and the differences can show up in the comfort cost of the sub-action.),
a second hyperparameter for a degree to which the moving object is able to avoid an obstacle (see Fig. 2, for grading trajectories based on “safety”. See Caldwell Fig. 2 and paragraphs 0083-0090. In the top illustration related to first action 204 the host vehicle 202 may collide with vehicle 224. Therefore, the “safety” cost is high at 16. In fact, it is higher than the others. That trajectory is obviously not that ideal overall.), and
a third hyperparameter for a global path to a destination (see Fig. 2 and paragraph 0013 in which “progress (e.g., movement toward destination)” is a factor. See paragraph 0124 for route planning. See paragraph 0072 for ranking the various costs, including the progress cost.), and
wherein the hyperparameters vary while the moving object travels (see paragraph 0022 for performing the analysis at 10 Hz.).
Regarding claims 13, 14, 16, and 17, they are substantially similar to claims 2,3, 5, and 6, respectively. Please see the rejections for those claims.
Regarding claim 20, Caldwell discloses:
A processor-implemented method comprising (see the method of Fig. 9 performed on the processor of Fig. 7, item 716):
selecting candidate trajectories based on a presence of an obstacle among a plurality of trajectories generated for a driving situation of a moving object based on either one or both of a speed and a steering of the moving object (see steps 904-910, in these steps nearby obstacles are considered. With a YES out of 908 the system will eliminate trajectories that are not safe. Then the system will consider the remaining trajectories as candidate trajectories. See Fig. 1 for various trajectories of the host vehicle 102. For the generating of the plurality of trajectories in the first place, see Fig. 9, step 902);
using a machine learning model, adjusting hyperparameters based on whether the obstacle is present within a front threshold distance of the moving object and whether a curvature of a path on which the moving object travels is within a threshold curvature (see steps 904-910, in these steps nearby obstacles are considered. With a YES out of 908 the system will eliminate trajectories that are not safe, which according to paragraph 0013, are trajectories that including “avoiding a collision between the vehicle and the object”. See paragraph 0025 for the teaching that a “collision may be based on threshold values associated with the distance [between the host vehicle and obstacle],” which can be “within 5 feet…etc.” A collision can be determined based on “threshold distance”. Then the system will consider the remaining trajectories as candidate trajectories. See Fig. 1 for various trajectories of the host vehicle 102. See Fig. 2 and paragraphs 0083-0090. In the top illustration related to first action 204 the host vehicle 202 may collide with vehicle 224. Therefore, the “safety” cost is high at 16. In fact, it is higher than the others. That trajectory is obviously not that ideal overall. This satisfies the limitations in which the parameter related to collision avoidance or safety swamps the parameters used to select the target trajectory, with the threshold distance being taught in paragraph 0056 (“threshold values associated with the distance”). The straight road is seen in Fig. 2. Note also that, according to Caldwell, paragraph 0054, turning onto a “substantially perpendicular” road is taught. The paragraph also teaches that an object can be a “threshold distance” from another vehicle. Depending on the route the host vehicle is going to take, that other vehicle may or may not be relevant.
What if there was no vehicle 224 present but everything else the same? Then the safety cost would obviously be 0 or something else very low. In that case, acceleration could be commanded. That case satisfies the limitations of present claim 5, with the threshold distance being taught in paragraph 0056 (“threshold values associated with the distance”). Note that the system in Caldwell is constantly executing commands, generating trajectories, scoring them, and selecting new ones. See paragraph 0022 for performing the analysis at 10 Hz.); and
determining a score for each of the candidate trajectories using the hyperparameters (see Fig. 2 for the “Total” cost for four different trajectories.); and
determining a target trajectory by selecting a candidate trajectory having a highest score among the scores (see paragraph 0077 for the trajectory selected being the one with the “lowest cost” which is synonymous with the “high score.” See also paragraph 0090.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 4, 10, 11, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Caldwell et al. (US2021/0046924) in view of Linscott et al. (U.S. 12,037,013).
Regarding claim 4, Caldwell discloses the method of claim 1.
Yet Caldwell does not further disclose:
The method of claim 1, wherein
the machine learning model is a model trained through reinforcement training according to a reward determined based on
a first factor for the speed of the moving object, a second factor for a degree to which the moving object is able to avoid an obstacle, and a third factor for a distance between the moving object and the destination.
However, Linscott teaches:
the machine learning model is a model trained through reinforcement training according to a reward determined based on (see col. 2, lines 10-11 for “Determining a performance metric may be based at least in part on a reward determined as part of a reinforcement learning technique for training a component of the autonomous vehicle.” )
a first factor for the speed of the moving object (see col. 2, lines 10-15 for “Determining a performance metric may be based at least in part on a reward determined as part of a reinforcement learning technique for training a component of the autonomous vehicle.”…the performance metric may be based on…a velocity”. See col. 2, line 65-col. 3, line 16 for: “The reward may be based at least in part on a variety of operating parameter(s)… a jerk and/or acceleration by the vehicle. See col. 3, lines 20-21 for “the reward may be a penalty that increase as the velocity increases”), a second factor for a degree to which the moving object is able to avoid an obstacle (see col. 2, lines 10-15 for “Determining a performance metric may be based at least in part on a reward determined as part of a reinforcement learning technique for training a component of the autonomous vehicle.” “The performance metric may be based on…whether an impact occurred”. See col. 2, line 65-col. 3, line 16 for: “The reward may be based at least in part on a variety of operating parameter(s), such as a minimum distance between the vehicle and a nearest object… For example, the reward output by the machine-learned model may be based at least in part on a predicted likelihood of impact), and a third factor for a distance between the moving object and the destination (see col. 2, line 65-col. 3, line 16 for: “The reward may be based at least in part on a variety of operating parameter(s), such as… a deviation from a route”. See col. 1, lines 59-66 for “training an autonomous vehicle to handle various scenarios…by taking a most efficient or logical route”.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Caldwell, to add the additional features of the machine learning model is a model trained through reinforcement training according to a reward determined based on a first factor for the speed of the moving object, a second factor for a degree to which the moving object is able to avoid an obstacle, and a third factor for a distance between the moving object and the destination, as taught by Linscott. The motivation for doing so would be to “train the autonomous vehicle to avoid impacts and act naturally”, as recognized by Linscott (see col. 3, lines 25-26).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
In summary, machine learning systems, such as the one taught by Caldwell, are frequently trained using reinforcement learning. Caldwell does not explicitly get how the system in Caldwell is trained, but Linscott cures this deficiency. Since the disclosures are both owned by Zoox, and teach highly similar trajectory generation and selection systems, the two are very compatible.
Regarding claim 10, Caldwell teaches:
A processor-implemented method comprising (see paragraph 0193):
selecting candidate trajectories based on a presence of an obstacle among a plurality of trajectories generated for a driving situation of the moving object based on either one or both of a speed and a steering of the moving object (see steps 904-910, in these steps nearby obstacles are considered. With a YES out of 908 the system will eliminate trajectories that are not safe. Then the system will consider the remaining trajectories as candidate trajectories. See Fig. 1 for various trajectories of the host vehicle 102. See also Fig. 9, step 902 for generating the trajectories that eventually get selected as candidates.);
outputting hyperparameters related to driving of the moving object by inputting data related to the driving situation to a machine learning model (see Fig. 2 in which various possible trajectories, first action 204 through fourth action 210, are graded by costs in various categories. This is just one example. Each of the trajectories shown in Fig. 1 could be adjusted as shown in Fig. 2 to determine an overall cost, as discussed in paragraph 0047.);
selecting a target trajectory from the candidate trajectories based on the hyperparameters (see Fig. 1, step 912 and 914. See also Fig. 8 for a YES out of 812.); and
controlling the steering and the speed such that the moving object moves along the target trajectory (see Fig. 8, step 814),
wherein the hyperparameters vary while the moving object travels.
Yet Caldwell does not further disclose:
wherein the machine learning model is a model trained through reinforcement training according to a reward determined based on
a first factor for the speed of the moving object, a second factor for a degree to which the moving object is able to avoid an obstacle, and a third factor for a distance between the moving object and the destination.
However, Linscott teaches:
wherein the machine learning model is a model trained through reinforcement training according to a reward determined based on
a first factor for the speed of the moving object (see col. 2, lines 10-15 for “Determining a performance metric may be based at least in part on a reward determined as part of a reinforcement learning technique for training a component of the autonomous vehicle.”…the performance metric may be based on…a velocity”. See col. 2, line 65-col. 3, line 16 for: “The reward may be based at least in part on a variety of operating parameter(s)… a jerk and/or acceleration by the vehicle. See col. 3, lines 20-21 for “the reward may be a penalty that increase as the velocity increases”), a second factor for a degree to which the moving object is able to avoid an obstacle (see col. 2, lines 10-15 for “Determining a performance metric may be based at least in part on a reward determined as part of a reinforcement learning technique for training a component of the autonomous vehicle.” “The performance metric may be based on…whether an impact occurred”. See col. 2, line 65-col. 3, line 16 for: “The reward may be based at least in part on a variety of operating parameter(s), such as a minimum distance between the vehicle and a nearest object… For example, the reward output by the machine-learned model may be based at least in part on a predicted likelihood of impact), and a third factor for a distance between the moving object and the destination (see col. 2, line 65-col. 3, line 16 for: “The reward may be based at least in part on a variety of operating parameter(s), such as… a deviation from a route”. See col. 1, lines 59-66 for “training an autonomous vehicle to handle various scenarios…by taking a most efficient or logical route”.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Caldwell, to add the additional features of the machine learning model is a model trained through reinforcement training according to a reward determined based on a first factor for the speed of the moving object, a second factor for a degree to which the moving object is able to avoid an obstacle, and a third factor for a distance between the moving object and the destination, as taught by Linscott. The motivation for doing so would be to “train the autonomous vehicle to avoid impacts and act naturally”, as recognized by Linscott (see col. 3, lines 25-26).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 11, Caldwell and Linscott teach the method of claim 10.
Caldwell further teaches:
The method of claim 10, wherein
the hyperparameters comprise
a first hyperparameter for the speed of the moving object (see Fig. 2 for various costs. See paragraph 0030 for the comfort cost being related to “speed”. The comfort cost includes abrupt accelerations as well. The comfort cost is a way to grade sub-actions. For example, as described in paragraph 0047 and Fig. 2, there can be “speed variations” of a trajectory, and the differences can show up in the comfort cost of the sub-action.),
a second hyperparameter for a degree to which the moving object is able to avoid an obstacle (see Fig. 2, for grading trajectories based on “safety”. See Caldwell Fig. 2 and paragraphs 0083-0090. In the top illustration related to first action 204 the host vehicle 202 may collide with vehicle 224. Therefore, the “safety” cost is high at 16. In fact, it is higher than the others. That trajectory is obviously not that ideal overall.), and
a third hyperparameter for a global path to a destination (see Fig. 2 and paragraph 0013 in which “progress (e.g., movement toward destination)” is a factor. See paragraph 0124 for route planning. See paragraph 0072 for ranking the various costs, including the progress cost.).
Regarding claim 15, it is substantially similar to claim 4. Please see the rejection for that claim.
Claims 7 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Caldwell et al. (US2021/0046924) in view of Ng et al. (US2024/0059285).
Regarding claim 7, Caldwell discloses the method of claim 1.
Yet Caldwell does not explicitly further disclose:
The method of claim 1, wherein,
in response to the driving situation being a situation in which the moving object travels on a driving lane comprising two or more branches, the third hyperparameter is determined to be greatest among the hyperparameters.
However, Ng teaches:
in response to the driving situation being a situation in which the moving object travels on a driving lane comprising two or more branches, the third hyperparameter is determined to be greatest among the hyperparameters (see Ng Fig. 1 for a trajectory scorer (motion planner) 138 in Fig. 1 and paragraph 0084. See paragraph 0088 teaches: “In some examples, such as when multiple future paths 104 were used to generate multiple trajectories 132, the trajectory scorer 138 may use additional factors to determine the scores for the trajectories 132. For example, in some situations one lane may work as well as any other available lane. As an illustration, a vehicle traveling down an interstate highway 20 miles away from an exit indicated by a route may use either lane without meaningfully impacting travel time. In this circumstance, if the two trajectories 132 are associated with two different lanes, then the trajectory scorer 138 may determine that the scores for the trajectories 132 are similar (e.g., at least just based on the lanes). On the other hand, as the vehicle approaches the exit, the score for the trajectory 132 that is associated with the exit lane may increase while the score for the trajectory 132 that is associated with the other lane decreases.” See Fig. 9A and paragraph 0090 for “a road 904 that splits into two additional roads 906(1)-(2).”
See also paragraph 0085 for “For example, the trajectory scorer 138 may determine that a trajectory(ies) 132 that includes no probability and/or a low probability (e.g., a probability less than a threshold probability) of collision with an object(s) includes a high score(s) while a trajectory(ies) 132 that includes a high probability (e.g., a probability that exceeds a threshold probability) of collision with an object(s) includes a low score(s). In some examples, the trajectory scorer 138 may determine the score(s) based on the velocity(ies) or speed(s) associated with the trajectory(ies) 132. For example, the trajectory scorer 138 may determine that a trajectory(ies) 132 that includes a velocity(ies) that is approximately equal to a speed limit(s) of a road(s) includes a high score(s) and that a trajectory(ies) 132 that includes a velocity(ies) that is different than the speed limit(s) of the road(s) includes a low score(s).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Caldwell, to add the additional features of, as taught by Ng. The motivation for doing so would be to maintain good “travel time,” as recognized by Ng (see paragraph 0088).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
This combination is especially obvious because Caldwell at least strongly teaches toward what Ng more explicitly teaches. See Caldwell, Fig. 2 and paragraph 0013 in which “progress (e.g., movement toward destination)” is a factor. See paragraph 0124 for route planning. See paragraph 0072 for ranking the various costs, including the progress cost. See paragraph 0033 for ranking some factors above others. Even though claim 7 does not say it explicitly, it would seem unreasonable to the examiner to interpret the claim to mean that the host vehicle does not care if it collides with another vehicle or runs over a pedestrian so long as the host vehicle makes its exit. In part that is why unsafe actions are initially excluded. After that, surely, the system may need to slow down to change lanes but is willing to do that to take the correct exit or turn in order to maintain the desired route. Caldwell teaches that. See paragraph 0037 for determining safe routes and then sorting them. See also paragraph 0070 which teaches that “factor(s) may be ranked in order of importance.” Various non-limiting examples, such as ranking the safety cost as highest, are provided, but these are characterized as only “examples”. Equations (1) and (2) in Caldwell could be different, and assign a high value to progress. That concept is in Caldwell paragraphs 0070-0072.
Regarding claim 18, it is substantially similar to claim 7. Please see the rejection for that claim.
Claims 8 and 19 rejected under 35 U.S.C. 103 as being unpatentable over Caldwell et al. (US2021/0046924) in view of Kumar et al. (US2022/0221294).
Regarding claim 8, Caldwell discloses the method of claim 1.
Yet Caldwell does not further disclose:
The method of claim 1, wherein,
in response to a density of obstacles increasing with respect to an empty space around a driving lane, on which the moving object travels, the second hyperparameter is determined to increase.
However, Kumar teaches:
in response to a density of obstacles increasing with respect to an empty space around a driving lane, on which the moving object travels, the second hyperparameter is determined to increase (see paragraph 0048 and table 1 for grading a plurality of trajectories based on weights of parameters, including lane congestion and road curvature. See paragraph 0053 for these weights being updated “dynamically” as the scenarios change. Thus, increasing congestion results in an increase cost. See paragraph 0024 for the trajectories being graded on road parameters such as curvature.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Caldwell, to add the additional features of: in response to a density of obstacles increasing with respect to an empty space around a driving lane, on which the moving object travels, the second hyperparameter is determined to increase, as taught by Kumar. The motivation for doing so would be to have a machine learning model trained to select the final maneuvering trajectory that increase efficiency in navigation, as recognized by Kuman (see paragraph 0007).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 19, it is substantially similar to claim 8. Please see the rejection for that claim.
Additional Art
The prior art made of record here, though not relied upon, is considered pertinent to the present disclosure.
Weast et al. (US2025/0206354), a Mobileye disclosure, shows in Fig. 1A that prior art taught top level route planning, then motion planning.
PNG
media_image1.png
824
564
media_image1.png
Greyscale
Johnson et al. (US2023/0166764). See paragraph 0054-0056 for inputting a set of information such as vehicle state variables and sensor data. See paragraph 0064 for “performing a set of simulations” related to potential trajectories of the host vehicle and nearby vehicles. The set of future scenarios can including “maintaining a current policy, implementing a particular set of control commands, updating a destination of the ego agent, updating a goal or set of objectives of the ego agent, encountering a particular set and/or change in environmental conditions” etc. See paragraph 0084 for avoiding “ ‘close call’ events” and instead having a “distance maintenance goal”.
See paragraph 0099 for applying a “scaling factor” to “parameters”. For example, an ego agent may have the right-of-way, and therefore “any impact on environmental agents can be minimized or eliminated. Alternatively, parameters can not be calculated, made equal to a constant…and/or otherwise adjusted.”
See paragraph 0090 for the environmental agent goal metric including location parameters, such as ones related to other agents and/or road geometry. For example, driving on the shoulder can be weighted differently or lower or decreased in weight.
Dong (WO2025075500), filed Oct 2, 2023, Dong teaches: For example, when approaching an intersection more or less together with another vehicle, an egotistic vehicle may accelerate and take precedence, whereas the prosocial vehicle may decelerate and give way. The methods according to embodiments described herein can handle also complex manoeuvres, e.g., driving through roundabouts (both single-lane and two-lane) with large curvature, which is one of the most accident-prone scenarios. In an embodiment, the first cost value, the second cost value, and/or the overall cost value is determined using a model with trained hyperparameters, wherein the trained hyperparameters have been trained by iteratively performing the steps of: selecting a set of hyperparameters of the model; simulating an environment comprising the vehicle and the other traffic participant; determining input parameters for the model based on the simulated environment; determining the optimised set of control parameters using the model with the selected set of hyperparameters;update the simulated environment, based on the optimised set of control parameters; and determining a reward associated with the hyperparameters, based on the updated simulated environment. Training the model may further comprise selecting a set of trained hyperparameters based on the rewards associated with the hyperparameters. Such a training method may also be referred to as deep reinforcement learning. Deep reinforcement learning may result in a well-trained model. In an aspect, this disclosure relates to a method of training one or more first hyperparameters of a first cost function for determining a first cost value associated with a set of control parameter values, the first cost value being based on a difference between one or more potential future states of a vehicle and one or more target states of the vehicle at one or more future time steps, and/or one or more second hyperparameters of a second cost function for determining a second cost value associated with the set of control parameter values, the second cost value being based on a perceived risk posed by the vehicle in the one or more future states of the vehicle to another traffic participant in respective predicted one or more future states of the other traffic participant at the one or more future time steps, and/or one or more third hyperparameters of a third cost function for determining an overall cost value associated with the set of control parameter values, the overall cost value being based on the first cost value and the second cost value. The method comprises iteratively performing the steps of: selecting a set of first, second, and/or third hyperparameters; simulating an environment comprising the vehicle and the other traffic participant; determining input parameters for the model based on the simulated environment; determining the optimised set of control parameters using the first, second, and/or thirs cost function with the selected set of first, second, and/or third hyperparameters; update the simulated environment, based on the optimised set of control parameters; and determining a reward associated with the selected set of first, second, and/or third hyperparameters, based on the updated simulated environment. The method further comprises determining a set of trained first, second, and/or third hyperparameters based on the rewards associated with the first, second, and/or third hyperparameters.
Cui (US2023/0322270), a Motional AD disclosure, see Fig 5, attached below, and paragraph 0080 for a machine learning model 512 that is “trained” using the “reward function 522.” The training is “based on demonstrations of an expert policy (e.g., one or more simulations) that includes the correct trajectories for the vehicle encountering a variety of scenarios.” Trajectories that closely match the expert trajectory are rewarded. “The trajectory that is consistent with the expert policy may avoid collision between the vehicle and one or more objects…[and] may enable the vehicle to operate in according with certain desirable characteristics such as path length, ride quality or comfort, required travel time, observance of traffic rules, adherence to driving practices, and/or the like.” See paragraph 0085 for: “In some embodiments, one or more trajectories for the vehicle (e.g., sequences of actions 520) can be pre-loaded/pre-stored by the system 500. Moreover, the motion planner 510 can, such as, during training of the machine learning model 512, generate and store additional trajectories and/or refine the pre-loaded/pre-stored trajectories as well as refine generated trajectories upon receiving further sensor data and/or any other information associated with the vehicle's health, environment, etc. In addition to the provided sensor data and/or pre-loaded/pre-stored trajectories, the machine learning model 512 can be trained to implement one or more safety rules 524 and reward values provided by the reward function 522. Reward values are generated based on the data 523 (e.g., vehicle's conditions, conditions present in the vehicle's surrounding environment, and/or the like) supplied to the reward function 522 from the system monitor 508, any trajectories that may have been generated (or selected), as well as safety rules 524.”
PNG
media_image2.png
604
534
media_image2.png
Greyscale
Kwon et al. (U.S. 12,091,025) teaches reinforcement learning.
Shi et al. (US2024/0157944) teaches in at least Fig. 6A, attached below, and paragraphs 0021 and 0049 a system in which “machine learning (ML) engine 122 can train a machine learning (reinforcement learning) model”. The system uses “rewards determiner 407.” See paragraph 0051 for the rewards determiner 407 applying positive or negative rewards for the autonomous vehicle “ADV being in a blind spot of an obstacle vehicle, for collision in the traffic, for causing the traffic to slowly down, etc.”
PNG
media_image3.png
466
722
media_image3.png
Greyscale
Kobilarov et al. (U.S. 11,891,088), a Zoox disclosure, teaches in at least Figs. 2 and 5, attached below, and col. 27, lines 11-36, a system that uses a reinforcement learning algorithm as part of an “action-reward feedback loop” in which the rewards are determined on a per trajectory basis. “Determining the reward may be based at least in part on a set of rules, such as the operating constraints, and may be accomplished by an ML model trained to score performance of the vehicle. For example, the ML model may comprise a reward function that determines a score based at least in part on the operating constraints discussed herein. This reward may be used to modify scenario data at operation 508 and/or as a basis for controlling an adversarial agent by the adversarial agent component at operation(s) 520 and/or 522.”
PNG
media_image4.png
606
930
media_image4.png
Greyscale
PNG
media_image5.png
912
626
media_image5.png
Greyscale
Minamoto et al. (US2022/0026234). See paragraph 0069 for a determination unit 23 that trains a machine learning model using reinforcement learning. “Speed” can be used as a reward. See paragraph 0070 for the reward being related to “taking into account the route information” so that vehicle can reach the destination efficiently. See paragraph 0071 for the reward being related to collisions.
Nister et al. (US2021/0253128) teaches in paragraph 0024 taking into account missing an exit when determining how to score trajectories. This includes increasing the importance of a lane as an exit draws near. Overall, the disclosure teaches generating a plurality of trajectories and picking one, as seen in Fig. 7 attached below.
PNG
media_image6.png
814
454
media_image6.png
Greyscale
Ahmed et al. (US2022/0219728). See paragraph 0074 for a system that can select trajectories based on the threshold of curvature.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL M. ROBERT whose telephone number is (571)270-5841. The examiner can normally be reached M-F 7:30-4:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at 571-272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL M. ROBERT/Primary Examiner, Art Unit 3665