DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 8, and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Schmidt et al. (US2023/0399027).
Regarding claim 1, Schmidt discloses:
An apparatus for controlling autonomous driving of a vehicle, the apparatus comprising (see Schmidt, Figs. 1 and 8):
a processor (see Schmidt, Fig. 8 and paragraph 0151); and
a memory configured to store one or more instructions, when executed by the processor, configured to cause the apparatus to (see Schmidt, Fig. 8 and paragraph 0151):
store trajectory history data of a target object (see Schmidt, paragraph 0107 for an “observed driving trajectories T1 of the detected road users 303”. In Schmidt there are the immediately observed trajectory, T1, which may be “normal” or not, and then there are a bunch of normal trajectories, T2, that the encoder has been trained on, according to paragraph 0107. Schmidt teaches comparing the two to determine of T1 is normal or not.);
generate, based on the trajectory history data, a trajectory history matrix for a time window of a sampling (in the present disclosure, see Fig. 2 and paragraph 0053 for a trajectory history matrix; see paragraph 0050 for an example of a 2 second window of sampling.
With that in mind, see Schmidt Fig. 4 and paragraph 0086 for generating a “latent-space representations 325, corresponding to the input data 321, in latent space.” See paragraph 0082 for trajectory information including “various time points.” See Fig. 4, item 339 for image data, which has “temporal sequences” according to paragraph 0121. The graphic in Fig. 4, item 339 indicates this window is limited and a person of ordinary skill in the art understands trajectories to be limited in time.
Although the Schmidt often refers to “image data” the term image does not connote a photographic camera image, as is sometimes implied by the word. Rather, as made clear in paragraph 0121, the image data together with map data and trajectory data, and even other kinds of data such as weather conditions, traffic regulations, and traffic volume, “have temporal sequences of image data 339, 341. The image data 339 of the trajectory data 327 here describe lateral changes in the driving trajectories T1, described by the trajectory data 327, of the respectively observed road user 303.” Furthermore, Schmidt teaches in paragraph 0112 that all this information is “merged into a common data set that serves as an input data set for the connection network 345.”);
input the trajectory history matrix into a machine learning model to determine reconstruction loss (in the present disclosure, see paragraph 0055 for that statement that the “reconstruction loss refers to a difference between input and output.” See paragraph 0056 for it referring to “the error or difference between the original input (the trajectory history matrix [of sensor data]) and the output generated by the autoencoder (the reconstructed trajectory history matrix).” Brackets added by examiner. The autoencoder is the neural network. The paragraph goes on to state that “the output (reconstructed trajectory history matrix) may rarely be a perfect copy of the input, the reconstruction loss quantifies the extent of this difference. The loss may be determined using various mathematical measures, such as the Mean Squared Error (MSE) or another distance metric.” The loss can be compared to a threshold, and if greater, can be considered “that the input data is abnormal (e.g., not fit the patterns the model has learned from normal trajectory data). This mechanism may be used for anomaly detection…abnormal data…doesn’t conform to the learned patterns.” See paragraph 0072 for: “If the target moves irregularly, reconstruction loss of the trajectory history matrix is likely to be large.”
With that in mind, see Schmidt, Fig. 4 and paragraph 0112 for the data 321 and 322 being fed into the “encoder module 317”. See paragraph 0003 for “A proper assessment of future driving behavior of further road users is essential for controlling vehicles, in particular autonomously controllable vehicles. For safely controlling the vehicle, it is in particular important to be able to distinguish between normal driving behavior, where unforeseeable actions are not expected, and abnormal driving behavior, which deviates from normal driving behavior and where unforeseeable actions are expected to be carried out in the future, in order to be able to adjust, if necessary, the driving behavior of the vehicle to be controlled.” This determination of normal versus abnormal driving is obtained by the distance module 318, as seen in Fig. 4. See paragraph 0096 for determining a distance D which is a comparison between the latent-space representation 325 and “normal trajectories based on normal driving behavior.” In Schmidt, “normal” does not mean perpendicular to, as in one definition of the word. Rather is means typical. See also paragraph 0097 for distance D being a classification of the driving behavior based on determining if the trajectory T1 is “less than a predetermined limit value” distance D from the “normal driving behavior.” See paragraph 0143 for the teaching that “On the other hand, in a further method step 109, the driving behavior of the road user 303 is classified as an abnormal driving behavior if the distance D of the latent-space representation 334 of the driving trajectory T1 of the road user 303 to the latent-space representation 335 of the at least one normal trajectory T2 in the latent space 333 exceeds a predetermined limit value (step 106).”),
wherein the machine learning model comprises an autoencoder trained based on previous trajectory history data associated with movement of at least one object (see paragraphs 0088, 0090, 0092);
determine, based on the reconstruction loss, a trajectory prediction accuracy (in the present disclosure, the “trajectory prediction accuracy,” or just “accuracy,” can be determined with Equation 1 in paragraph 0066, for example, or Equation 2, in paragraph 0068. According to paragraph 0070 “Accuracy of trajectory prediction may indicate uncertainty about a future state of the target. For example, if the target is a vehicle and the driver tends to drive aggressively, it is unlikely that the driver will drive on a consistent route. Therefore, the accuracy of trajectory prediction is low. Conversely, a vehicle that is stopped for a certain period of time is likely to remain stopped at a certain point in the future. Therefore, the accuracy of trajectory prediction is high.” This paragraph may contain the most detailed examples in the spec.
In the present disclosure, paragraph 0056, is one of the most important paragraphs in the spec. Based on this paragraph, it seems that sensor data that is processed into a history matrix is inputted into a neural network (or “autoencoder”) and the neural network outputs a “reconstructed trajectory history matrix”. The difference between this past trajectory as measured by sensors, and the reconstruction of it by the neural network is measured. The difference between the original history matrix and the reconstructed one is the “reconstruction loss”. If the loss is above a threshold that means that “the input data is abnormal (e.g., not fit the patterns the model has learned from normal trajectory data).” In a broad reasonable interpretation, this could mean that if a car is in the left turn lane but then takes a right turn, or even merges right and drives straight, this could be considered abnormal behavior. Most cars in the left turn lane, in the extensive data that the neural network has been trained on, turn left. So the large that does not do that is behaving abnormally.
Yet none of this has to do with “prediction.” How does prediction figure into the disclosure? In a broad reasonable interpretation, the present disclosure identifies abnormal drivers or pedestrians and then determines that “trajectory predictions,” which are extrapolations into the future, will probably not be accurate, because the neural network has already found that what the object is currently behaving in a way that does not conform to patterns of the norm. So the present disclosure does not ever predict a trajectory. It does not attempt to forecast exactly where an object is going to go in the future based on data from the past. But the present disclosure does assign what could be considered an erratic driving grade, to objects. That grade is called “a trajectory prediction accuracy.” If the object is behaving abnormally, the system detects that and then essentially says: future trajectory predictions are probably not going to be that accurate; the trajectory prediction accuracy for this object is low.
So while paragraph 0033 teaches that “In the present disclosure, trajectory prediction refers to predicting…a future trajectory of an object,” the actual predicting is not the focus of the disclosure. Rather, the focus is on determining a level of confidence in any prediction that might be made. So paragraph 0033 further states: “In the present disclosure, accuracy of trajectory prediction refers to a level of confidence of a predicted future trajectory.”
Note that the input into the neural network is a history matrix, which is processed sensor data or a driving trajectory. The neural network has already been trained on a large data set of driving data. So the neural network compares a small amount of “trajectory history data” from the recent past, such as 2 seconds of sensor data showing the behavior of an object, and compares it to the large data set. This comparison products a loss function that is a measure of how typical the behavior is compared to drivers in the large data set.
With that in mind, see Schmidt, paragraph 0096 for determining a distance D which is a comparison between the latent-space representation 325 and “normal trajectories based on normal driving behavior.” In Schmidt, “normal” does not mean perpendicular to, as in one definition of the word. Rather is means typical. See also paragraph 0097 for distance D being a classification of the driving behavior based on determining if the trajectory T1 is “less than a predetermined limit value” distance D from the “normal driving behavior.” See paragraph 0143 for the teaching that “On the other hand, in a further method step 109, the driving behavior of the road user 303 is classified as an abnormal driving behavior if the distance D of the latent-space representation 334 of the driving trajectory T1 of the road user 303 to the latent-space representation 335 of the at least one normal trajectory T2 in the latent space 333 exceeds a predetermined limit value (step 106).” See Fig. 6, items 109, and 107.);
generate a signal indicating the trajectory prediction accuracy (see Schmidt, Fig. 4, output D. See Fig. 6, item 111.); and
control, based on the signal, the autonomous driving of the vehicle (see Fig. 7, steps 203 and 205 and paragraphs 0147-0150.).
Regarding claim 8, Schmidt discloses:
A method performed by an apparatus for controlling autonomous driving of a vehicle, the method comprising (see the title of Schmidt.):
storing trajectory history data of a target object (for the remainder of the rejection, see the substantially similar bullets in the rejection of claim 1.);
generating, based on the trajectory history data, a trajectory history matrix for a time window of a sampling;
inputting the trajectory history matrix into a machine learning model to determine reconstruction loss,
wherein the machine learning model comprises an autoencoder trained based on previous trajectory history data associated with movement of at least one object;
determining, based on the reconstruction loss, a trajectory prediction accuracy;
generating a signal indicating the trajectory prediction accuracy; and
controlling, based on the signal, the autonomous driving of the vehicle.
Regarding claim 8, Schmidt discloses:
A non-transitory computer-readable medium storing instructions, when executed, cause an apparatus to (see claim 16 of Schimdt):
store trajectory history data of a target object (for the remainder of the rejection, see the substantially similar bullets in the rejection of claim 1.);
generate, based on the trajectory history data, a trajectory history matrix for a time window of a sampling;
input the trajectory history matrix into a machine learning model to determine reconstruction loss, wherein the machine learning model comprises an autoencoder trained based on previous trajectory history data associated with movement of at least one object;
determine, based on the reconstruction loss, a trajectory prediction accuracy;
generate a signal indicating the trajectory prediction accuracy; and
control, based on the signal, autonomous driving of a vehicle.
Potentially Allowable Subject Matter
Claims 2-7, 9-14, and 16-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter.
Claim 2 is not taught by the prior art of record, alone or in combination. The claim recites:
The apparatus of claim 1, wherein
the one or more instructions, when executed by the processor, further configured to cause the apparatus to determine a rank of the trajectory history matrix,
wherein the determination of the trajectory prediction accuracy is further based on the rank of the trajectory history matrix.
In the present discloses, a rank is related to the dimensions of the matrix. The closest prior art is Schmidt et al. (US2023/0399027), but Schmidt does not further teach the limitations of claim 2. Schmidt teaches that the trajectories input into the encoder can include additional information such as weather conditions, time of day, day of the week, etc. But Schmidt does not state that the more of these sources of information, the more accurate the trajectory prediction should be rated.
Claim 3 is allowable for at least the reasons of claim 2.
Claim 4 is not taught by the prior art of record, alone or in combination. The claim recites:
The apparatus of claim 1, wherein
the determination of the trajectory prediction accuracy is further based on a target recognition confidence level of a sensor.
In the present disclosure, see paragraph 0041 for the target recognition confidence level being basically exactly what it sounds like. Reasonably, vehicles sometimes determine if a sensor is properly working using fault detection, or determining that a video camera, for example, should not be outputting an apparently blank screen, therefore it must be covered by debris, snow, or ice. Such a confidence level can be reported along with the sensor data.
One close prior art is Marcotte et al. (US2022/0126865 A1) teaches an “uncertainty level” corresponding to “a driving data stream,” which can mean, according to paragraph 0020, a camera sensor that is dirty or providing poor image quality resulting in the image data having a high uncertainty level. In Fig. 1 and paragraph 0023, one can see that a high uncertainty level is shown with a dotted line, a medium uncertainty level with a dashed line, and a low uncertainty level with a solid line.
But Marcotte does not teach a neural network, much less weighting the output of a neural network based on the confidence level of a sensor, as in the present claim.
Claim 5 is allowable for at least the reasons of claim 4. One close prior art to claim 5 is Mummadi et al. (U.S. 11,947,625), also by Bosch. But Mummadi does not teach at least the first bullet of present claim 5, nor would combining be obvious because Mummadi does not teach anything about a trajectory prediction accuracy.
Claim 6 is not taught by the prior art of record, alone or in combination. The claim recites:
The apparatus of claim 1, wherein
the one or more instructions, when executed by the processor, are further configured to cause the apparatus to set, based on a driving environment of the vehicle, the time window of a sampling to a different time window.
Several prior art references teach adjusting a sampling rate based on if the vehicle is in a busy urban area compared to sparse rural area, or base on a low speed compared to a high speed (urban vs. highway). But these references do not teach an adaptive time window based on the driving environment. For examples, see:
Wee (US2012/0242972) teaches adjusting the sampling rate depending on the speed of the vehicle.
Ma et al. (US2023/0060383), paragraph 0065 teaches that “a user expecting heavy computational loads,” such as that found on a busy high-speed road compared to a sparse low-speed road, “may select a high-performance setting, while a user expecting light computational loads (e.g., sparse object density) may select a low-performance setting to reduce operating costs relative to performing computations at a higher performance setting.”
Wilkens (U.S. 9,494,511) teaches in claim 8 “a processing unit coupled to the sensor and the navigation system and configured to: integrate the first navigation unit and the second navigation unit to calculate position data and velocity data, adjust the adjustable sampling rate based on the velocity data such that a sample period of the sensor is shorter when a velocity is higher and longer when the velocity is lower, receive, from the sensor, the data related to the plurality of atmospheric samples taken at the adjustable sampling rate, combine the atmospheric sample data with position data into data packets, and transmit, the data packets to a server.” This is done using a vehicle but sampling atmospheric data is much different than object trajectories, and the sampling is opposite from the example in the present disclosure, which essentially teaches longer sampling at higher velocities.
Claim 7 is not taught by the prior art of record, alone or in combination. The claim recites:
The apparatus of claim 1, wherein
the one or more instructions, when executed by the processor, are further configured to cause the apparatus to set the time window of a sampling to be longer for a highway driving than a downtown driving.
In the present disclosure, see paragraph 0050 for sampling a 5 seconds in the case of highway driving while sampling at 2 seconds for downtown driving, as examples.
The closest prior art is that same as that for claim 5.
Claims 9-14 are substantially similar to claims 2-7 and claims 16-20 are substantially similar to claims 2-6. These claims are allowable for at least the reasons of their substantially similar claims, which are detailed in this section above.
Additional Art
The prior art made of record here, though not relied upon, is considered pertinent to the present disclosure.
Nakaya (US2022/0327932) teaches comparing a vehicle’s behavior to that of other vehicle’s to determine if the vehicle is driving dangerously.
PNG
media_image1.png
586
494
media_image1.png
Greyscale
PNG
media_image2.png
802
528
media_image2.png
Greyscale
w of
PNG
media_image3.png
706
538
media_image3.png
Greyscale
PNG
media_image4.png
792
540
media_image4.png
Greyscale
Abroshan (US2025/0050873) teaches a system to determine a hazard index for each of the one or more target vehicles, according to the detected motion parameters , a vehicle traffic rules violation index
Craig et al. (U.S. 12,067,812) teaches sampling driving data and comparing it to a reference population (group of other drivers’ data) to determine how safe or risky a driver’s behavior is.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL M. ROBERT whose telephone number is (571)270-5841. The examiner can normally be reached M-F 7:30-4:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at 571-272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL M. ROBERT/Primary Examiner, Art Unit 3665
Prosecution Insights