DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This action is in reply to the Application Number 18/320,776 filed on 5/19/2023.
Claims 1-26 are currently pending and have been examined.
This action is made NON- FINAL in response to the “Amendment” and “Remarks” filed on 10/21/2025.
This action is made NON-FINAL in response to the “Request for Continued Examination” filed on 12/1/2025.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 7-10, 12, 14-17, 19, 21 are rejected under 35 U.S.C. 103 as being unpatentable over RoyChowdhury (U.S. Patent Publication 2022/0044034 A1) in view of Mao (U.S. Patent Publication 2020/0125112 A1), in further view of Postnikov (U.S. Patent Publication 2022/0105947 A1).
In regard to Claim 1, RoyChowdhury teaches a method comprising (see Paragraph 5 lines 5-12 teaching a vehicle system that uses sensory data to optimize image processing based on machine learning):
Obtaining a set of data corresponding to operation of a vehicle, wherein the set of data includes a first set of data corresponding to operation of a vision-based detection system and a second set of data corresponding to operation of a detection system including a non-vision-based sensor (see Paragraph 5 lines 5-12 teaching that the system may access two-dimensional data provided by a vehicle camera, and three-dimensional data provided by vehicle LiDAR equipment);
Processing the first set of data to correspond to a common format for detection (see Figure 1A, Paragraph 5 lines 1-4 teaching that the camera data is formatted to be compatible with the LiDAR data);
Processing the second set of data to correspond to the common format for detection (see Figure 1A, Paragraph 5 lines 1-4 teaching that the LiDAR data is formatted to be compatible with the camera data);
Combining the processed first set of data and the processed second set of data to form a common set of data (see Paragraph 5 lines 5-12 teaching that the two data sets are combined);
Processing the combined set of data (see Paragraph 5 lines 5-12 teaching that the combined data set is processed); and
Training a machine learning model for vision-based detection system based on the processing of the combined set of data (see Paragraph 4 lines 4-6, Paragraph 5 lines 5-12, Paragraph 9, Paragraph 21 lines 1-6 teaching that the system can train a machine learning construct, using 2D images and associated LiDAR cloud points which are combined and correlated by a processor, to allow for efficient image processing).
RoyChowdhury fails to teach wherein the first and second sets of data correspond to a common timestamp.
However, Mao teaches wherein the first and second sets of data correspond to a common timestamp (see Paragraph 2 lines 7-16 teaching a vehicle object classification system that matches LiDAR and camera data using correlating timestamps).
RoyChowdhury and Mao are both considered to be analogous to the claimed invention because they are in the same field of vehicle systems that fuse sensory data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified RoyChowdhury’s invention to include a feature that uses timestamps to combine LiDAR and camera data as taught by Mao. Doing so could improve a vehicle sensory system by associating multiple data types with specific time instances. This could improve the accuracy of the detection system, as multiple sensors could be used to validate the features and locations of detected objects. This could improve the safety and reliability of the detection system.
RoyChowdhury further fails to teach wherein the trained machine learning model is configured to generate an output using only an input of data corresponding to the vision-based detection system; and
Automatically labeling the first set of data using the second set of data as ground truth, without human intervention.
However, Postnikov teaches wherein the trained machine learning model is configured to generate an output using only an input of data corresponding to the vision-based detection system (see Abstract lines 1-4, Paragraph 10 lines 1-10, Paragraph 12 lines 1-2, Paragraph 88 lines 4-6 teaching a vehicle road detection training system which uses a combination of LiDAR and camera data to generate a training dataset for a machine learning algorithm, wherein once the algorithm is trained, it may be used on many vehicles that have only a camera, without requiring a LiDAR system); and
Automatically labeling the first set of data using the second set of data as ground truth, without human intervention (see Paragraph 67 lines 7-10, Paragraph 92 lines 1-7 teaching that the process can automatically use collected data to generate a training dataset and that “ground truth”/”correct” results can be used to label training data).
Here, the Examiner is interpreting the descriptor “automatically” as sufficiently teaching a process that occurs “automatically . . . without human intervention.”
RoyChowdhury and Postnikov are both considered to be analogous to the claimed invention because they are in the same field of vehicle systems that use both camera and LiDAR sensors to detect conditions external to a host vehicle. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified RoyChowdhury’s invention to include an automatic feature wherein LiDAR data can be used as ground truth data, to then label training data, which the vehicle can then operate on without LiDAR data, as taught by Postnikov. Doing so could improve a vehicle sensor system, by eliminating the need to have both types of sensors installed and functioning on a given vehicle in order to achieve the desired output, and by enabling the system to operate automatically, without human intervention.
In regard to Claim 2, RoyChowdhury fails to teach wherein the second set of data corresponds to characterization of moving objects, and wherein the characterization includes at least one of velocity, acceleration, or direction of the moving objects.
However, Mao teaches wherein the second set of data corresponds to characterization of moving objects, and wherein the characterization includes at least one of velocity, acceleration, or direction of the moving objects (see Paragraph 47 lines 1-16 teaching that objects detected by the LIDAR system may be assigned labels correlating to features, such as speed, acceleration, or heading).
RoyChowdhury and Mao are both considered to be analogous to the claimed invention because they are in the same field of vehicle systems that fuse sensory data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified RoyChowdhury’s invention to include a feature that recognizes features of detected object movement, such as direction, speed, and acceleration, as taught by Mao. Doing so could improve a vehicle sensory system by obtaining information correlating to where objects are, but also to where they might be in the future, based on their current movement. This could help operators or autonomous systems determine a driving operation to avoid the detected obstacle.
In regard to Claim 3, RoyChowdhury fails to teach wherein each set of combined first and second sets of data has the common timestamp.
However, Mao teaches wherein each set of combined first and second sets of data has the common timestamp (see Paragraph 2 lines 7-16 teaching a vehicle object classification system that matches LiDAR and camera data using correlating timestamps).
RoyChowdhury and Mao are both considered to be analogous to the claimed invention because they are in the same field of vehicle systems that fuse sensory data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified RoyChowdhury’s invention to include a feature that uses timestamps to combine LiDAR and camera data as taught by Mao. Doing so could improve a vehicle sensory system by associating multiple data types with specific time instances.
In regard to Claim 5, RoyChowdhury further teaches wherein processing the second set of data includes identifying a set number of attributes for each detected object (see Abstract lines 8-10, Paragraph 28 lines 23-27 teaching that the LiDAR data is used to identify regions in three-dimensional space, and identify attributes of the regions, such as road damage).
In regard to Claim 7, RoyChowdhury fails to teach wherein the trained machine learning model is transmitted to the vehicle.
However, Mao teaches wherein the trained machine learning model is transmitted to the vehicle (see Paragraph 55 lines 1-6 teaching that the model may be trained offline, and then sent to a vehicle via a network).
RoyChowdhury and Mao are both considered to be analogous to the claimed invention because they are in the same field of vehicle systems that fuse sensory data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified RoyChowdhury’s invention to include a feature that sends the learning model to a host vehicle as taught by Mao. Doing so could improve a vehicle sensory system by using superior computing power at a remote location to train a learning model, and then sending the model to the vehicle afterword, eliminating the need for training software and hardware at the vehicle itself.
In regard to Claim 8, RoyChowdhury further teaches a system comprising one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to generate a set of machine learning model training data, wherein the system is included in a network service, and generates training data (see Paragraph 9, Paragraph 21 lines 1-6 teaching that the system contains a computer-readable storage medium comprising processor-executable instructions that, when executed, cause the device to use a trainable network architecture to generate truth data).
The rest of Claim 8 is substantially similar to Claim 1 (the bulk of both claims). Please see the rejection of Claim 1 above for analysis.
Claim 9 is substantially similar to Claim 2 (the bulk of both claims). Please see the rejection of Claim 2 above for analysis.
Claim 10 is substantially similar to Claim 3 (the bulk of both claims). Please see the rejection of Claim 3 above for analysis.
Claim 12 is substantially similar to Claim 5 (the bulk of both claims). Please see the rejection of Claim 5 above for analysis.
Claim 14 is substantially similar to Claim 7 (the bulk of both claims). Please see the rejection of Claim 7 above for analysis.
In regard to Claim 15, RoyChowdhury further teaches a non-transitory computer storage media storing instructions that when executed by a system of one or more processors which are included in an autonomous or semi-autonomous vehicle, cause the system to perform operations (see Figure 1B, Paragraph 9, Paragraph 47 lines 23-27 teaching that the system contains a computer-readable storage medium comprising processor-executable instructions that, when executed, cause the device to send information to an autonomous driving application 152).
The rest of Claim 15 is substantially similar to Claim 1 (the bulk of both claims). Please see the rejection of Claim 1 above for analysis.
Claim 16 is substantially similar to Claim 2 (the bulk of both claims). Please see the rejection of Claim 2 above for analysis.
Claim 17 is substantially similar to Claim 3 (the bulk of both claims). Please see the rejection of Claim 3 above for analysis.
Claim 19 is substantially similar to Claim 5 (the bulk of both claims). Please see the rejection of Claim 5 above for analysis.
In regard to Claim 21, RoyChowdhury further teaches wherein the output corresponds to a velocity or a distance of a detected object (see Paragraph 82 lines 11-14 teaching that the system outputs a message indicating to the user that a pothole is oncoming, and provides a distance to the pothole).
In regard to Claim 22, RoyChowdhury fails to teach wherein the output is based on a ground truth that uses both the vision-based detection system and the detection system including the non-vision-based sensor.
However, Postnikov teaches wherein the output is based on a ground truth that uses both the vision-based detection system and the detection system including the non-vision-based sensor (see Figures 4, 6, 9, Paragraph 10 lines 1-10, Paragraph 12 lines 1-2, Paragraph 92 teaching that odometry data obtained from a LiDAR sensor is combined with image data to create training datasets, which include ground truth data, and are used to train the algorithm, which in turn is used to generate outputs, such as self-driving).
RoyChowdhury and Postnikov are both considered to be analogous to the claimed invention because they are in the same field of vehicle systems that use both camera and LiDAR sensors to detect conditions external to a host vehicle. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified RoyChowdhury’s invention to include a feature wherein the data from both sensors contribute to a ground truth that is capable of producing an output as taught by Postnikov. Doing so could improve a vehicle sensor system, by eliminating the need to have both types of sensors installed and functioning on a given vehicle in order to achieve the desired output, since ground truth data could be accessed by the vehicle.
Claim 23 is substantially similar to Claim 21 (the bulk of both claims). Please see the rejection of Claim 21 above for analysis.
Claim 24 is substantially similar to Claim 22 (the bulk of both claims). Please see the rejection of Claim 22 above for analysis.
Claim 25 is substantially similar to Claim 21 (the bulk of both claims). Please see the rejection of Claim 21 above for analysis.
Claim 26 is substantially similar to Claim 22 (the bulk of both claims). Please see the rejection of Claim 22 above for analysis.
Claims 4, 11, 18 are rejected under 35 U.S.C. 103 as being unpatentable over RoyChowdhury (U.S. Patent Publication 2022/0044034 A1) in view of Mao (U.S. Patent Publication 2020/0125112 A1), in further view of Postnikov (U.S. Patent Publication 2022/0105947 A1), in further view of Tu (U.S. Patent Publication 2021/0303922 A1).
In regard to Claim 4, RoyChowdhury fails to teach wherein processing the first set of data includes generating representations of detected objects included in the first set of data via bounding boxes and three-dimensional positions.
However, Tu teaches wherein processing the first set of data includes generating representations of detected objects included in the first set of data via bounding boxes and three-dimensional positions (see Paragraph 90 lines 14-18 teaching an object detection training model which obtains data correlating to objects seen by a front camera and labeled with three-dimensional bounding boxes).
RoyChowdhury and Tu are both considered to be analogous to the claimed invention because they are in the same field of systems that fuse sensory data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified RoyChowdhury’s invention to include a feature that uses three-dimensional bounding boxes for image data as taught by Tu. Doing so could improve a sensory system by furthering its capability to detect and classify objects, as well as their positions.
Claims 11, 18 are substantially similar to Claim 4 (the bulk of all claims). Please see the rejection of Claim 4 above for analysis.
Claims 6, 13, 20 are rejected under 35 U.S.C. 103 as being unpatentable over RoyChowdhury (U.S. Patent Publication 2022/0044034 A1) in view of Mao (U.S. Patent Publication 2020/0125112 A1), in further view of Postnikov (U.S. Patent Publication 2022/0105947 A1), in further view of Francis (U.S. Patent 8,874,266 B1).
In regard to Claim 6, RoyChowdhury fails to teach wherein processing the combined set of data uses at least one of smoothing, extrapolation of missing information, applying kinetic models, and applying confidence values technique.
However, Francis teaches wherein processing the combined set of data uses at least one of smoothing, extrapolation of missing information, applying kinetic models, and applying confidence values technique (see Column 12 lines 3-9 teaching a sensor data enhancement system that extrapolates different types of data, such as audio and image data, to a common time or period).
RoyChowdhury and Francis are both considered to be analogous to the claimed invention because they are in the same field of systems that fuse sensory data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified RoyChowdhury’s invention to include a feature that uses extrapolation of different types of data as taught by Francis. Doing so could improve a sensory system by furthering its capability to detect and classify objects.
Claims 13, 20 are substantially similar to Claim 6 (the bulk of all claims). Please see the rejection of Claim 6 above for analysis.
Response to Arguments
The Applicant’s arguments and remarks with regard to the 35 USC § 103 rejection of Claim 1 has been fully considered, but is unpersuasive.
The Applicant argues that the amended limitation within the independent claims, wherein the method “automatically label[s] the first set of data using the second set of data as ground truth, based on the common timestamp, without human intervention,” is not taught by any of the previous references used to reject Claim 1. The Examiner disagrees. As stated above in the rejection of Claim 1, Mao teaches wherein the first and second sets of data correspond to a common timestamp (see Paragraph 2 lines 7-16 teaching a vehicle object classification system that matches LiDAR and camera data using correlating timestamps).
RoyChowdhury and Mao are both considered to be analogous to the claimed invention because they are in the same field of vehicle systems that fuse sensory data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified RoyChowdhury’s invention to include a feature that uses timestamps to combine LiDAR and camera data as taught by Mao. Doing so could improve a vehicle sensory system by associating multiple data types with specific time instances. This could improve the accuracy of the detection system, as multiple sensors could be used to validate the features and locations of detected objects. This could improve the safety and reliability of the detection system.
Furthermore, Postnikov teaches automatically labeling the first set of data using the second set of data as ground truth, without human intervention (see Paragraph 67 lines 7-10, Paragraph 92 lines 1-7 teaching that the process can automatically use collected data to generate a training dataset and that “ground truth”/”correct” results can be used to label training data).
RoyChowdhury and Postnikov are both considered to be analogous to the claimed invention because they are in the same field of vehicle systems that use both camera and LiDAR sensors to detect conditions external to a host vehicle. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified RoyChowdhury’s invention to include an automatic feature wherein LiDAR data can be used as ground truth data, to then label training data, which the vehicle can then operate on without LiDAR data, as taught by Postnikov. Doing so could improve a vehicle sensor system, by eliminating the need to have both types of sensors installed and functioning on a given vehicle in order to achieve the desired output, and by enabling the system to operate automatically, without human intervention.
All claim objections have been withdrawn in light of the amendments.
Claims 2-26 remain rejected for reasons similar to those used to reject Claim 1, or under the rationales provided in the previous office action.
Conclusion
The prior art made of record and not relied upon is considered pertinent to the Applicant's disclosure:
Henel (U.S. Patent Publication 20170015327 A1) teaches a vehicle hill start assist system that combines radar or LiDAR data with vehicle speed data, through extrapolation, to create an environment profile for the vehicle’s axles (see Claim 13).
Pesterev (U.S. Patent Publication 20080208454 A1) teaches a vehicle path approximation system that combines measurements of inertial sensors with GNSS measured position and velocity for smoothing of vehicle position and velocity determinations through data integration and processing (see Paragraph 15 lines 20-25).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL W ARELLANO whose telephone number is (571)270-0102. The examiner can normally be reached M-F 7:30-4:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, the Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vivek Koppikar can be reached on (571) 272-5109. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (in USA or Canada) or 571-272-1000.
/PAUL W ARELLANO/Examiner, Art Unit 3667B
/VIVEK D KOPPIKAR/Supervisory Patent Examiner, Art Unit 3667