DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/15/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 1 recites both “obtaining, for a machine learning model configured to perform a task of the automated driving system, an internal representation of the sensor data,” and “wherein the internal representation is formed by inputting the sensor data to the machine learning model and extracting the internal representation from said machine learning model” Examiner is unsure how the internal representation of sensor data can both be obtained for the machine learning model and formed by the machine learning model.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim 1-5, 7, 10-15 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Kale et al (Kale hereinafter US 12670705 B2)
As per claim 1
Kale teaches A computer-implemented method performed in a server (Figure 16D) the method comprising: obtaining sensor data pertaining to a scene of a surrounding environment of a vehicle equipped with an automated driving system (Figure 16C Figure 16B Figure 16A. Paragraph 207: In at least one embodiment, controller(s) 1636 provide signals for controlling one or more components and/or systems of vehicle 1600 in response to sensor data received from one or more sensors (e.g., sensor inputs). In at least one embodiment, sensor data may be received from, for example and without limitation…” Paragraph 278 :“ flash LIDAR may allow for highly accurate and distortion-free images of surroundings to be generated with every laser flash. In at least one embodiment, four flash LIDAR sensors may be deployed, one at each side of vehicle 1600”) obtaining, for a machine learning model configured to perform a task of the automated driving system, an internal representation of the sensor data (Figure 16D,” Paragraph 267 In at least one embodiment, vehicle may provide additional artificial intelligence functionality, such as by executing redundant and/or different neural networks, and may be used to train and/or update neural networks based at least in part on input (e.g., sensor data) from sensors of vehicle 1600” Paragraph 303 “ In at least one embodiment, server(s) 1678 may receive, over network(s) 1690 and from vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work.” ) wherein the internal representation is formed by inputting the sensor data to the machine learning model and extracting the internal representation from said machine learning model (Figure 16D, Paragraph 303 “neural networks 1692, updated neural networks 1692, and/or map information 1694, including, without limitation, information regarding traffic and road conditions. In at least one embodiment, updates to map information 1694 may include, without limitation, updates for HD map 1622, such as information regarding construction sites, potholes, detours, flooding, and/or other obstructions. In at least one embodiment, neural networks 1692, updated neural networks 1692, and/or map information 1694 may have resulted from new training and/or experiences represented in data received from any number of vehicles in environment, and/or based at least in part on training performed at a data center” Paragraph 304 “In at least one embodiment, server(s) 1678 may be used to train machine learning models (e.g., neural networks) based at least in part on training data. In at least one embodiment, training data may be generated by vehicles,” Kale takes the sensor data, updates a map and sends it to another neural network which is then sent to the server as depicted from the directional flow of labels 1600, 1694, 1692 and 1690 in figure 16D) generating synthetic sensor data for subsequent comparison with the obtained sensor data (Paragraph 210 “Inference and/or training logic 715 are used to perform inferencing and/or training operations associated with one or more embodiments. In at least one embodiment, inference and/or training logic 715 may be used in system FIG. 16A for inferencing or predicting operations based …Such components can be used to generate synthetic data imitating failure cases in a network training process, which can help to improve performance of the network while limiting the amount of synthetic data to avoid overfitting.” ) wherein the synthetic sensor data is generated by inputting the internal representation into a generative model trained to generate synthetic sensor data based on internal representations. (Figure 16C Figure 7A. Paragraph 172 “In at least one embodiment, output model(s) 1316 and/or pre-trained model(s) 1406 may include any types of machine learning models …may include machine learning model(s) using… deconvolutional, generative adversarial” Paragraph 210 “Inference and/or training logic 715 are used to perform inferencing and/or training operations associated with one or more embodiments. In at least one embodiment, inference and/or training logic 715 may be used in system FIG. 16A for inferencing or predicting operations based …Such components can be used to generate synthetic data imitating failure cases in a network training process, which can help to improve performance of the network while limiting the amount of synthetic data to avoid overfitting Paragraph 267 “In at least one embodiment, vehicle 1600 may include GPU(s) 1620 (e.g., discrete GPU(s), or dGPU(s)), that may be coupled to SoC(s) …may provide additional artificial intelligence functionality, such as by executing redundant and/or different neural networks, and may be used to train and/or update neural networks based at least in part on input (e.g., sensor data) from sensors of vehicle.” Figure 16C shows the sensor systems being connected to GPU 1620 and further SoC’s which handle machine learning tasks. This would include the neural networks that generate the synthetic components.)
As per claim 2
Kale discloses all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection .
Kale teaches wherein the internal representation of the sensor data is obtained in response to identifying the scene as a scene of interest. (Figure 4A Figure 4B and Figure 16D, Paragraph 54 “In some embodiments, the object data 108 determined based on the one or more obtained outputs may correspond to one or more objects detected in the given input image 106 (or the modified input image). For example, object data 108 may include region of interest (ROI) data associated with input image 106 (or the modified input image).” Paragraph 73 “The computing device may execute an application which detects an activity (e.g., based on data received from one or more sensors of or coupled to the autonomous vehicle or the smart surveillance camera), receives an image generated in response to the detected motion (e.g., by an audiovisual component of or coupled to the autonomous vehicle or the smart surveillance camera), and performs object detection and/or classification based on the generated image (e.g., using trained object detection model 222)… in a process pipeline associated with such computing device, in accordance with embodiments described with respect to FIGS. 4A-4B” Paragraph 74 “the application may be an object detection and/or classification application that is configured to perform object detection and/or classification based on one or more images generated by a camera at or coupled to an autonomous vehicle… In response to the one or more sensors detecting a motion or activity within an environment, the camera at or coupled to computing device 102 may generate an image (e.g., image 106) depicting the environment and, in some embodiments, one or more objects within the environment. … may provide image 106 to be used as input to a trained (or retrained) object detection model 222, as described above…. may determine a class of the object and/or other data associated with the object based on one or more outputs of model 222. Paragraph 216 “In at least one embodiment, cameras with a field of view that include portions of environment in front of vehicle 1600 (e.g., front-facing cameras) may be used for surround view, to help identify forward facing paths and obstacles, as well as aid in, with help of one or more of controllers “ )
As per claim 3
Kale discloses all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection .
Kale wherein the scene, to which the obtained sensor data pertains, is a scene of interest. (Paragraph 73 “The computing device may execute an application which detects an activity (e.g., based on data received from one or more sensors of or coupled to the autonomous vehicle or the smart surveillance camera), receives an image generated in response to the detected motion (e.g., by an audiovisual component of or coupled to the autonomous vehicle or the smart surveillance camera), and performs object detection and/or classification based on the generated image (e.g., using trained object detection model 222)… in a process pipeline associated with such computing device, in accordance with embodiments described with respect to FIGS. 4A-4B” Paragraph 74 “In response to the one or more sensors detecting a motion or activity within an environment, the camera at or coupled to computing device 102 may generate an image “ A scene where motion is being detected is a scene of interest particularly in regards to motion sensing cameras within an autonomous vehicle)
As per claim 4
Kale discloses all claim limitations previously rejected in claim 2’s 102 rejection. See claim 2’s 102 rejection .
Kale teaches wherein the scene is identified as the scene of interest based on a detected deviating behavior of the vehicle at the scene.( Figure 4A Figure 4B and Figure 16D, Paragraph 54 “In some embodiments, the object data 108 determined based on the one or more obtained outputs may correspond to one or more objects detected in the given input image 106 (or the modified input image). For example, object data 108 may include region of interest (ROI) data associated with input image 106 (or the modified input image. Paragraph 296 output of ADAS system 1638 may be fed into primary computer's perception block and/or primary computer's dynamic driving task block. For example, in at least one embodiment, if ADAS system 1638 indicates a forward crash warning due to an object immediately ahead, perception block may use this information when identifying objects. In at least one embodiment, secondary computer may have its own neural network which is trained and thus reduces risk of false positives, as described herein. Furthermore Paragraph 303 “server(s) 1678 may receive, over network(s) 1690 and from vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work.” Deviating behavior here is driving down an unexpected road conditions)
As per claim 5
Kale discloses all claim limitations previously rejected in claim 4’s 102 rejection. See claim 4’s 102 rejection.
Kale teaches wherein the deviating behavior is one of an activation of an emergency braking system, an evasive maneuver performed by a driver, an evasive maneuver performed by the automated driving system, a driving incident, or a notification from an occupant of the vehicle. (Paragraph (306) “For example, in at least one embodiment, deep-learning infrastructure may receive periodic updates from vehicle 1600, such as a sequence of images and/or objects that vehicle 1600 has located in that sequence of images (e.g., via computer vision and/or other machine learning object classification techniques). In at least one embodiment, deep-learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 1600 and, if results do not match and deep-learning infrastructure concludes that AI in vehicle 1600 is malfunctioning, then server(s)” Kale is showing that when an image is sent to the deep learning infrastructure of server 1678, the server will run its own neural network to compare the image of the scene taken by the vehicle. The “driving incident” here is the vehicle malfunctioning and or inquisition of being in state of malfunction. Furthermore Paragraph 296 “. For example, in at least one embodiment, if ADAS system 1638 indicates a forward crash warning due to an object immediately ahead, perception block may use this information when identifying objects” Here Kale shows that when the ADAS system notices a potential object it sends this to the perception block which then identifies the object within the scene of interest. Here the driving incident which indicates a deviating behavior is the ADAS indication of potential hazardous object.)
As per claim 7
Kale discloses all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection .
Kale teaches wherein the sensor data comprises one or more of image data, LIDAR data, radar data or ultrasonic data. (Figure 16C )
As per claim 10
Kale discloses all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Kale teaches wherein the generative model is a generative adversarial network or a diffusion model. (Figure 14, Paragraph 172 “ output models…may include …generative adversarial,… and/or other types of machine learning models. In paragraph 211 Kale states the components of the inference logic used in autonomous vehicle 1600 generates synthetic data used to )
As per claim 11
Kale discloses all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Claim 11 is the non-transitory computer readable storage medium storing instructions of claim 1 and will be rejected under the same premise.
As per claim 12
Claim 12 is the server claim where claim 1’s methodology is performed. Server claim 12 parallels method claim 1. Kale’s method utilizes a server for the methodology of claim 1 (See figure 16C and Figure 16D) It will be rejected under the same premise
As per claim 13
Kale teaches A computer-implemented method, performed by a vehicle equipped with an automated driving system (Figure 16A-16C) the method comprising: in response to detecting a deviating behavior of the vehicle, obtaining sensor data pertaining to a scene of a surrounding environment at which the deviating behavior was detected ( Paragraph 303 “server(s) 1678 may receive, over network(s) 1690 and from vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work.” Deviating behavior here is driving down an unexpected road conditions) determining, for a machine learning model configured to perform a task of the automated driving system, an internal representation of the sensor data (Paragraph 267 “In at least one embodiment, vehicle may provide additional artificial intelligence functionality, such as by executing redundant and/or different neural networks, and may be used to train and/or update neural networks based at least in part on input (e.g., sensor data) from sensors of vehicle 1600” Paragraph 303 “ In at least one embodiment, server(s) 1678 may receive, over network(s) 1690 and from vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work.”) by inputting the sensor data to the machine learning model and extracting the internal representation from said machine learning model (Paragraph 296 “In at least one embodiment, output of ADAS system 1638 may be fed into primary computer's perception block and/or primary computer's dynamic driving task block. For example, in at least one embodiment, if ADAS system 1638 indicates a forward crash warning due to an object immediately ahead, perception block may use this information when identifying objects. In at least one embodiment, secondary computer may have its own neural network which is trained and thus reduces risk of false positives, as described herein.” ADAS data is input into a perception block and the vehicles identification/labeling of the object is extracted internal representation. Furthermore Paragraph 303 ‘neural networks 1692, updated neural networks 1692, and/or map information 1694, including, without limitation, information regarding traffic and road conditions. In at least one embodiment, updates to map information 1694 may include, without limitation, updates for HD map 1622, such as information regarding construction sites, potholes, detours, flooding, and/or other obstructions. In at least one embodiment, neural networks 1692, updated neural networks 1692, and/or map information 1694 may have resulted from new training and/or experiences represented in data received from any number of vehicles in environment, and/or based at least in part on training performed at a data center” Kale takes the sensor data, updates a map and sends it to another neural network which is then sent to the server as depicted from the directional flow of labels 1600, 1694, 1692 and 1690 in figure 16D) transmitting the sensor data and the internal representation of the sensor data to a server for subsequent generation of synthetic sensor data based on the internal representation. (Figure 16D, Figure 2. Paragraph 268 “network interface 1624 may be used to enable wireless connectivity over Internet with cloud (e.g., with server(s) and/or other network devices), with other vehicles, and/or with computing devices “ paragraph 300 “ Inference and/or training logic 715 are used to perform inferencing and/or training operations associated with one or more embodiments. Details regarding inference and/or training logic 715 are provided below. In at least one embodiment, inference and/or training logic 715 may be used in system FIG. 16C for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and/or architectures, or neural network use cases described herein…components can be used to generate synthetic data “ Figure 2 shows that object data (Sensor data) and the image (internal representation) held in memory 220 are passed to the inference module. Kale states that the inference logic which is a neural network, creates synthetic data. )
As per claim 14
Kale teaches all claim limitations previously rejected in claim 13’s 102 rejection. See claim 13’s 102 rejection.
Claim 14 is the parallel a non-transitory computer readable storage medium of claim 13 and will be rejected under the same premise.
As per claim 15
Claim 15 is the parallel device claim (oriented towards a vehicle equipped with an automated driving system) of method claim 13. See Kale’s figure 16A 16B and 16C for vehicle sensor architecture.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6, 8 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Kale et al (Kale hereinafter US 12670705 B2) in view of Nguyen et al (Nguyen hereinafter “Out-of-Distribution Detection for LiDAR-based 3D Object Detection”)
As per claim 6
Kale discloses all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection .
Kale does not teach identifying one or more discrepancies between the obtained sensor data and the generated synthetic sensor data based on a comparison thereof.
Nguyen teaches identifying one or more discrepancies between the obtained sensor data and the generated synthetic sensor data based on a comparison thereof. (Table 3.2, 3.3 OOD Dataset description: “For example, in Carla OOD datasets, bench and swing couch have high injection failures because they are large objects. Thus, they easily overlap with existing objects in a frame. These objects also have high detection failures because a model cannot easily detect them. Furthermore, by comparing the statistics among datasets, we can conclude that it is easier to insert Waymo and KITTI FP objects than to insert Carla objects. It makes sense because LiDAR sensors in both Waymo and KITTI data sets have a similar specification. In contrast ,the Carla objects are synthetic data from a simulation.” Conclusion: “We evaluate the OOD detection methods on the KITTI dataset augmented with a diverse set of real and synthetic OOD objects, revealing a nuanced landscape of how the current OOD detection methods perform in the context of LiDAR-based 3D object detection. The results demonstrate that each method is biased toward detecting certain types of OOD objects” The identified discrepancies Huang finds are the likelihood and accuracy of obstacle determination and in particular “out of distribution” 3D objects .)
Accordingly, a person of ordinary skill in the art would have been motivated to modify Kale’s pipeline with Huang’s concept of comparing synthetic data to sensor data to identify one or more discrepancies in object detection. Kale states that “components can be used to generate synthetic data imitating failure cases in a network training process, which can help to improve performance of the network while limiting the amount of synthetic data to avoid overfitting.” The components being the inference logic 715 presented in paragraph 300. A person or ordinary skill in the art is aware that models that models that are fed OOD data blindly often give overconfident incorrect predictions. A person of ordinary skill in the art understands that actively contrasting real world sensor data against synthetic data during Kale’s inference logic allows the vehicle to; detect novel anomalies by actively comparing live data to template synthetic data to flag deviations, prevent overconfidence by evaluating synthetic baselines vs new real world unfamiliar settings rather than making blind overconfident decisions caused by overfitting and assess safety boundaries by comparing live data against simulated scenarios. Nguyen states that “deep models are notorious for assigning high confidence scores to out-of-distribution (OOD) inputs, that is, inputs that are not drawn from the training distribution.” and that their method investigates and attempts to remedy this.
As per claim 8
Kale discloses all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Kale teaches in response to receiving an indication of a discrepancy between the obtained sensor data and the synthetic sensor data: assigning annotation data to the sensor data; and storing the sensor data with assigned annotation data for subsequent training of the machine learning model. (Figure 14, Figure 15A Figure 15B)
Within the Kale/Nguyen system a person of ordinary skill in the art would have found it obvious to follow Kale’s pipeline in the stated figures in response to receiving an indication of discrepancy between synthetic and sensor data as taught by Nguyen. A person of ordinary skill in the art would have found it obvious to apply it to Kale’s inference/model training pipeline shown in figure 4A and figure 4B.
As per claim 9
Kale discloses all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection .
Kale teaches in response to receiving an indication of a discrepancy between the obtained sensor data and the synthetic sensor data: obtaining additional sensor data pertaining to the scene of the physical environment, and storing the additional sensor data for subsequent training of the machine learning model. (Figure 4A and 4B Paragraph 73 “ The computing device may execute an application which detects an activity (e.g., based on data received from one or more sensors of or coupled to the autonomous vehicle or the smart surveillance camera), receives an image generated in response to the detected motion (e.g., by an audiovisual component of or coupled to the autonomous vehicle or the smart surveillance camera), and performs object detection and/or classification based on the generated image (e.g., using trained object detection model 222). In some embodiments, scheduler module 326 may schedule a process to use training data 358 to retrain trained machine learning model 222 in a process pipeline associated with such computing device, in accordance with embodiments described with respect to FIGS. 4A-4B.” An “application which detects an activity” and an image generated because of this activity is the additional sensor data. A person of ordinary skill in the art would have found it obvious to add the discrepancy found by comparison in Nguyens pipeline within the inferencing of figures 4A and 4B and aforementioned disclosure of Kale. )
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE WRENSFORD CODRINGTON whose telephone number is (571)272-8130. The examiner can normally be reached 8:00am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHANE WRENSFORD CODRINGTON/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights