Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The lengthy specification has not been checked to the extent necessary to
determine the presence of all possible minor errors. Applicant's cooperation is
requested in correcting any errors of which applicant may become aware in the
specification.
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. The entire reference is considered to provide disclosure relating to the claimed invention. The claims & only the claims form the metes & bounds of the invention. Office personnel are to give the claims their broadest reasonable interpretation in light of the supporting disclosure. Unclaimed limitations appearing in the specification are not read into the claim. Prior art was referenced using terminology familiar to one of ordinary skill in the art. Such an approach is broad in concept and can be either explicit or implicit in meaning. Examiner's Notes are provided with the cited references to assist the applicant to better understand how the examiner interprets the applied prior art. Such comments are entirely consistent with the intent & spirit of compact prosecution.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-4, 7-13, 16-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Dolan et al (US 2021/0365610 A1) hereinafter Dolan.
Claim 1. A computer-implemented method comprising:
Dolan discloses accessing road data generated in association with operation of an autonomous vehicle (AV) in a real-world environment;
Dolan: [0047-0048] “As described above, the vehicle 302 can send sensor data to the computing device(s) 334, via the network(s) 332. … In some examples, the vehicle 302 can send raw sensor data to the computing device(s) 334. In other examples, the vehicle 302 can send processed sensor data and/or representations of sensor data to the computing device(s) 334 (e.g., data output from the localization system 320, the perception system 322, the prediction system 324, and/or the planning system 326). In some examples, the vehicle 302 can send sensor data to the computing device(s) 334 at a particular frequency, after a lapse of a predetermined period of time, in near real-time, etc. The computing device(s) 334 can receive the sensor data (raw or processed) from the vehicle 302 and/or one or more data collection devices 336 (which can include other vehicles like vehicle 302), as well as data from one or more third-party sources and/or systems 338.”
Dolan discloses generating a simulation environment for running a software stack associated with controlling the AV based on the road data;
Dolan: [0049] “The simulation system 344 can generate simulated environments. In at least one example, the simulation system 344 can generate simulated environments via procedural generation (e.g., creating data algorithmically), as described above with reference to FIGS. 1-2G. …. In additional or alternative examples, the stored object footprint(s) data storage 356 and/or the surface detail data storage 358 can be stored remotely and accessible to the computing device(s) 334 and/or data stored therein can be provided to the computing device(s) 334 from a third-party source and/or system 338. In some examples, stored object data and/or surface texture data can be generated in near-real time.”
Dolan [0045] “… Additionally, the drive system(s) 314 can include a drive module controller which can receive and preprocess data from the sensor system(s) and to control operation of the various vehicle systems. In some examples, the drive module controller can include processor(s) and memory communicatively coupled with the processor(s). The memory can store one or more modules to perform various functionalities of the drive system(s) 314. Furthermore, the drive system(s) 314 also include communication connection(s) that enable communication by the respective drive module with other local or remote computing device(s).”
Examiner interpret “software stack” as “software module or program”.
Dolan discloses modifying a portion of the software stack to generate a modified software stack as part of a test of the modified software stack;
Dolan: [0107] “… accessing supplemental data associated with the real environment, wherein the supplemental data provides information associated with the real environment that is unavailable to the plurality of data collection devices; associating the supplemental data with the simulated environment to supplement the simulated environment as a modified supplemental environment; and outputting the modified simulated environment for at least one of testing, validating, or training an algorithm used by an autonomous robotic computing device for at least one of navigating, planning, or decision making.”
See [0027] for detail of modification of simulation environment which is corresponding to the modifying a portion of the software stack.
Dolan discloses running the modified software stack as part of the test using sensor data included as part of the road data as input in running the modified software stack in the simulation environment; and
Dolan [0055] “As described above, simulated environments can be useful for enhancing training, testing, and/or validating systems (e.g., one or more components of an AI stack) onboard an autonomous vehicle, such as vehicle 302. In at least one example, simulated environments can be useful for training data models where training data from real environments is insufficient (e.g., as is the case with rare objects, rare scenarios, etc.). In such examples, a resulting data model can be provisioned to, or accessible by, the vehicle 302, and the vehicle 302 can utilize the data model for classifying objects in real-time (e.g., while driving or otherwise operating in the real environment). That is, the perception system 322 can utilize the data model (trained based on simulated data associated with a simulated environment) onboard in near real-time to classify objects.”
Dolan discloses validating the test of the modified software stack based on the running of the modified software stack in the simulation environment in relation to the operation of the AV in the real- world environment.
Dolan [0014] “… That is, simulated environments resulting from generation techniques described herein can be used for testing, training, and validating systems onboard an autonomous vehicle to ensure such systems can operate autonomous vehicles safely when deployed in real environments. That is, simulated environments resulting from generation techniques described herein can be used for testing, training, and validating a planner system, which can be used by an autonomous vehicle to navigate the autonomous vehicle along a trajectory in a real environment. Thus, such training, testing, and validating enabled by techniques described herein can provide opportunities to ensure that autonomous vehicles can operate in real world environments safely. …” See [0057] for detail of validating step.
Regarding Claim 10, the same ground of rejection is made as discussed above for substantially similar rationale.
In addition, Claim 10 recites “A system comprising: one or more processors; and at least one computer-readable storage medium having stored therein instructions which, when executed by the one or more processors, cause the one or more processors to:”.
Dolan discloses a system comprising: one or more processors; and at least one computer-readable storage medium having stored therein instructions which, when executed by the one or more processors, cause the one or more processors to: on [0033] “The vehicle computing device(s) 304 can include processor(s) 316 and memory 318 communicatively coupled with the processor(s) 316. In the illustrated example, the memory 318 of the vehicle computing device(s) 304 stores a localization system 320, a perception system 322, a prediction system 324, a planning system 326, and one or more system controllers 328. Additionally, the memory 318 can include a storage 230, which can store map(s), model(s), etc.”
Regarding Claim 19, the same ground of rejection is made as discussed above for substantially similar rationale.
In addition, Claim 19 recites “A non-transitory computer-readable storage medium having stored therein instructions which, when executed by one or more processors, cause the one or more processors to:”.
Dolan discloses a non-transitory computer-readable storage medium having stored therein instructions which, when executed by one or more processors, cause the one or more processors to: on [0033] “The vehicle computing device(s) 304 can include processor(s) 316 and memory 318 communicatively coupled with the processor(s) 316. In the illustrated example, the memory 318 of the vehicle computing device(s) 304 stores a localization system 320, a perception system 322, a prediction system 324, a planning system 326, and one or more system controllers 328. Additionally, the memory 318 can include a storage 230, which can store map(s), model(s), etc.”
Claims 2 and 11
Dolan discloses wherein the test of the modified software stack is validated based on a position of the AV in the test in relation to a position of the AV in the real-world environment.
Dolan [0057] “…. For instance, in real environments, GPS sensors experience positional drifts and may, as a result, accumulate error. Accordingly, to validate a localization algorithm that is used for localizing the vehicle 302, the evaluating system 348 can use a simulated environment, where the pose of the vehicle 302 is known at various times (including at all times) and evaluate the sensor data associated with a corresponding real environment to validate the localization algorithm (e.g., by relying on simulated poses as position and/or orientation ground truth). In such an example, the sensor system(s) 306 can generate sensor data associated with the simulated environment and the sensor data can be analyzed by the perception system 322. An output of the perception system 322 (e.g., associated with a position in a real environment) can be validated in view of the sensor data associated with the corresponding position in the simulated environment. That is, the sensor data associated with a position in a simulated environment can serve as the ground truth for the corresponding position in the real environment. As an example, LIDAR data recorded in association with a simulated environment (e.g., where the pose of the vehicle 302 is known) can be compared to LIDAR data recorded in association with a corresponding position in a real environment and the localization algorithm can be updated as appropriate. Furthermore, simulated environments can be useful for validating RADAR or other sensors of the sensor system(s) 306. In some examples, simulated environments can offer ground truth data for calibrating sensors (e.g., of the sensor system(s) 106). …”
Claims 3 and 12
Dolan discloses wherein the test of the modified software stack is validated based on an orientation of the AV in the test in relation to an orientation of the AV in the real-world environment.
Dolan [0057] “…. For instance, in real environments, GPS sensors experience positional drifts and may, as a result, accumulate error. Accordingly, to validate a localization algorithm that is used for localizing the vehicle 302, the evaluating system 348 can use a simulated environment, where the pose of the vehicle 302 is known at various times (including at all times) and evaluate the sensor data associated with a corresponding real environment to validate the localization algorithm (e.g., by relying on simulated poses as position and/or orientation ground truth). In such an example, the sensor system(s) 306 can generate sensor data associated with the simulated environment and the sensor data can be analyzed by the perception system 322. An output of the perception system 322 (e.g., associated with a position in a real environment) can be validated in view of the sensor data associated with the corresponding position in the simulated environment. That is, the sensor data associated with a position in a simulated environment can serve as the ground truth for the corresponding position in the real environment. As an example, LIDAR data recorded in association with a simulated environment (e.g., where the pose of the vehicle 302 is known) can be compared to LIDAR data recorded in association with a corresponding position in a real environment and the localization algorithm can be updated as appropriate. Furthermore, simulated environments can be useful for validating RADAR or other sensors of the sensor system(s) 306. In some examples, simulated environments can offer ground truth data for calibrating sensors (e.g., of the sensor system(s) 106). …”
Claims 4 and 13
Dolan discloses wherein validating the test of the modified software stack further comprises: identifying a simulated path of the AV created by running the test of the modified software stack in the simulation environment;
identifying a real-world path of the AV in the operation of the AV in the real-world environment;
determining a quantification of an amount of divergence between the simulated path of the AV and the real-world path of the AV; and
validating the test of the modified software stack based on the quantification of the amount of divergence between the simulated path of the AV and the real-world path of the AV.
Dolan: [0057] “Furthermore, simulated environments can be useful for validating and/or updating a localization algorithm used by the localization system 320. [correspond to validating the test of the modified software stack] For instance, in real environments, GPS sensors experience positional drifts and may, as a result, accumulate error. Accordingly, to validate a localization algorithm that is used for localizing the vehicle 302, the evaluating system 348 can use a simulated environment, [correspond to the simulated path] where the pose of the vehicle 302 is known at various times (including at all times) and evaluate the sensor data associated with a corresponding real environment [correspond to real-world path] to validate the localization algorithm (e.g., by relying on simulated poses as position and/or orientation ground truth). In such an example, the sensor system(s) 306 can generate sensor data associated with the simulated environment and the sensor data can be analyzed by the perception system 322. An output of the perception system 322 (e.g., associated with a position in a real environment) can be validated in view of the sensor data associated with the corresponding position in the simulated environment. That is, the sensor data associated with a position in a simulated environment can serve as the ground truth for the corresponding position in the real environment. [correspond to determining a quantification of an amount of divergence] As an example, LIDAR data recorded in association with a simulated environment (e.g., where the pose of the vehicle 302 is known) can be compared to LIDAR data recorded in association with a corresponding position in a real environment and the localization algorithm can be updated as appropriate. Furthermore, simulated environments can be useful for validating RADAR or other sensors of the sensor system(s) 306. …”
Claims 7 and 16
Dolan discloses wherein validating the test of the modified software stack further comprising:
identifying a real-world perception output of the AV in the operation of the AV in the real-world environment; identifying a simulated perception output of the AV in the test of the modified software stack in the simulation environment; and
validating the test of the modified software stack based on a comparison of the real-world perception output and the simulated perception output.
Dolan [0057] “Furthermore, simulated environments can be useful for validating and/or updating a localization algorithm used by the localization system 320. For instance, in real environments, GPS sensors experience positional drifts and may, as a result, accumulate error. Accordingly, to validate a localization algorithm that is used for localizing the vehicle 302, the evaluating system 348 can use a simulated environment, where the pose of the vehicle 302 is known at various times (including at all times) and evaluate the sensor data associated with a corresponding real environment to validate the localization algorithm (e.g., by relying on simulated poses as position and/or orientation ground truth). In such an example, the sensor system(s) 306 can generate sensor data associated with the simulated environment and the sensor data can be analyzed by the perception system 322. An output of the perception system 322 (e.g., associated with a position in a real environment) can be validated in view of the sensor data associated with the corresponding position in the simulated environment. That is, the sensor data associated with a position in a simulated environment can serve as the ground truth for the corresponding position in the real environment. As an example, LIDAR data recorded in association with a simulated environment (e.g., where the pose of the vehicle 302 is known) can be compared to LIDAR data recorded in association with a corresponding position in a real environment and the localization algorithm can be updated as appropriate. Furthermore, simulated environments can be useful for validating RADAR or other sensors of the sensor system(s) 306. In some examples, simulated environments can offer ground truth data for calibrating sensors (e.g., of the sensor system(s) 106). Other examples include, but are not limited to validating rolling shutter in simulation, calibration (e.g., of one or more of intrinsics or extrinsics) of various sensors, and the like. As would be appreciated, the techniques described herein may be used in validation, calibration, training, etc. for various other systems, subsystems, etc.”
Claims 8 and 17
Dolan discloses wherein the real-world perception output and the simulated perception output are compared based on ray tracing.
Dolan: [0057] “… As an example, LIDAR data recorded in association with a simulated environment (e.g., where the pose of the vehicle 302 is known) can be compared to LIDAR data recorded in association with a corresponding position in a real environment and the localization algorithm can be updated as appropriate. Furthermore, simulated environments can be useful for validating RADAR or other sensors of the sensor system(s) 306. In some examples, simulated environments can offer ground truth data for calibrating sensors (e.g., of the sensor system(s) 106). ….”
It is known in the art that LIDAR (Light Detection and Ranging) is using laser light for measurement (i.e. ray tracing).
Claims 9 and 18
Dolan discloses further comprising facilitating implementation of the modified software stack in real-world operation of the AV based on whether the test of the modified software stack is validated.
Dolan: [0029] “FIG. 2F illustrates a non-limiting example of the resulting simulated environment 134, which can be for enhancing training, testing, and/or validating systems (e.g., one or more components of an AI stack) onboard an autonomous vehicle, as described herein. That is, in some examples, the resulting simulated environment 134 can be used for training, testing, and/or validating algorithm(s) used by onboard systems of an autonomous vehicle, which can be used by the autonomous vehicle to control the autonomous vehicle.” See [0057] for more detail.
Claim 20. The non-transitory computer-readable storage medium of claim 19, Dolan discloses wherein the test of the modified software stack is validated based on either or both a position and an orientation of the AV in the test in relation to either or both a position and an orientation of the AV in the real- world environment.
Dolan [0057] “…. For instance, in real environments, GPS sensors experience positional drifts and may, as a result, accumulate error. Accordingly, to validate a localization algorithm that is used for localizing the vehicle 302, the evaluating system 348 can use a simulated environment, where the pose of the vehicle 302 is known at various times (including at all times) and evaluate the sensor data associated with a corresponding real environment to validate the localization algorithm (e.g., by relying on simulated poses as position and/or orientation ground truth). In such an example, the sensor system(s) 306 can generate sensor data associated with the simulated environment and the sensor data can be analyzed by the perception system 322. An output of the perception system 322 (e.g., associated with a position in a real environment) can be validated in view of the sensor data associated with the corresponding position in the simulated environment. That is, the sensor data associated with a position in a simulated environment can serve as the ground truth for the corresponding position in the real environment. As an example, LIDAR data recorded in association with a simulated environment (e.g., where the pose of the vehicle 302 is known) can be compared to LIDAR data recorded in association with a corresponding position in a real environment and the localization algorithm can be updated as appropriate. Furthermore, simulated environments can be useful for validating RADAR or other sensors of the sensor system(s) 306. In some examples, simulated environments can offer ground truth data for calibrating sensors (e.g., of the sensor system(s) 106). …”
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 5-6 and 14-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dolan et al (US 2021/0365610 A1) hereinafter Dolan, in view of Chen et al (NPL: Mixed Test Environment-based Vehicle-in-the-loop Validation - A New Testing Approach for Autonomous Vehicles, 2020), hereinafter Chen.
Claims 5 and 14
wherein validating the test of the modified software stack further comprises:
Dolan does not appear to explicitly disclose the following limitations: -
identifying an intent of the test of the modified software stack in relation to the operation of the AV in the real-world environment;
identifying actual characteristics of the test of the modified software stack in relation to the operation of the AV in the real-world environment; and validating the test of the modified software stack based on a comparison of the intent of the test of the modified software stack and the actual characteristics of the test of the modified software stack in relation of the operation of the AV in the real-world environment.
However, Chen discloses identifying an intent of the test of the modified software stack in relation to the operation of the AV in the real-world environment;
identifying actual characteristics of the test of the modified software stack in relation to the operation of the AV in the real-world environment; and
Chen (page 1288) “unprotected left turn at an intersection” correspond to the intent of the test.
PNG
media_image1.png
544
472
media_image1.png
Greyscale
Chen discloses validating the test of the modified software stack based on a comparison of the intent of the test of the modified software stack and the actual characteristics of the test of the modified software stack in relation of the operation of the AV in the real-world environment.
Chen (page 1288)
PNG
media_image2.png
210
478
media_image2.png
Greyscale
Dolan and Chen are analogous art because they are from the “same field of endeavor” autonomous vehicle simulation.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Dolan and Chen before him or her, to modify the simulation of Dolan and the testing feature of Chen because this combination provides realistic drive safety test before actual road testing.
The suggestion/motivation for doing so would have been Chen (Abstract)
PNG
media_image3.png
458
480
media_image3.png
Greyscale
Therefore, it would have been obvious to combine Dolan and Chen to obtain the invention as specified in the instant claim(s).
Claims 6 and 15
wherein validating the test of the modified software stack further comprises:
Dolan does not appear to explicitly disclose the following limitations: -
identifying a real-world intent of the AV in the operation of the AV in the real-world environment;
identifying a simulated intent of the AV in the test of the modified software stack in the simulation environment; and validating the test of the modified software stack based on a comparison of the real-world intent of the AV and the simulated intent of the AV.
However, Chen discloses identifying a real-world intent of the AV in the operation of the AV in the real-world environment; identifying a simulated intent of the AV in the test of the modified software stack in the simulation environment; and
Chen (page 1288) “unprotected left turn at an intersection” correspond to the real-world / simulated intent of the AV.
PNG
media_image1.png
544
472
media_image1.png
Greyscale
Chen discloses validating the test of the modified software stack based on a comparison of the real-world intent of the AV and the simulated intent of the AV.
Chen (page 1288)
PNG
media_image2.png
210
478
media_image2.png
Greyscale
Dolan and Chen are analogous art because they are from the “same field of endeavor” autonomous vehicle simulation.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Dolan and Chen before him or her, to modify the simulation of Dolan and the testing feature of Chen because this combination provides realistic drive safety test before actual road testing.
The suggestion/motivation for doing so would have been Chen (Abstract)
PNG
media_image3.png
458
480
media_image3.png
Greyscale
Therefore, it would have been obvious to combine Dolan and Chen to obtain the invention as specified in the instant claim(s).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHUEN-MEEI GAN whose telephone number is (469)295-9127. The examiner can normally be reached Monday-Friday 9:00 am to 4:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rehana Perveen can be reached at 571-272-3676. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHUEN-MEEI GAN/Primary Examiner, Art Unit 2189
Prosecution Insights