Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The Office Action is in response to the application filed on 02/27/2023. Claims 1-20 are pending in the application. Claims 1, 8 and 15 are independent claims.
Specification
The abstract dated 02/27/2023 has been reviewed. It has 130 words and 9 lines and no legal
phraseology. It is accepted.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The claims 1-20 are rejected under 35 USC § 101 because the claimed invention is directed to
judicial exception, an abstract idea, it has not been integrated into practical application and the claims further do not recite significantly more than the judicial exception. Examiner has evaluated
the claims under the framework provided in the 2019 Revised Patent Subject Matter Eligibility Guidance
published in the Federal Register 01/07/2019, as well as subsequent USPTO eligibility guidance updates,
and has provided such analysis below.
Step 1: Are the claims to a process, machine, manufacture or composition of matter?"
Yes, Claims 1-7 are directed to system and fall within the statutory category of machine;
Yes, Claims 8-14 are directed to method and fall within the statutory category of process;
Yes, Claims 15-20 are directed to non-transitory computer-readable media and fall within the statutory category of manufacture.
In order to evaluate the Step 2A inquiry "Is the claim directed to a law of nature, a natural phenomenon or an abstract idea?" we must determine, at Step 2A Prong 1, whether the claim recites a law of nature, a natural phenomenon or an abstract idea and further whether the claim recites additional elements that integrate the judicial exception into a practical application.
Step 2A Prong 1:
Claim 1: The limitations of “identify, based on … the collection of data, a plurality of traffic scene datasets each corresponding to an instance in which the one or more autonomous vehicles encountered a traffic scene,” as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation (BRI) in light of the specification, covers performance of the limitation in the human mind. For example, a person is capable of observing/reviewing information collected during operation of one or more autonomous vehicles, identifying a plurality of traffic scenes or events from the collected information, wherein each of the plurality of traffic scenes or events corresponds to an instance or occurrence in which the one or more autonomous vehicle encountered a traffic scene or event. The steps include observation, evaluation, judgment, and reasoning processes that can be performed mentally or with the aid of pen and paper (The courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011)) – MPEP 2106.04(a)(2)(III).
Claim 1: The limitations of “determine, based on the plurality of traffic scene datasets, one or more metrics for characterizing an operation of the one or more autonomous vehicles in relation to the traffic scene,” as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation (BRI) in light of the specification, covers performance of the limitation in the human mind. For example, after identified a plurality of traffic scenes or events from the collected information, a person is capable of determining performance measures characterizing how the autonomous vehicles operated in relation to the instance or occurrence in which the one or more autonomous vehicle encountered the traffic scene or event. The steps include observation, evaluation, judgment, and reasoning processes that can be performed mentally or with the aid of pen and paper (The courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011)) – MPEP 2106.04(a)(2)(III).
Claim 1: The limitations of “determine, based on the one or more metrics, a first prediction of future performance of a fleet of autonomous vehicles in relation to the traffic scene, wherein the fleet of autonomous vehicles are configured to execute the first version of autonomous vehicle software,” as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation (BRI) in light of the specification, covers performance of the limitation in the human mind. For example, after determined performance measures characterizing how the autonomous vehicles operated in relation to the instance or occurrence in which the one or more autonomous vehicle encountered the traffic scene or event, a person is capable of estimating how a fleet of autonomous vehicles using the same version of autonomous vehicle software may perform in future encounters with the traffic scene or event. The steps include observation, evaluation, judgment, and reasoning processes that can be performed mentally or with the aid of pen and paper (The courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011)) – MPEP 2106.04(a)(2)(III).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under step 2A Prong I.
The elements of claims 8 and 15 are substantially the same as those of claim 1. Therefore, the elements of claims 8 and 15 are rejected due to the same reasons as outlined above for claim 1.
Therefore, claims 1, 8 and 15 recite judicial exceptions. The claims have been identified to recite judicial exceptions, Step 2A Prong 2 will evaluate whether the claim as a whole integrates the exception into a practical application of that exception.
Step 2A Prong 2: Claims 1, 8 and 15: The judicial exception is not integrated into a practical application.
In particular, the claims recite the following additional elements – “A system comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: " and “at least one traffic scene selector” and “A non-transitory computer-readable media comprising instructions stored thereon which, when executed are configured to cause a computer or processor to:,” which are merely recitation of instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to implement the judicial exception with the broad reasonable interpretation in light of specification, which does not integrate judicial exception into a practical application (see MPEP § 2106.05(f)).
Further, the following additional element – “receive a collection of data compiled by one or more autonomous vehicles while navigating a real-world environment, wherein the one or more autonomous vehicles are configured to execute a first version of autonomous vehicle software,” which is merely adding a recitation of insignificant extra-solution activities such as data gathering (i.e., receiving information associated with first version of autonomous vehicle software for subsequent analysis), and merely uses generic autonomous vehicles as a source of information, which does not integrate a judicial exception into practical application (see MPEP 2106.05(g)).
Therefore, "Do the claims recite additional elements that integrate the judicial exception into a practical application? No, these additional elements do not integrate the abstract idea into a practical application and they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
After having evaluated the inquires set forth in Steps 2A Prong 1 and 2, it has been concluded that claims 1, 8 and 15 not only recite a judicial exception but that the claims are directed to the judicial exception as the judicial exception has not been integrated into practical application.
Step 2B: Claims 1, 8 and 15: The claim does not include additional elements, alone or in combination, that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements amount to no more than generic computing components which do not amount to significantly more than the abstract idea. Limitations that the courts have found not to be enough to qualify as "significantly more" when recited in a claim with a judicial exception include: i. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, e.g., a limitation indicating that a particular function such as creating and maintaining electronic records is performed by a computer, as discussed in Alice Corp., 573 U.S. at 225-26, 110 USPQ2d at 1984 (see MPEP § 2106.05(f)); ii. Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry, as discussed in Alice Corp., 573 U.S. at 225, 110 USPQ2d at 1984 (see MPEP § 2106.05(d)); iii. Adding insignificant extra-solution activity to the judicial exception, e.g., mere data gathering in conjunction with a law of nature or abstract idea such as a step of obtaining information about credit card transactions so that the information can be analyzed by an abstract mental process, as discussed in CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011) (see MPEP § 2106.05(g)); …
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. i. Receiving or transmitting data over a network, …; ii. Performing repetitive calculations, … iii. Electronic recordkeeping, … (updating an activity log). iv. Storing and retrieving information in memory, …
The claimed memory, one or more processors, non-transitory computer-readable media, instructions, and traffic scene selector are recited at a generic function level and are used in their ordinary capacities to recited data, process data, identify selected information, determine metrics, and generate a prediction. The claim does not recite any unconventional arrangement of computer components, any particular technical implementation of the traffic scene selector, any improved computer functionality, or any specific control technique for improving operation of the autonomous vehicles, the functioning of a computer or other technology or technological field. Further, reciting the collection of data from autonomous vehicles merely adds insignificant extra-solution data gathering and uses the autonomous vehicles as a source of information for the abstract analysis. Thus, the additional limitations merely place generic computer component and generic data gathering activity to perform the abstract idea. Accordingly, the additional limitations, considered individually and in combination, do not provide significantly more than the judicial exception.
Therefore, "Do the claims recite additional elements that amount to significantly more than the judicial exception? No, these additional elements, alone or in combination, do not amount to significantly more than the judicial exception. Having concluded analysis within the provided framework, claims 1, 8 and 15 do not recite patent eligible subject matter under 35 U.S.C. § 101.
Dependent claims 2-7, 9-14, and 16-20 are also similar rejected under same rationale as cited above wherein these claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. These claims are merely further elaborate the mental process and/or mathematical concepts, or providing additional definition of process which does not impose any meaningful limits on practicing the abstract idea. Claims 2-7, 9-14, and 16-20 are also rejected for incorporating the deficiency of their independent claims 1, 8 and 15.
Claim 2 recites “The system of claim 1, wherein the one or more processors are further configured to:
perform one or more simulated tests of the first version of autonomous vehicle software using a simulation environment configured to implement one or more simulation test scenarios, wherein at least a portion of the one or more simulation test scenarios are based on one or more of the plurality of traffic scene datasets;
determine, based on the one or more simulated tests, one or more additional metrics for characterizing the operation of the first version of autonomous vehicle software in relation to the traffic scene; and
update, based on the one or more additional metrics, the first prediction of future performance of the fleet of autonomous vehicles in relation to the traffic scene. ”
These limitations merely extend the abstract idea of evaluating traffic scene information, determining performance measures, and estimating future autonomous vehicles performance into a simulation environment. For example, after estimated how a fleet of autonomous vehicles using the same version of autonomous vehicle software may perform in future encounters with the traffic scene or event, a person is capable of observing/reviewing simulated results, mentally determining additional performance measures characterizing how the first version of autonomous vehicle software operated in relation to one or more simulation test scenarios based on the one or more of the plurality of traffic scene scenes or events, and mentally changing/updating the prior estimation based on the additional measures. Thus, the steps still include observation, evaluation, judgment, and reasoning processes that can be performed mentally or with the aid of pen and paper.
The recitation of “perform one or more simulated tests of the first version … using a simulation environment configured to implement one or more simulation test scenarios, wherein at least a portion of the one or more simulation test scenarios are based on one or more of the plurality of traffic scene datasets,” merely links the use of the judicial exception to a particular technological environment (i.e., autonomous vehicle software simulation). See MPEP § 2106.05(h). The limitation recites at high level of generality, and does not recite any particular simulation architecture, sensor model, vehicle dynamic model, test execution technique, or technical improvement to the simulation environment or to the autonomous vehicles software. Rather, the simulation environment is used to obtain additional results that are then analyzed to determine additional metrics and update the prediction, which does not integrate judicial exception into a practical application and does not amount to significantly more than judicial exception. Therefore, the office finds that the claim 2 is ineligible under 35 USC 101.
Claim 3 recites “The system of claim 1, wherein the one or more processors are further configured to:
perform one or more simulated tests of a second version of autonomous vehicle software using a simulation environment configured to implement one or more simulation test scenarios, wherein at least a portion of the one or more simulation test scenarios are based on one or more of the plurality of traffic scene datasets;
determine, based on the one or more simulated tests, one or more revised metrics for characterizing the operation of the second version of autonomous vehicle software in relation to the traffic scene; and
determine, based on the one or more revised metrics, a second prediction of future performance of the fleet of autonomous vehicles in relation to the traffic scene, wherein the fleet of autonomous vehicles are configured to execute the second version of autonomous vehicle software.”
These limitations merely extend the abstract idea of evaluating traffic scene information, determining performance measures, and estimating future autonomous vehicles performance into a simulation environment. For example, after estimated how a fleet of autonomous vehicles using the same version of autonomous vehicle software may perform in future encounters with the traffic scene or event, a person is capable of observing/reviewing simulated results, mentally determining revised performance measures characterizing how a second version of autonomous vehicles software operated in relation to the traffic scene or event, and estimating how the fleet of autonomous vehicles using the second version of autonomous vehicle software may perform in future encounters with the traffic scene or event. Thus, the steps still include observation, evaluation, judgment, and reasoning processes that can be performed mentally or with the aid of pen and paper.
The recitation of “perform one or more simulated tests of a second version … using a simulation environment configured to implement one or more simulation test scenarios, wherein at least a portion of the one or more simulation test scenarios are based on one or more of the plurality of traffic scene datasets,” merely links the use of the judicial exception to a particular technological environment (i.e., autonomous vehicle software simulation). See MPEP § 2106.05(h). The limitation recites at high level of generality, and does not recite any particular simulation architecture, sensor model, vehicle dynamic model, test execution technique, or technical improvement to the simulation environment or to the autonomous vehicles software. Rather, the simulation environment is used to obtain additional results that are then analyzed to determine revised metrics and the second prediction of future performance of the fleet of autonomous vehicles in relation to the traffic scene, which does not integrate judicial exception into a practical application and does not amount to significantly more than judicial exception. Therefore, the office finds that the claim 3 is ineligible under 35 USC 101.
Claim 4 recites “The system of claim 1, wherein the at least one traffic scene selector includes at least one of an autonomous vehicle detector, a detector confidence level, and a scene descriptor.”
The limitation further defines traffic scene selector includes at least one of an autonomous vehicle detector, a detector confidence level, and a scene descriptor; therefore, it merely a recitation of instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to implement the judicial exception, which does not integrate judicial exception into a practical application (see MPEP § 2106.05(f)). Therefore, the office finds that the claim 4 is ineligible under 35 USC 101.
Claim 5 recites “The system of claim 4, wherein the scene descriptor includes at least one of an object type, an object size, an object action, and a map location.”
The limitation further defines the scene descriptor of traffic scene selector, includes at least one of an object type, an object size, an object action, and a map location; therefore, it merely a recitation of instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to implement the judicial exception, which does not integrate judicial exception into a practical application (see MPEP § 2106.05(f)). Therefore, the office finds that the claim 5 is ineligible under 35 USC 101.
Claim 6 recites “The system of claim 1, wherein the traffic scene corresponds to a temporary traffic scene, and wherein the temporary traffic scene includes at least one of a stopped school bus, a human controlling traffic, a road closure, a construction zone, a traffic redirection, a traffic blockage, and an emergency vehicle.”
The limitation further defines the traffic scene corresponds to a temporary traffic scene includes at least one of a stopped school bus, a human controlling traffic, a road closure, a construction zone, a traffic redirection, a traffic blockage, and an emergency vehicle.; therefore, it merely an extension of mental process (e.g., identifying a plurality of traffic scenes or events within the information corresponds to a temporary traffic scene or event). Therefore, the office finds that the claim 6 is ineligible under 35 USC 101.
Claim 7 recites “The system of claim 1, wherein the one or more metrics for characterizing the operation of the one or more autonomous vehicles in relation to the traffic scene include at least one of an exposure rate, a precision metric, and a recall metric.”
The limitation further defines the one or more metrics include at least one of an exposure rate, a precision metric, and a recall metrics; therefore, it merely an extension of mental process (e.g., how many times the traffic scene or event is identified corresponding to metrics). Therefore, the office finds that the claim 7 is ineligible under 35 USC 101.
Claims 9-14 and 16-20 recite the similar elements as claims 2-7, and are rejected for the same reasons under 35 U.S.C. 101.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and
103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for
the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 4-5, 7-8, 11-12, 14-15, 18, and 20 are rejected under 35 U.S.C. 102(a)(2) as being
anticipated by Nag US11886193B1.
Claim 1, Nag teaches A system comprising:
a memory; and
one or more processors coupled to the memory, the one or more processors being configured to (Fig.12, a computing device 1200; processor(s) 1205 and memory element(s) 1210 …):
receive a collection of data compiled by one or more autonomous vehicles while navigating a real-world environment, wherein the one or more autonomous vehicles are configured to execute a first version of autonomous vehicle software (Col.8, lines 31-42, “…at 410, in which the collection module 354 processor obtains data. Data may generally be collected from a vehicle as the vehicle drives, e.g., by an autonomy system as the autonomy system is tested in various controlled environments. Data may be collected as the autonomy system of the vehicle is tested in real-world environments … the data may be collected and stored in a memory on the vehicle, and then subsequently uploaded to a server and/or a database for processing.” Col.6, lines 23-35, “when autonomous vehicle 101 is in an autonomous mode, autonomous vehicle 101 is able to generally operate without a driver or a remote operator controlling autonomous vehicle … When autonomous vehicle 101 operates in a fully autonomous mode, autonomous vehicle 101 typically operates substantially only under the control of an autonomy system.” Col.17, lines 9-11, “… the systems of an autonomous vehicle, as described above with respect to FIG. 3, may include hardware, firmware, and/or software embodied on a tangible medium.” Examiner note: The reference teaches obtaining data collected from an autonomous vehicle as the vehicle drives and as the autonomy system of the vehicle is tested in real-world environments. The reference further teaches that the systems of the autonomous vehicle may include software embodied on a tangible medium, which corresponds to the first version of autonomous vehicle software. Thus, the reference teaches receiving data compiled by an autonomous vehicle operating in a real-world environment and configured to execute a first version of autonomous vehicle software);
identify, based on at least one traffic scene selector and the collection of data, a plurality of traffic scene datasets each corresponding to an instance in which the one or more autonomous vehicles encountered a traffic scene (Col.8, lines 49-62, “Data obtained at 410 … is defined as a test data set obtained based on random routing in one or more simulations and/or real driving environments. At 420, the processing module 358 processes the data to generate scenes and group scenes into scenario categories. In one embodiment, the processing module 358 reduces or otherwise processes test data set to generate scenario categories based on various scenes in the environment in which the vehicle is operating. Specifically, the processing module 358 analyzes all the miles of the test data set, calculates all events (e.g., collisions) that were encountered in these miles, and slices and dices all of these miles into scenes …”. Examiner note: the reference teaches processing module 358 processing the collected vehicle data to generate scenes into scenario categories. In particular, processing module 358 analyzes the miles constrained in the collected data, determine events encountered in those miles, and slices the miles into individual scenes for categorization. Thus, processing module 358 function as the traffic scene selector because it selects and separates portions of the collected data corresponding to particular encountered traffic scenes and organized those portions into scene datasets. The generated scenes corresponds to instances in which the vehicle encountered traffic scenes);
determine, based on the plurality of traffic scene datasets, one or more metrics for characterizing an operation of the one or more autonomous vehicles in relation to the traffic scene (Col.7, lines 22-27, “To obtain scene-based metrics, miles driven by the autonomy system are analyzed to determine frequency and type of events encountered along the way and the type of route that is being driven. That is, the miles driven are sliced into portions (scenes) and assessed separately.” Col.8, lines 14-19, “At 430, the processing module 358 generates scene-based metrics based on frequency of collisions for each scenario category i.e., MPCD per scenario category. The frequency of collisions is calculated as a function of the number of scenes (distance in miles) for each scenario category and number of failures that occurred for these scenes.” Examiner note: the reference teaches generating scene-based metrics for each scenario category. In particular, the processing module generates metric based on frequency of collisions and failures occurring within the corresponding scenario category. These metrics characterize operation of the autonomy system in relation to the traffic scene because they quantify how the autonomous vehicles performs when operating within the corresponding scene/scenario category); and
determine, based on the one or more metrics, a first prediction of future performance of a fleet of autonomous vehicles in relation to the traffic scene, wherein the fleet of autonomous vehicles are configured to execute the first version of autonomous vehicle software (Col.9, lines 41-49, “at 450, the mapping module 362 maps scene-based metrics of the test data directly to distance-based metrics. Specifically, the mapping module 362 is configured to map the MPCD per scenario category to a distance metric. The mapping module 362 maps, translates, or transforms the scene-based metrics into distance-based metrics, or substantially standard metrics that may be used to evaluate whether an autonomy system meets deployment standards, e.g., minimum deployment standards.” Col.9, lines 53-67 and col.10, lines 1-13, “At 460, the mapping module 362 obtains distance-based metrics. … The distance-based metrics are then provided to the readiness determination module 366. At 470, the readiness determination module 366 evaluates the distance-based metrics to determine whether the autonomy system is ready to deploy safely, or otherwise meets minimum readiness or deployment standards … an evaluation or verification of an autonomy system may include testing the autonomy system to substantially determine when the autonomy system is within an acceptable residual risk on a per scenario category basis.” Col.17, lines 9-11, “… the systems of an autonomous vehicle, as described above with respect to FIG. 3, may include hardware, firmware, and/or software embodied on a tangible medium.” Examiner note: the reference teaches mapping scene-based metrics, including MPCD per scenario category, into distance-based metrics used to evaluate whether an autonomy system meets deployment standards. The resulting readiness/safety evaluation corresponds to a prediction of future performance because it determine whether the autonomy system is expected to operate safety based on metrics associated with the corresponding scenario category. The reference also teaches that the autonomous vehicle system includes software embodied on a tangible medium, which corresponds to the first version of autonomous vehicle software. Thus, the reference teaches determine, based on the metrics, a first prediction of future performance of autonomous vehicles in relation to the traffic scene, wherein the autonomous vehicles are configured to execute the first version of autonomous vehicle software).
Claim 4, Nag teaches The system of claim 1, wherein the at least one traffic scene selector includes at least one of an autonomous vehicle detector, a detector confidence level, and a scene descriptor. (Col.7, lines 30-39, “In one embodiment, scenes or scenarios may be defined as a combination of an Operational Design Domain (ODD) and an Object and Event Detection and Response (OEDR). A scenario may be a combination of parameters or parameter options, and may be of any suitable length defined in distance and/or time. An ODD may identify a static scene parameter such road type that is traversed by a vehicle or a robot, and an OEDR may characterize an intelligent, reactionary environment around the vehicle or robot, e.g., may characterize dynamic agents and events in the environment.” Examiner note: the reference teaches that scenes or scenarios are defined using ODD and OEDR parameters, where the ODD identifies static scene parameters and the OEDR characterizes dynamic agents and events in the environment. These ODD/OEDR parameters correspond to scene descriptor because they describe ethe traffic scene used to identify and categorize the scene).
Claim 5, Nag teaches The system of claim 4, wherein the scene descriptor includes at least one of an object type, an object size, an object action, and a map location (Col.7, lines 43-49, “In such an example, an OEDR may indicate, but is not limited to indicate, the presence of pedestrians, the presence of vehicles with drivers in cross-traffic, the presence of vehicles with drivers in front of the vehicle, the presence of vehicles with drivers behind the vehicle, the presence of parked vehicles, and/or the presence of foreign objects such as debris.” Examiner note: the reference teaches OEDR parameters identifying pedestrians, vehicles, parked vehicles, and foreign objects such as debris in that traffic scene. These identified categories correspond to object tyles included in the scene descriptor).
Claim 7, Nag teaches The system of claim 1, wherein the one or more metrics for characterizing the operation of the one or more autonomous vehicles in relation to the traffic scene include at least one of an exposure rate, a precision metric, and a recall metric (Col.8, lines 14-19, “At 430, the processing module 358 generates scene-based metrics based on frequency of collisions for each scenario category i.e., MPCD per scenario category. The frequency of collisions is calculated as a function of the number of scenes (distance in miles) for each scenario category and number of failures that occurred for these scenes.” Examiner note: the reference teaches a scene-based metric calculated using the number of scenes or distance in miles associated with a scenario category and the failures occurring within that scenario category. Thus, the reference teaches an exposure rate type metric because the metric characterizes vehicle operation relative to the amount of exposure to the correspond traffic scene).
The elements of claims 8, 11-12, 14-15, 18 and 20 are substantially the same as those of claims 1, 4-5 and 7. Therefore, the elements of claims 8, 11-12, 14-15, 18 and 20 are rejected due to the same reasons as outlined above for claims 1, 4-5 and 7. For the limitation of claim 15, “A non-transitory computer-readable media comprising instructions stored thereon which, when executed are configured to cause a computer or processor to:” (see Nag, Col.17, lines 50-57).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and
103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set
forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103
are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2-3, 9-10, 16, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Nag
US11886193B1 in view of Pedersen US20220198107A1.
Claim 2, Nag fails to teach, but Pedersen teaches The system of claim 1, wherein the one or more processors are further configured to:
perform one or more simulated tests of the first version of autonomous vehicle software using a simulation environment configured to implement one or more simulation test scenarios, wherein at least a portion of the one or more simulation test scenarios are based on one or more of the plurality of traffic scene datasets ([0054], “the storage system 450 may also store autonomous control software which is to be used by vehicles, such as vehicle 100, to operate a vehicle in an autonomous driving mode. This autonomous control software stored in the storage system 450 may be a version which has not yet been validated.” [0056], “…The log data may include characteristics of features detected in the vehicle's environment while the vehicle drives along the path, such as road agents, objects, traffic control features, or road features.” [0060], “The one or more processors 410 may construct environment data for the given area using the log data.” [0063], “The one or more processors 410 may run the set of simulations for a particular maneuver using the constructed environment data according to the same parameters. In some simulations, a destination may be set in the given area, and the autonomous vehicle software may be run to navigate a simulated autonomous vehicle to perform the maneuver as it approaches the destination.” Examiner note: the reference teaches autonomous control software stored for use by autonomous vehicles and further teaches that the software may be a version that have not yet been validated. The reference also teaches log data including road agents, objects, traffic control feature, and road features detected in the vehicle environment, which correspond to traffic scene information. The reference further teaches constructing environment data from the log data and running simulations using the constructed environment data, where the autonomous vehicle software is run to navigate a simulated autonomous vehicle. Thus, the reference teaches perform simulated tests of a first version of autonomous vehicle software using simulation scenarios based on traffic scene datasets).
determine, based on the one or more simulated tests, one or more additional metrics for characterizing the operation of the first version of autonomous vehicle software in relation to the traffic scene ([0072], “The one or more processors 410 or a separate system may extract one or more metrics from the set of simulations. For each simulation in the set of simulations, the same one or more metrics may be extracted … The one or more metrics may include compliance metrics 812 (e.g., an amount of overlap with a driveway or no parking zone), location metrics 814 (e.g., distance or angle to the curb if we are picking up a passenger), scene impact metrics 816 (e.g., degree to which other road agents were delayed or otherwise inconvenienced), overall quality metric 818 (i.e., a weighted combination of the other metrics), or improvement cost metrics 820 …”. Examiner note: the reference teaches extracting metrics from the set of simulations. The extracted metrics as additional metrics determined from the simulated tests. The metrics characterize operation of the autonomous vehicle software in relation to the traffic scene because they measure how the simulated autonomous vehicle performs in the constructed environment containing road agents, objects, traffic control features, and road features); and
update, based on the one or more additional metrics, the first prediction of future performance of the fleet of autonomous vehicles in relation to the traffic scene. ([0029], “In other words, this technology provides the opportunity to predict and test how autonomous vehicles may behave in situations and locations that have not yet been tried in the real world.” [0087], “At block 940, one or more metrics may be extracted from the set of simulations. At block 950, an evaluation of the set of simulations may be performed using the one or more metrics. For example, the one or more metrics for a first simulation in the set may be compared with the one or more metrics for a second simulation in the set. At block 960, an adjustment to an autonomous vehicle software may be determined based on the evaluation.” Examiner note: the reference teaches extracting metrics from simulated tests, evaluating the simulations using the metrics, and determining an adjustment based on the evaluation. The reference further teaches predicting how autonomous vehicles may behave in situations and locations that have not yet been tried in the real world. Thus, the reference teaches using additional metrics obtained from simulated tests to update an assessment of expected future autonomous vehicle performance in relation to the traffic scene).
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Nag to incorporate the teachings of Pedersen, and apply simulation based evaluation of autonomous vehicle software using simulation derived metrics in order to obtain additional performance assessment information beyond the scene-based metrics derived from collected traffic scene data and improve evaluation of autonomous vehicle behavior before deployment. In this case, Nag teaches generating scene-based metrics from traffic scenes/scenario categories and evaluating an autonomous driving system based on those metrics to assess deployment readiness. Pedersen teaches running simulation using autonomous vehicle software environments constructed from logged driving data, extracting metrics from the simulations, evaluating the simulations using the extracted metrics, determining adjustments based on the evaluation, and predicting how autonomous vehicles may behave in situations and locations not yet tried in the real world. The combination of teachings would predictably provide benefit of supplementing Nag’s traffic scene based evaluation with additional simulation derived metrics, thereby improving assessment of expected autonomous vehicle behavior before deployment and reducing uncertainty associated with operation of autonomous vehicles in corresponding traffic scenes.
Claim 3, Nag fails to teach, but Pedersen teaches The system of claim 1, wherein the one or more processors are further configured to:
perform one or more simulated tests of a second version of autonomous vehicle software using a simulation environment configured to implement one or more simulation test scenarios, wherein at least a portion of the one or more simulation test scenarios are based on one or more of the plurality of traffic scene datasets ([0019], “Scenarios may be run in the simulation using autonomous vehicle software where ... the software is different from what is used to collect the logs.” [0028], “… the verified simulation system may be used to test various software for autonomous vehicles. Using these tests, the verified simulation system may compare how each of the various software performs in comparison to one another or in comparison to software used in existing runs in the log data.” [0020], “The log data may include characteristics of features detected in the vehicle's environment while the vehicle drives along the path, such as road agents, objects, traffic control features, or road features.” [0023], “The simulation system may run the set of simulations according to the parameters. In some simulations, a destination may be set in the given area, and the autonomous vehicle software may be run to navigate a simulated autonomous vehicle … Portions of a logged run may be used in the simulation.” Examiner note: the reference teaches running simulation scenarios using autonomous vehicle software that is different from the software used to collect the logged runs and further teaches testing and comparing various autonomous vehicle software implementations. The software being tested in the simulations is a different software version than the software associated with the logged runs and corresponds to the second version of autonomous vehicles software. The reference further teaches using log data containing road agents, objects, traffic control features, and road features detected in the vehicle environment, and using portions of the logged runs in the simulations. Thus, the reference teaches performing simulated tests of a second version of autonomous vehicle software using simulation scenarios based on traffic scene datasets);
determine, based on the one or more simulated tests, one or more revised metrics for characterizing the operation of the second version of autonomous vehicle software in relation to the traffic scene ([0072], “The one or more processors 410 or a separate system may extract one or more metrics from the set of simulations. For each simulation in the set of simulations, the same one or more metrics may be extracted … The one or more metrics may include compliance metrics 812 (e.g., an amount of overlap with a driveway or no parking zone), location metrics 814 (e.g., distance or angle to the curb if we are picking up a passenger), scene impact metrics 816 (e.g., degree to which other road agents were delayed or otherwise inconvenienced), overall quality metric 818 (i.e., a weighted combination of the other metrics), or improvement cost metrics 820 …”. [0080], “Alternatively, comparing the decision process simulation and the replay simulation may show how changes in the autonomous vehicle software affect performance of the particular maneuver. In this scenario, the autonomous vehicle software simulated in the decision process simulation may differ from the autonomous vehicle software used in the replayed run. As such, the differences between the metrics of these two simulations may be attributed to the differences between the two software.” Examiner note: the reference teaches extracting metrics from simulations and comparing metrics for simulations using different autonomous vehicle software. The software simulated in the decision process simulation differs from the software used in the replayed run, and the difference between the metrics are attributed to differences between the two software versions. Thus, the metrics determined from the simulation using the different software correspond to revised metrics for characterizing operation of the second version of autonomous vehicle software in relation to the traffic scene); and
determine, based on the one or more revised metrics, a second prediction of future performance of the fleet of autonomous vehicles in relation to the traffic scene, wherein the fleet of autonomous vehicles are configured to execute the second version of autonomous vehicle software. ([0020], “The log data may include characteristics of features detected in the vehicle's environment while the vehicle drives along the path, such as road agents, objects, traffic control features, or road features.” [0028], “… the verified simulation system may be used to test various software for autonomous vehicles. Using these tests, the verified simulation system may compare how each of the various software performs in comparison to one another or in comparison to software used in existing runs in the log data.” [0029], “In other words, this technology provides the opportunity to predict and test how autonomous vehicles may behave in situations and locations that have not yet been tried in the real world.” Examiner note: the reference teaches testing log derived environments containing road agent, objects, traffic control features, and road features. The log derived environment features correspond to traffic scene information. The reference further teaches comparing how each software performs and predicting how autonomous vehicles may behave in situations and locations not yet tried in the real world. The different software being tested corresponds to the second version of autonomous vehicle software).
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Nag to incorporate the teachings of Pedersen, and apply simulation based testing and comparison of different version of autonomous vehicle software using log derived traffic scene environments and simulation derived metrics in order to evaluate how software modifications affect autonomous vehicle behavior and predict expected performance of a modified autonomous vehicle software version before real world deployment. In this case, Nag teaches evaluating an autonomy system using scene-based metrics generated from traffic scenes/scenario categories and determining whether the autonomous driving system is ready for deployment. Pedersen teaches running simulation using autonomous vehicle software in log derived environments, testing software that is different from software used in existing logged runs, extracting metrics from the simulations, and comparing metrics associated with different software version, and predicting how autonomous vehicles may behave in situations and locations not yet tried in the real world. The combination of teachings would predictably provide benefit of supplementing Nag’s traffic scene based readiness evaluation with simulation derived performance comparisons for different autonomous vehicle software version, thereby improving assessment of expected future autonomous vehicle performance for a fleet operating a modified autonomous vehicle software version in corresponding traffic scenes prior to deployment.
The elements of claims 9-10, 16, and 17 are substantially the same as those of claims 2-3. Therefore, the elements of claims 9-10, 16, and 17 are rejected due to the same reasons as outlined above for claims 2-3.
Claim(s) 6, 13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Nag
US11886193B1 in view of Green US20180190111A1.
Claim 6, Nag fails to teach, but Green teaches The system of claim 1, wherein the traffic scene corresponds to a temporary traffic scene, and wherein the temporary traffic scene includes at least one of a stopped school bus, a human controlling traffic, a road closure, a construction zone, a traffic redirection, a traffic blockage, and an emergency vehicle. ([0030], “the local machine learning models 112 can be trained to recognize traffic scenes, such as an accident, construction, a broken down vehicle, a pedestrian waiting to cross, an emergency vehicle approaching, etc.”).
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Nag to incorporate the teachings of Green, and apply traffic scene recognition for temporary or unusual traffic conditions in order to identify traffic scenes that may affect traffic flow and autonomous vehicle operation. In this case, Nag teaches identifying traffic scenes/scenario categories and evaluating autonomous vehicle performance using scene based metrics. Green teaches recognizing traffic scenes, including construction and an emergency vehicle approaching. The combination of teachings would predictably provide benefit of extending Nag’s scene based evaluation to temporary traffic scenes, thereby improving evaluation of autonomous vehicle performance in traffic conditions that my require different chicle behavior or increased safety consideration.
The elements of claims 13 and 19 are substantially the same as those of claim 6. Therefore, the elements of claims 13 and 19 are rejected due to the same reasons as outlined above for claim 6.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hendy US11912301B1, discloses techniques may include determining a level of exposure associated
with scenarios, searching for similar scenarios, and generating new additional scenarios. A driving scenario may be represented as top-down multi-channel data. The top-down multi-channel data may be provided as input to a neural network trained to output a prediction of future events..
Zhang US20190278290A1, discloses determine the requirement of a perception range for a
particular type of vehicles and a particular planning and control technology. A shadow filter is used to connect a scenario based simulator and a PnC module, and tuning the parameters (e.g. decreasing the filter range, tuning the probability of obstacles to be observed among frames) of shadow filter to mimic the real world perceptions with a limited range and reliabilities. Based on the simulation results (e.g., a failure rate, smoothness, etc.), the system is able to determine the required perception distance for the current PnC module..
Phillips US20190146492A1, discloses evaluating driving performance under a plurality of driving scenarios and conditions. More specifically, the application teaches a method and apparatus for testing a driving scenario repetitively while altering a parametric variation, such as fog level, in order to evaluate driving system performance under changing conditions.
Any inquiry concerning this communication or earlier communications from the examiner should be
directed to YI HAO whose telephone number is (571)270-1303. The examiner can normally be reached Monday - Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emerson Puente can be reached on (571)272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YI . HAO/
Examiner, Art Unit 2187
/EMERSON C PUENTE/Supervisory Patent Examiner, Art Unit 2187
Prosecution Insights