DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claim 1 was submitted on September 13, 2024. In Preliminary Amendment, submitted October 29, 2024, claim 1 was amended, and claims 2-20 were added. No new matter was added in the Preliminary Amendment. Therefore, claims 1-20 are currently pending and have been examined in this application. This communication is the first action on the merits.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on December 11, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1
Claims 1-10 are directed to a method (i.e., a process); claims 11-15 are directed to a system (i.e., a machine); claims 16-20 are directed to a non-transitory computer-readable storage medium (i.e., a machine). Therefore, claims 1-20 all fall within the one of the four statutory categories of invention.
Step 2A, Prong One
Independent claims 1, 11, and 16 substantially recite receiving vehicle operation data comprising a plurality of driving events that occurred during operation of one or more vehicles over a plurality of road segments;
associating each of the plurality of driving events with one of the plurality of road segments in which the driving event occurred;
determining a performance metric for each of the plurality of road segments based on the driving events associated with the road segment;
identifying a subset of the plurality of road segments with performance metrics exceeding a threshold; and
modifying an operational design domain (ODD) of the one or more vehicles, wherein the modifying comprises expanding the ODD of the one or more vehicles to include the identified subset of road segments.
The limitations stated above are processes/functions that under broadest reasonable interpretation covers “mental processes” (such as evaluation) of monitoring surroundings and navigation (See PG Publication Para. 3) and “certain methods of organizing human activity” (commercial interactions) of determining ODDs (See PG Publication Para. 8) and safely navigating to target destinations (See PG Publication Para. 45). Therefore, the claim recites an abstract idea.
Step 2A, Prong Two
The judicial exception is not integrated into a practical application. Claims 1, 11, and 16 as a whole amount to: (i) merely invoking generic components as a tool to perform the abstract idea or “apply it” (or an equivalent). The claim recites the additional elements of: (i) a computing system (claims 1, 16), (ii) at least one processor (claims 11, 16), (iii) a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations (claim 11), (iv) a non-transitory computer-readable storage medium including instructions (claim 16).
The additional elements of (i) a computing system, (ii) at least one processor, (iii) a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations, (iv) a non-transitory computer-readable storage medium including instructions are recited at a high level of generality (see [0103] of the Applicants PG Publication discussing the computing system, [0105] discussing the at least one processor, [0106] discussing the remote devices, [0062] discussing memory storing instructions that, when executed by the at least one processor, cause the system to perform operations, and the non-transitory computer-readable storage medium including instructions) such that, when viewed as whole/ordered combination, it amounts to no more than mere instruction to apply the judicial exception using generic computer components or “apply it” (See MPEP 2106.05(f)).
Accordingly, these additional elements, when viewed as a whole/ordered combination [See Figure 7 showing all the additional (i) a computing system, (ii) at least one processor, (iii) a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations, (iv) a non-transitory computer-readable storage medium including instructions in combination], do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Thus, the claim is directed to an abstract idea.
Step 2B
As discussed above with respect to Step 2A Prong Two, the additional elements amount to no more than: (i) “apply it” (or an equivalent), and (ii) generally link the use of a judicial exception to a particular technological environment or field of use, and are not a practical application of the abstract idea. The same analysis applies here in Step 2B, i.e., (i) merely invoking the generic components as a tool to perform the abstract idea or “apply it” (See MPEP 2106.05(f)); and (ii) generally linking the use of a judicial exception to a particular technological environment or field of use (See MPEP 2106.05(h)), does not integrate the abstract idea into a practical application at Step 2A or provide an inventive concept at Step 2B. Thus, even when viewed as a whole/ordered combination, nothing in the claims adds significantly more (i.e., an inventive concept) to the abstract idea. Thus, the claims 1, 11, and 16 are ineligible.
Dependent Claims 5-10 merely narrow the previously recited abstract idea limitations. For reasons described above with respect to claim 1 these judicial exceptions are not meaningfully integrated into a practical application or significantly more than the abstract idea. Thus, claims 5-10 are also ineligible.
Step 2A, Prong Two
Dependent Claims 2, 12, and 17 further narrow the previously recited abstract idea limitations, substantially reciting the additional abstract idea of: identifying a second subset of the plurality of road segments with performance metrics below the threshold; and
training based on data collected from traveling the second subset of road segments.
Claims 2, 12, and 17 also recites the additional element of training the one or more vehicles, which is recited at a high-level of generality (See [0067] of the Applicants PG Publication disclosing training the one or more vehicles) such that when viewed as whole/ordered combination, the additional elements do no more than generally link the use of the judicial exception to a particular technological environment or field of use (i.e., autonomous driving vehicles) (See MPEP 2106.05(h)).
Accordingly, the additional elements, when viewed individually and as a whole/ordered combination, do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Thus, the claims are directed to an abstract idea.
Step 2B
As discussed above with respect to Step 2A Prong Two, the additional element amounts to no more than: generally linking the use of a judicial exception to a particular technological environment or field of use, and is not a practical application of the abstract idea. The same analysis applies here in Step 2B, i.e., (i) generally linking the use of a judicial exception to a particular technological environment or field of use (See MPEP 2106.05(h)), does not integrate the abstract idea into a practical application at Step 2A or provide an inventive concept at Step 2B.
Therefore, the additional element of training the one or more vehicles does not integrate the abstract idea into a practical application at Step 2A or provide an inventive concept at Step 2B. Thus, even when viewed as a whole/ordered combination, nothing in the claim adds significantly more (i.e., an inventive concept) to the abstract idea. Thus, claims 2, 12, and 17 are ineligible.
Step 2A, Prong One
Dependent Claims 3, 13, and 18 further narrow the previously recited abstract idea limitations, substantially reciting in response to a ride request, identifying an autonomous vehicle (AV)-executable route within the ODD; and
dispatching to service the ride request based on the AV-executable route within the ODD.
The limitations stated above are processes/functions that under broadest reasonable interpretation covers “certain methods of organizing human activity” (commercial interactions) of managing transportation requests. (See PG Publication Para. 14). Therefore, these claims recite an abstract idea.
Step 2A, Prong Two
Claims 3, 13, and 18 also recites the additional element of an AV, which is recited at a high-level of generality (See [0087] of the Applicants PG Publication disclosing the AV) such that when viewed as whole/ordered combination, the additional elements do no more than generally link the use of the judicial exception to a particular technological environment or field of use (i.e., autonomous driving vehicles) (See MPEP 2106.05(h)).
Accordingly, the additional elements, when viewed individually and as a whole/ordered combination, do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Thus, the claims are directed to an abstract idea.
Step 2B
As discussed above with respect to Step 2A Prong Two, the additional element amounts to no more than: generally linking the use of a judicial exception to a particular technological environment or field of use, and is not a practical application of the abstract idea. The same analysis applies here in Step 2B, i.e., (i) generally linking the use of a judicial exception to a particular technological environment or field of use (See MPEP 2106.05(h)), does not integrate the abstract idea into a practical application at Step 2A or provide an inventive concept at Step 2B.
Therefore, the additional element of training the one or more vehicles does not integrate the abstract idea into a practical application at Step 2A or provide an inventive concept at Step 2B. Thus, even when viewed as a whole/ordered combination, nothing in the claim adds significantly more (i.e., an inventive concept) to the abstract idea. Thus, claims 3, 13, and 18 are ineligible.
Dependent Claims 4, 14, 15, 19, and 20 merely narrow the previously recited abstract idea limitations. For reasons described above with respect to claims 3, 13, and 18 these judicial exceptions are not meaningfully integrated into a practical application or significantly more than the abstract idea. Thus, claims 4, 14, 15, 19, and 20 are also ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 7, 11-14, and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Bin-Nun (US 20190163185) (hereafter Bin-Nun) in view of Mui (“Driverless Cars and the '90-90 Rule'”; March 19, 2018) (hereafter Mui).
In regards to claim 1, Bin-Nun discloses a computer-implemented method comprising: receiving, by a computing system, vehicle operation data comprising a plurality of driving events that occurred during operation of one or more vehicles over a plurality of road segments; (Para. 6, 40-45, 88) (“a method for evaluating safety performance of an autonomous vehicle is provided, the method comprising … comparing second sensor data characterizing an autonomous-vehicle-operation to a second threshold to obtain a second driving quality value, and determining … a safety performance of the autonomous-operated vehicle” “vehicle operational data can be procured directly from vehicle components or sensors with the capability to transmit data … Algorithms that can extract ranging and kinematic information from the vehicles (how fast they are going; their acceleration; the headway between vehicles at a given moment in time) can be applied to sensor data to extract events (i.e. receiving, by a computing system, vehicle operation data comprising a plurality of driving events that occurred during operation of one or more vehicles over a plurality of road segments) which are predictive of crash risk (e.g. hard accelerations or near-misses). … to allow for the statistically significant comparison, AV performance in a localized geographic area is compared to a population of human drivers operating in the same geographical area is obtained, under similar road and ambient conditions as the AV, or under similar operational constraints, or some combination of these conditions, … To further improve the comparison, validation can take place for a specific set of roads (i.e. driving events that occurred during operation of one or more vehicles over a plurality of road segments), time of day, operational speed, weather conditions, or any combination thereof.” “the present invention is directed to one or more hardware computer-readable media, having stored thereon instructions executable by a processor to perform the methods described herein (i.e. a computer-implemented method) (e.g., comparison device, determination device, calculation device). For example, an example embodiment of the present invention is directed to a computer system.”)
Bin-Nun discloses associating, by the computing system, each of the plurality of driving events with one of the plurality of road segments in which the driving event occurred; (Para. 44-45) (“AV performance in a localized geographic area is compared to a population of human drivers operating in the same geographical area is obtained, under similar road and ambient conditions as the AV, or under similar operational constraints, or some combination of these conditions, e.g., operational design domain (“ODD”). Accordingly, within an ODD, the performance of AVs can be examined by aggregating performance data for the AV. Any relevant occurrences can be logged and the frequency of incidents in the ODD (i.e. associating, by the computing system, each of the plurality of driving events with one of the plurality of road segments in which the driving event occurred) as well as the driver score of the AV can be compared relative to the human driven benchmark in the ODD. … This can be done, for example, by aggregating the frequency of events which may not be collisions, but are indicative of a propensity for collisions (“near-misses”) or other metrics for assessing human driver performance across the domain. Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads, time of day, operational speed, weather conditions, or any combination thereof.”)
Bin-Nun discloses determining, by the computing system, a performance metric for each of the plurality of road segments based on the driving events associated with the road segment; (Para. 43-45) (“in a first step 100, identifying the correct performance metric or metrics … AV performance (e.g., “second sensor data”). … Metrics capturing information regarding AV safety performance are weighed to create an aggregate performance score …These metrics can overlap with the set of metrics that are used to gauge human performance, but can also include additional metrics that are specific to AVs. Once the correct performance metrics are identified, in a second step 102, a sample can be created of … AV driving operation … AV performance in a localized geographic area (i.e. determining, by the computing system, a performance metric for each of the plurality of road segments based on the driving events associated with the road segment) is compared to a population of human drivers operating in the same geographical area is obtained, under similar road and ambient conditions as the AV, or under similar operational constraints, or some combination of these conditions, e.g., operational design domain (“ODD”). Accordingly, within an ODD, the performance of AVs can be examined by aggregating performance data for the AV. … This can be done, for example, by aggregating the frequency of events which may not be collisions, but are indicative of a propensity for collisions (“near-misses”) or other metrics for assessing human driver performance across the domain. Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads (i.e. determining, by the computing system, a performance metric for each of the plurality of road segments based on the driving events associated with the road segment)”)
Bin-Nun discloses identifying, by the computing system, a subset of the plurality of road segments with performance metrics exceeding a threshold; and (Para. 45) (“This can be done, for example, by aggregating the frequency of events which may not be collisions, but are indicative of a propensity for collisions (“near-misses”) or other metrics for assessing human driver performance across the domain. Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads … performance standards can be specified for AVs such as being a certain percentage better than humans, and operational design domains that meet the specified criteria can be identified (i.e. identifying, by the computing system, a subset of the plurality of road segments with performance metrics exceeding a threshold).”)
Bin-Nun discloses modifying, by the computing system, an operational design domain (ODD) of the one or more vehicles, wherein the modifying comprises including the identified subset of the plurality of road segments. (Para. 45, 49, 67, 70-71) (“Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads (i.e. the identified subset of the plurality of road segments), time of day, operational speed, weather conditions, or any combination thereof.” “Accordingly, AV developers, for example, whose product has been validated in a particular ODD or set of ODDs (i.e. including the identified subset of the plurality of road segments) can determine what other ODDs are proximate to the validated ODD. This can inform further development or deployment decisions as safety validation can be easiest for ODDs proximate to ones which have already been validated.” “AV developers and other interested parties can evaluate the similarity of different ODDs to focus development efforts or understand the challenges in expanding AV operational capabilities to another ODD (i.e. wherein the modifying comprises including the identified subset of the plurality of road segments).” “For example, an ODD can have three elements (time of day, weather, and road type), and each element can have three potential values (1,2,3). These can correspond to the elements as follows: time of day: day, night, twilight; weather: Sunny, rainy, snowy; road type: highway, urban road, rural road. … However, it will be appreciated that ODD's can have considerably more than three elements and each element can have considerably more than three potential values.” That is, the system validates for a specific set of roads (i.e. the identified subset of the plurality of road segments) in order to evaluate and validate a particular ODD or set of ODDs for a product (i.e. modifying, by the computing system, an operational design domain (ODD) of the one or more vehicles, wherein the modifying comprises including the identified subset of the plurality of road segments).)
Bin-Nun does not explicitly disclose, however Mui, in the same field of endeavor, discloses the modifying the ODD of the one or more vehicles (to include the identified subset of the plurality of road segments of Bin-Nun comprises expanding the ODD of Bin-Nun. (Pg. 4) (“One next step is expanding the “operational design domains” (ODD) of the cars (i.e. ODD of Bin-Nun). This includes expanding into “dense urban cores,” such as San Francisco (in which Waymo recently announced it is expanding its testing program). The other ODD was additional weather conditions, such as hard rain, snow and fog (i.e. modifying comprises expanding the ODD of Bin-Nun). (Waymo CEO John Krafcik recently told an audience that he was “jumping up and down” recently when it snowed 12 inches near Detroit, because it would enable Waymo’s testing in snow (i.e. include the identified subset of the plurality of road segments of Bin-Nun.)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the vehicle safety systems of Bin-Nun with the driverless car system of Mui in order to improve the vehicles ability to work safely in an unpredictable world. (Mui – Pg. 3)
In regards to claim 2, Bin-Nun in view of Mui disclose the limitations of claim 1. Bin-Nun discloses identifying a second subset of the plurality of road segments with performance metrics below the threshold; and (Para. 46, 47, 67) (“performance standards can be specified for AVs such as being a certain percentage better than humans, and operational design domains that meet the specified criteria can be identified.” “Specify routes can also be identified for use to fulfill a trip, for example, from point A to point B and beginning at a certain time, that are within ODDs that meet a specified safety benchmark, and/or the difference in timing it would take to fulfill the trip with different tolerances of safety performance and/or an AV or human driver who can operate without restrictions in ODD can be evaluated.” “AV developers and other interested parties can evaluate the similarity of different ODDs to focus development efforts or understand the challenges in expanding AV operational capabilities to another ODD.” That is, while Bin-Nun specifically discusses identifying road segments that exceed and performance threshold, it is clear that identifying road segments below the performance threshold is also considered within the scope as evidenced by the presence of ODDs that are restricted to AVs in Para. 47, and the focus of developers to expand into other ODDs in Para. 67 (i.e. ODDs that fall below the performance threshold.).)
Bin-Nun does not explicitly disclose, however Mui, in the same field of endeavor, discloses training the one or more vehicles based on data collected from traveling the second subset of road segments. (Pg. 3) (“He highlighted Waymo’s three-prong testing program of real-world driving, simulation and structured testing as key to iterating on and productizing the technology. Much is made of the millions public-road miles that Waymo’s cars have driven autonomously. … Even more important is the ability to test against “fuzzed” versions of those millions of miles, such as seeing how the software would handle cars going at slightly different speeds, an extra car, pedestrians crossing in front of the car and so on (i.e. training the one or more vehicles based on data collected from traveling the second subset of road segments) . Arnoud described Waymo’s simulation-based testing capability as the equivalent of 25,000 virtual cars driving 2.5 billion real and modified miles in 2017. The third component of Waymo’s testing program is its structured testing program. Arnoud said that there is a ‘long tail’ of driving situations that happen very rarely. Rather than trying to encounter every possibility in real-world driving, Waymo set up a 90-acre mock city at the decommissioned Castle Air Force base where it can test its cars against such edge cases (i.e. training the one or more vehicles based on data collected from traveling the second subset of road segments). These tests are then fed into the simulation engine and fuzzed to create variations for more testing. … This includes expanding into ‘dense urban cores,’ such as San Francisco (in which Waymo recently announced it is expanding its testing program). The other ODD was additional weather conditions, such as hard rain, snow and fog. (Waymo CEO John Krafcik recently told an audience that he was ‘jumping up and down’ recently when it snowed 12 inches near Detroit, because it would enable Waymo’s testing in snow (i.e. training the one or more vehicles based on data collected from traveling the second subset of road segments).)”)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the vehicle safety systems of Bin-Nun with the driverless car system of Mui in order to improve the vehicles ability to work safely in an unpredictable world. (Mui – Pg. 3)
In regards to claim 3, Bin-Nun in view of Mui disclose the limitations of claim 1. Bin-Nun discloses in response to a ride request, identifying an autonomous vehicle (AV)-executable route within the ODD; and dispatching an AV to service the ride request based on the AV-executable route within the ODD. (Para. 46, 52, 82-83) (“It can also allow AVs to fulfill trips solely by operating in low-risk operating design domains in which AVs outperform their human counterparts by an appropriate margin. Additionally, performance standards can be specified for AVs such as being a certain percentage better than humans, and operational design domains that meet the specified criteria can be identified.” “Different AV designs can have different safety performances for the same ODD. For a particular route, one AV can perform better. The quantification of the safety of each AV on a particular combination of route segment and operating conditions can be performed. Accordingly, transportation operators with multiple AVs designs with different safety performances can dispatch one of the AVs based on its performance over the needed route.” “A transportation provider can decide whether to dispatch an AV to a customer desiring a trip from a particular origin and destination should have an autonomous vehicle dispatched to fulfill the ride. If the most direct route for the fulfillment of the trip is route R, the AV can only be dispatched if the safety of the autonomous vehicle exceeds that of a human driver by a specified margin, where k is a value greater than 1: P.sub.A(R)>k*P.sub.H(R). Additionally, multiple routes can be considered, each with different safety performances for both humans and autonomous vehicles. There can be n potential routes between the origin and destination, denoted by R.sup.1, R.sup.2, . . . , R.sup.n. An autonomous vehicle can be dispatched”)
In regards to claim 4, Bin-Nun in view of Mui disclose the limitations of claim 3. Bin-Nun discloses wherein the identifying the AV-executable route within the ODD comprises: determining a route exists from a pick-up location to a drop off location of the ride request within a region defined by the ODD associated with one or more autonomous vehicles, wherein the road segments within the region meet a performance metric-based criteria. (Para. 46, 82-83) (“It can also allow AVs to fulfill trips solely by operating in low-risk operating design domains in which AVs outperform their human counterparts by an appropriate margin. Additionally, performance standards can be specified for AVs such as being a certain percentage better than humans, and operational design domains that meet the specified criteria can be identified.” “A transportation provider can decide whether to dispatch an AV to a customer desiring a trip from a particular origin and destination should have an autonomous vehicle dispatched to fulfill the ride. If the most direct route for the fulfillment of the trip is route R, the AV can only be dispatched if the safety of the autonomous vehicle exceeds that of a human driver by a specified margin, where k is a value greater than 1: P.sub.A(R)>k*P.sub.H(R). Additionally, multiple routes can be considered, each with different safety performances for both humans and autonomous vehicles. There can be n potential routes between the origin and destination, denoted by R.sup.1, R.sup.2, . . . , R.sup.n. An autonomous vehicle can be dispatched if the safety performance on at least one route is greater than the human safety performance on its best route”)
In regards to claim 7, Bin-Nun in view of Mui disclose the limitations of claim 1. Bin-Nun discloses determining a utility metric for each of the plurality of road segments, wherein the utility metric indicates a volume of ride requests in the road segment successfully handled by an autonomous vehicle; (Para. 49-50) (“it can be estimated what fraction of trips are traveled within a certain ODD. … From the database collected by the central data hub, data on trips and safety incidents occurring in the ODD or set of ODDs can be selected 300. … the AV for an ODD or selected ODDs can be created 304, which can demonstrate the relative strengths of AVs to human drivers and where AVs overperform or underperform metrics relative to human drivers.”)
Bin-Nun discloses selecting road segments from the plurality of road segments with the utility metrics exceeding a predetermined utility threshold; and modifying the ODD to include the selected road segments. (Para. 49) (“it can be estimated what fraction of trips are traveled within a certain ODD. This can allow, for example, developers to prioritize the development of AVs both by similarity to ODDs that have already been validated, but also by how significant ODDs are. For example, it can be more valuable to develop and validate AVs for ODDs that contain greater demand for travel.”)
Bin-Nun does not explicitly disclose, however Mui, in the same field of endeavor, discloses the modifying the ODD of Bin-Nun is expanding the ODD of Bin-Nun. (Pg. 4) (“One next step is expanding the “operational design domains” (ODD) of the cars (i.e. ODD of Bin-Nun). This includes expanding into “dense urban cores,” such as San Francisco (in which Waymo recently announced it is expanding its testing program). The other ODD was additional weather conditions, such as hard rain, snow and fog (i.e. modifying comprises expanding the ODD of Bin-Nun). (Waymo CEO John Krafcik recently told an audience that he was “jumping up and down” recently when it snowed 12 inches near Detroit, because it would enable Waymo’s testing in snow.)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the vehicle safety systems of Bin-Nun with the driverless car system of Mui in order to improve the vehicles ability to work safely in an unpredictable world. (Mui – Pg. 3)
In regards to claim 11, Bin-Nun discloses a system comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising: receiving vehicle operation data comprising a plurality of driving events that occurred during operation of one or more vehicles over a plurality of road segments; (Para. 6, 40-45, 88) (“a method for evaluating safety performance of an autonomous vehicle is provided, the method comprising … comparing second sensor data characterizing an autonomous-vehicle-operation to a second threshold to obtain a second driving quality value, and determining … a safety performance of the autonomous-operated vehicle” “vehicle operational data can be procured directly from vehicle components or sensors with the capability to transmit data … Algorithms that can extract ranging and kinematic information from the vehicles (how fast they are going; their acceleration; the headway between vehicles at a given moment in time) can be applied to sensor data to extract events (i.e. receiving, by a computing system, vehicle operation data comprising a plurality of driving events that occurred during operation of one or more vehicles over a plurality of road segments) which are predictive of crash risk (e.g. hard accelerations or near-misses). … to allow for the statistically significant comparison, AV performance in a localized geographic area is compared to a population of human drivers operating in the same geographical area is obtained, under similar road and ambient conditions as the AV, or under similar operational constraints, or some combination of these conditions, … To further improve the comparison, validation can take place for a specific set of roads (i.e. driving events that occurred during operation of one or more vehicles over a plurality of road segments), time of day, operational speed, weather conditions, or any combination thereof.” “the present invention is directed to one or more hardware computer-readable media, having stored thereon instructions (i.e. a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations) executable by a processor (i.e. at least on processor) to perform the methods described herein (e.g., comparison device, determination device, calculation device). For example, an example embodiment of the present invention is directed to a computer system.”)
Bin-Nun discloses associating each of the plurality of driving events with one of the plurality of road segments in which the driving event occurred; (Para. 44-45) (“AV performance in a localized geographic area is compared to a population of human drivers operating in the same geographical area is obtained, under similar road and ambient conditions as the AV, or under similar operational constraints, or some combination of these conditions, e.g., operational design domain (“ODD”). Accordingly, within an ODD, the performance of AVs can be examined by aggregating performance data for the AV. Any relevant occurrences can be logged and the frequency of incidents in the ODD (i.e. associating each of the plurality of driving events with one of the plurality of road segments in which the driving event occurred) as well as the driver score of the AV can be compared relative to the human driven benchmark in the ODD. … This can be done, for example, by aggregating the frequency of events which may not be collisions, but are indicative of a propensity for collisions (“near-misses”) or other metrics for assessing human driver performance across the domain. Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads, time of day, operational speed, weather conditions, or any combination thereof.”)
Bin-Nun discloses determining a performance metric for each of the plurality of road segments based on the driving events associated with the road segment; (Para. 43-45) (“in a first step 100, identifying the correct performance metric or metrics … AV performance (e.g., “second sensor data”). … Metrics capturing information regarding AV safety performance are weighed to create an aggregate performance score …These metrics can overlap with the set of metrics that are used to gauge human performance, but can also include additional metrics that are specific to AVs. Once the correct performance metrics are identified, in a second step 102, a sample can be created of … AV driving operation … AV performance in a localized geographic area (i.e. determining a performance metric for each of the plurality of road segments based on the driving events associated with the road segment) is compared to a population of human drivers operating in the same geographical area is obtained, under similar road and ambient conditions as the AV, or under similar operational constraints, or some combination of these conditions, e.g., operational design domain (“ODD”). Accordingly, within an ODD, the performance of AVs can be examined by aggregating performance data for the AV. … This can be done, for example, by aggregating the frequency of events which may not be collisions, but are indicative of a propensity for collisions (“near-misses”) or other metrics for assessing human driver performance across the domain. Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads (i.e. determining a performance metric for each of the plurality of road segments based on the driving events associated with the road segment)”)
Bin-Nun discloses identifying a subset of the plurality of road segments with performance metrics exceeding a threshold; and (Para. 45) (“This can be done, for example, by aggregating the frequency of events which may not be collisions, but are indicative of a propensity for collisions (“near-misses”) or other metrics for assessing human driver performance across the domain. Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads … performance standards can be specified for AVs such as being a certain percentage better than humans, and operational design domains that meet the specified criteria can be identified (i.e. identifying a subset of the plurality of road segments with performance metrics exceeding a threshold).”)
Bin-Nun discloses modifying an operational design domain (ODD) of the one or more vehicles, wherein the modifying comprises the ODD of the one or more vehicles to include the identified subset of road segments. (Para. 45, 49, 67, 70-71) (“Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads, time of day, operational speed, weather conditions, or any combination thereof.” “Accordingly, AV developers, for example, whose product has been validated in a particular ODD or set of ODDs can determine what other ODDs are proximate to the validated ODD. This can inform further development or deployment decisions as safety validation can be easiest for ODDs proximate to ones which have already been validated.” “AV developers and other interested parties can evaluate the similarity of different ODDs to focus development efforts or understand the challenges in expanding AV operational capabilities to another ODD.” “For example, an ODD can have three elements (time of day, weather, and road type), and each element can have three potential values (1,2,3). These can correspond to the elements as follows: time of day: day, night, twilight; weather: Sunny, rainy, snowy; road type: highway, urban road, rural road. … However, it will be appreciated that ODD's can have considerably more than three elements and each element can have considerably more than three potential values.”)
Bin-Nun does not explicitly disclose, however Mui, in the same field of endeavor, discloses the modifying the ODD of Bin-Nun is modifying comprises expanding the ODD of Bin-Nun. (Pg. 4) (“One next step is expanding the “operational design domains” (ODD) of the cars (i.e. ODD of Bin-Nun). This includes expanding into “dense urban cores,” such as San Francisco (in which Waymo recently announced it is expanding its testing program). The other ODD was additional weather conditions, such as hard rain, snow and fog (i.e. modifying comprises expanding the ODD of Bin-Nun). (Waymo CEO John Krafcik recently told an audience that he was “jumping up and down” recently when it snowed 12 inches near Detroit, because it would enable Waymo’s testing in snow.)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the vehicle safety systems of Bin-Nun with the driverless car system of Mui in order to improve the vehicles ability to work safely in an unpredictable world. (Mui – Pg. 3)
In regards to claim 12, Bin-Nun in view of Mui disclose the limitations of claim 11. The remainder of the limitations of this claim are rejected using the same rationale as claim 2.
In regards to claim 13, Bin-Nun in view of Mui disclose the limitations of claim 11. The remainder of the limitations of this claim are rejected using the same rationale as claim 3.
In regards to claim 14, Bin-Nun in view of Mui disclose the limitations of claim 13. The remainder of the limitations of this claim are rejected using the same rationale as claim 4.
In regards to claim 16, Bin-Nun discloses a non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform operations comprising: receiving vehicle operation data comprising a plurality of driving events that occurred during operation of one or more vehicles over a plurality of road segments; (Para. 6, 40-45, 88) (“a method for evaluating safety performance of an autonomous vehicle is provided, the method comprising … comparing second sensor data characterizing an autonomous-vehicle-operation to a second threshold to obtain a second driving quality value, and determining … a safety performance of the autonomous-operated vehicle” “vehicle operational data can be procured directly from vehicle components or sensors with the capability to transmit data … Algorithms that can extract ranging and kinematic information from the vehicles (how fast they are going; their acceleration; the headway between vehicles at a given moment in time) can be applied to sensor data to extract events (i.e. receiving, by a computing system, vehicle operation data comprising a plurality of driving events that occurred during operation of one or more vehicles over a plurality of road segments) which are predictive of crash risk (e.g. hard accelerations or near-misses). … to allow for the statistically significant comparison, AV performance in a localized geographic area is compared to a population of human drivers operating in the same geographical area is obtained, under similar road and ambient conditions as the AV, or under similar operational constraints, or some combination of these conditions, … To further improve the comparison, validation can take place for a specific set of roads (i.e. driving events that occurred during operation of one or more vehicles over a plurality of road segments), time of day, operational speed, weather conditions, or any combination thereof.” “the present invention is directed to one or more hardware computer-readable media, having stored thereon instructions (i.e. a non-transitory computer-readable storage medium including instructions) executable by a processor (i.e. at least on processor of a computing system) to perform the methods described herein (e.g., comparison device, determination device, calculation device). For example, an example embodiment of the present invention is directed to a computer system.”)
Bin-Nun discloses associating each of the plurality of driving events with one of the plurality of road segments in which the driving event occurred; (Para. 44-45) (“AV performance in a localized geographic area is compared to a population of human drivers operating in the same geographical area is obtained, under similar road and ambient conditions as the AV, or under similar operational constraints, or some combination of these conditions, e.g., operational design domain (“ODD”). Accordingly, within an ODD, the performance of AVs can be examined by aggregating performance data for the AV. Any relevant occurrences can be logged and the frequency of incidents in the ODD (i.e. associating each of the plurality of driving events with one of the plurality of road segments in which the driving event occurred) as well as the driver score of the AV can be compared relative to the human driven benchmark in the ODD. … This can be done, for example, by aggregating the frequency of events which may not be collisions, but are indicative of a propensity for collisions (“near-misses”) or other metrics for assessing human driver performance across the domain. Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads, time of day, operational speed, weather conditions, or any combination thereof.”)
Bin-Nun discloses determining a performance metric for each of the plurality of road segments based on the driving events associated with the road segment; (Para. 43-45) (“in a first step 100, identifying the correct performance metric or metrics … AV performance (e.g., “second sensor data”). … Metrics capturing information regarding AV safety performance are weighed to create an aggregate performance score …These metrics can overlap with the set of metrics that are used to gauge human performance, but can also include additional metrics that are specific to AVs. Once the correct performance metrics are identified, in a second step 102, a sample can be created of … AV driving operation … AV performance in a localized geographic area (i.e. determining a performance metric for each of the plurality of road segments based on the driving events associated with the road segment) is compared to a population of human drivers operating in the same geographical area is obtained, under similar road and ambient conditions as the AV, or under similar operational constraints, or some combination of these conditions, e.g., operational design domain (“ODD”). Accordingly, within an ODD, the performance of AVs can be examined by aggregating performance data for the AV. … This can be done, for example, by aggregating the frequency of events which may not be collisions, but are indicative of a propensity for collisions (“near-misses”) or other metrics for assessing human driver performance across the domain. Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads (i.e. determining a performance metric for each of the plurality of road segments based on the driving events associated with the road segment)”)
Bin-Nun discloses identifying a subset of the plurality of road segments with performance metrics exceeding a threshold; and (Para. 45) (“This can be done, for example, by aggregating the frequency of events which may not be collisions, but are indicative of a propensity for collisions (“near-misses”) or other metrics for assessing human driver performance across the domain. Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads … performance standards can be specified for AVs such as being a certain percentage better than humans, and operational design domains that meet the specified criteria can be identified (i.e. identifying a subset of the plurality of road segments with performance metrics exceeding a threshold).”)
Bin-Nun discloses modifying an operational design domain (ODD) of the one or more vehicles, wherein the modifying comprises the ODD of the one or more vehicles to include the identified subset of road segments. (Para. 45, 49, 67, 70-71) (“Then the AVs can be benchmarked across the operational design domain or set of conditions/geographies 204. To further improve the comparison, validation can take place for a specific set of roads, time of day, operational speed, weather conditions, or any combination thereof.” “Accordingly, AV developers, for example, whose product has been validated in a particular ODD or set of ODDs can determine what other ODDs are proximate to the validated ODD. This can inform further development or deployment decisions as safety validation can be easiest for ODDs proximate to ones which have already been validated.” “AV developers and other interested parties can evaluate the similarity of different ODDs to focus development efforts or understand the challenges in expanding AV operational capabilities to another ODD.” “For example, an ODD can have three elements (time of day, weather, and road type), and each element can have three potential values (1,2,3). These can correspond to the elements as follows: time of day: day, night, twilight; weather: Sunny, rainy, snowy; road type: highway, urban road, rural road. … However, it will be appreciated that ODD's can have considerably more than three elements and each element can have considerably more than three potential values.”)
Bin-Nun does not explicitly disclose, however Mui, in the same field of endeavor, discloses the modifying the ODD of Bin-Nun is modifying comprises expanding the ODD of Bin-Nun. (Pg. 4) (“One next step is expanding the “operational design domains” (ODD) of the cars (i.e. ODD of Bin-Nun). This includes expanding into “dense urban cores,” such as San Francisco (in which Waymo recently announced it is expanding its testing program). The other ODD was additional weather conditions, such as hard rain, snow and fog (i.e. modifying comprises expanding the ODD of Bin-Nun). (Waymo CEO John Krafcik recently told an audience that he was “jumping up and down” recently when it snowed 12 inches near Detroit, because it would enable Waymo’s testing in snow.)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the vehicle safety systems of Bin-Nun with the driverless car system of Mui in order to improve the vehicles ability to work safely in an unpredictable world. (Mui – Pg. 3)
In regards to claim 17, Bin-Nun in view of Mui disclose the limitations of claim 16. The remainder of the limitations of this claim are rejected using the same rationale as claim 2.
In regards to claim 18, Bin-Nun in view of Mui disclose the limitations of claim 16. The remainder of the limitations of this claim are rejected using the same rationale as claim 3.
In regards to claim 19, Bin-Nun in view of Mui disclose the limitations of claim 18. The remainder of the limitations of this claim are rejected using the same rationale as claim 4.
Claims 8 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Bin-Nun in view of Mui and further in view of Corporate Partnership Board (CPB) (“Safer Roads with Automated Vehicles?”; May 23, 2018) (hereafter CPB) and even further in view of Kislovskiy (US 10,789,835) (hereafter Kislovskiy).
In regards to claim 8, Bin-Nun in view of Mui disclose the limitations of claim 1. Bin-Nun in view of Mui does not explicitly disclose, however CPB, in the same field of endeavor, discloses wherein the driving events of Bin-Nun comprise a plurality of disengagements in which autonomous operation of a vehicle is disengaged, the method further comprises: categorizing, by the computing system, the plurality of disengagements based on expected outcomes had the plurality of disengagements not occurred, (Pg. 6-7) (“Nearly all highly automated driving systems currently in testing rely on humans as back-up drivers who will take over when the computer reaches the limits of its performance. The disengagement of automated driving systems (i.e. the driving events of Bin-Nun comprise a plurality of disengagements in which autonomous operation of a vehicle is disengaged), not just crashes, are a relevant metric for tracking safety performance because the vehicle might have crashed without human intervention (i.e. categorizing, by the computing system, the plurality of disengagements based on simulated outcomes had the plurality of disengagements not occurred). Since different automated driving systems have different design parameters and functional boundaries, metadata on system capabilities should accompany disengagement reports.”)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the vehicle safety systems of Bin-Nun in view of Mui with the automated vehicle data of CPB in order to improve road safety of self-driving cars. (CPB – Pg. 5)
Bin-Nun in view of Mui does not explicitly disclose, however CPB, in the same field of endeavor, discloses the expected outcomes of CPB are simulated outcomes (Col. 5, Ln. 15-45) (“it may first be pre-certified through a set of simulations, such as log data from actual AV trips … and simulation analysis capable of adjusting simulation parameters (e.g., simulating fault conditions and failures) and incorporating additional actors … Furthermore, beyond simulation, examples described herein can also leverage recorded log data from AVs executing software while being run”)
Bin-Nun in view of Mui does not explicitly disclose, however Kislovskiy, in the same field of endeavor, discloses wherein the determining the performance metric for each of the plurality of road segments comprises: determining the performance metric for each of the plurality of road segments based on the simulated outcomes of the disengagements associated with the road segment. (Col. 5, Ln. 46-60; Col. 31, Ln. 48-65) (“thresholds to achieve verification can be determined or adjusted based on simulation results” “the AV software management system can distribute the new software version to AVs (e.g., only SDAVs) operating throughout the given region … the AV software management system can train a new risk regressor to couple with the new software version (945). The new risk regressor can determine fractional risk values for a hypothetical AV executing the new software version along each path segment of the given region (i.e. determining the performance metric for each of the plurality of road segments based on the simulated outcomes of the disengagements associated with the road segment). The AV software management system can further train a new trip classifier to couple with the new software version (950). As described herein, the new trip classifier can establish risk thresholds for the new software version in servicing transport requests (952).”)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the vehicle safety systems of Bin-Nun in view of Mui and further in view of Kislovskiy with the risk analysis of in order to improve on road vehicular safety. (Kislovskiy – Col. 1, Ln. 7-37)
In regards to claim 9, Bin-Nun in view of Mui further in view of CPB and even further in view of Kislovskiy disclose the limitations of claim 8. Bin-Nun in view of Mui does not explicitly disclose, however CPB, in the same field of endeavor, discloses wherein the simulated outcomes are associated with two or more categories selected from collision category, near- collision category, traffic rule violation category, elegance violation category, and no adverse outcome category. (Pg. 6-7) (“are a relevant metric for tracking safety performance because the vehicle might have crashed without human intervention (i.e. collision category) … the number of near misses avoided (i.e. near-collision category)”)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the vehicle safety systems of Bin-Nun in view of Mui with the automated vehicle data of CPB in order to improve road safety of self-driving cars. (CPB – Pg. 5)
Novel and Non-Obvious Over the Prior Art
Claims 5, 6, 10, 15, and 20 are novel and non-obvious over the prior art; however, these claims are subject to the above rejections.
Claims 5, 6, 15, and 20:
The closest prior art is U.S. Patent Application No. 2019/0163185 to Bin-Nun et al (hereafter Bin-Nun). Bin-Nun discloses receiving and analyzing operation data from a plurality of AVs to determine performance metrics associated with driving events and road segments and modifying ODDs.
The next closest prior art is non-patent literature “Driverless Cars and the ’90-90 Rule’” by Mui (hereafter Mui). Mui discloses receiving and analyzing operation data from a plurality of AVs to expand the ODDs of the vehicles.
The next closest prior art is U.S. Patent Application No. 2017/0336213 to Fowe et al (hereafter Fowe). Fowe discloses calibrating performance metrics for a geographical region based on performance metrics of the road segments within the region.
While the closest prior art above teaches the various aspects of the claimed invention individually, the combination of these references are not obvious in such a way that they would have been obvious to one of ordinary skill in the art at the time of invention. Specifically, Bin-Nun in view of Mui and further in view of Fowe does not explicitly disclose determining weights of weighted average based on frequency of traversal of each road segment. Therefore, the claims are rendered novel and non-obvious over the prior art.
Claim 10:
The closest prior art is U.S. Patent Application No. 2019/0163185 to Bin-Nun et al (hereafter Bin-Nun). Bin-Nun discloses receiving and analyzing operation data from a plurality of AVs to determine performance metrics associated with driving events and road segments and modifying ODDs.
The next closest prior art is non-patent literature “Driverless Cars and the ’90-90 Rule’” by Mui (hereafter Mui). Mui discloses receiving and analyzing operation data from a plurality of AVs to expand the ODDs of the vehicles.
The next closest prior art is non-patent literature “Safer Roads with Automated Vehicles?” by CPB (hereafter CpB). CPB discloses receiving and analyzing operation data of disengagements and the likely outcomes if a disengagement had not occurred.
The next closest prior art is U.S. Patent No. 10,789,835 to Kislovskiy et al (hereafter Kislovskiy). Kislovskiy discloses using data generated from AV drive logs in simulations to determine expected outcomes and responses.
While the closest prior art above teaches the various aspects of the claimed invention individually, the combination of these references are not obvious in such a way that they would have been obvious to one of ordinary skill in the art at the time of invention. Specifically, Bin-Nun in view of Mui and further in view of CPB and even further in view of Kislovskiy does not explicitly disclose determining performance metrics within a road segment while excluding specific categories of disengagements. Therefore, the claims are rendered novel and non-obvious over the prior art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bondor – US 10482003 – discussing determining data regarding categorization of disengagements using simulations.
Kwant – US 2019/0102692 – discussing receiving and analyzing autonomous vehicle operating data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID G GODBOLD whose telephone number is (571)272-5036. The examiner can normally be reached M-F 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Shannon S Campbell can be reached at 571-272-5587. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID G. GODBOLD/Examiner, Art Unit 3628
/RUPANGINI SINGH/Primary Examiner, Art Unit 3628