DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Non-final Office Action on the merits. Claims 1-15 are currently pending and are addressed below.
Priority
Acknowledgement is made of applicant’s claim of priority for foreign application EP24160446.1, filed 02/29/2024.
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 02/25/2025 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement(s) is/are being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
[0003] recites “…an advanced driver-assistance systems (ADASs) and/or autonomous driving (AD) systems…”, in which the underlined portion appears to be grammatically incorrect.
[0005] recites “A problem can. arise…”, in which the period after “can” appears to be a typographical error.
[0062] recites “…which describes any aspect of the input data, such that its visual content, what it depicts, etc.”, in which the underlined portion appears to be grammatically incorrect.
[0062] recites “…can also applied…”, which appears to be grammatically incorrect.
[0106] recites “…underlying the one or more embeddings…”, which appears to be grammatically incorrect.
[0107] recites “…scenario description be used…”, which appears to be grammatically incorrect.
[0120] recites “…may be configured execute…”, which appears to be grammatically incorrect.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 and 7-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Independent Claim 1:
Step 1: Claim 1 is directed to a method for providing data-insight for development of an ADS of a vehicle (i.e., a process). Therefore, claim 1 is within at least one of the four statutory categories.
Step 2A Prong 1: Regarding Prong 1 of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity and/or c) mental processes.
Independent claim 1 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 1 recites:
A computer-implemented method for providing data-insight for development of an automated driving system (ADS) of a vehicle, the method comprising:
obtaining sensor data pertaining to a driving scenario, the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle;
monitoring a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest; and
in response to determining at least one of the one or more scenario triggers being fulfilled: generating, by a description generator network, a scenario description, based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains; and
storing the generated scenario description.
The examiner submits that the foregoing bolded limitations constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitations in the human mind. For example, the limitation “obtaining sensor data pertaining to a driving scenario” in the context of this claim encompasses mentally obtaining (i.e., reading) sensor data. The limitation “monitoring a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest” encompasses mentally monitoring a fulfillment of a scenario trigger. Lastly, the limitation “in response to determining at least one of the one or more scenario triggers being fulfilled: generating…a scenario description, based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains” encompasses mentally generating a textual scenario description based on obtained sensor data.
Step 2A Prong 2: Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A computer-implemented method for providing data-insight for development of an automated driving system (ADS) of a vehicle, the method comprising:
obtaining sensor data pertaining to a driving scenario, the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle;
monitoring a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest; and
in response to determining at least one of the one or more scenario triggers being fulfilled: generating, by a description generator network, a scenario description, based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains; and
storing the generated scenario description.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
The additional limitation(s) “the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle” and “storing the generated scenario description” are recited at a high level of generality and amount to mere data gathering and data storage, which are a form of insignificant extra-solution activity. Lastly, the additional limitations “computer-implemented”, ”for providing data-insight for development of an automated driving system (ADS) of a vehicle”, and “by a description generator network” merely describe how to generally “apply” the otherwise mental judgements in a generic vehicle control environment using generic computer components.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B: Regarding Step 2B of the 2019 PEG, representative independent claim 1 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements “computer-implemented”, ”for providing data-insight for development of an automated driving system (ADS) of a vehicle”, and “by a description generator network” are recited at a high-level of generality and amount to nothing more than applying the exception to a technological environment using generic computer components. The examiner also submits that the additional limitations “the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle” and “storing the generated scenario description” are insignificant extra-solution activity.
Further, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field. The additional limitations “the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle” and “storing the generated scenario description” are well-understood, routine, and conventional activity in light of MPEP 2106.05(d)(II) and the cases cited therein, including Intellectual Ventures I, LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016), TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610 (Fed. Cir. 2016), and OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015), which indicate that mere collection or receipt of data over a network is a well‐understood, routine, and conventional function when it is claimed in a merely generic manner.
Therefore, claim 1 is ineligible under 35 U.S.C §101.
Regarding Independent Claim 13:
Step 1: Claim 13 is directed to a computing device for providing data-insight for development of an ADS of a vehicle (i.e., a machine). Therefore, claim 14 is within at least one of the four statutory categories.
Step 2A Prong 1: Regarding Prong 1 of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity and/or c) mental processes.
Independent claim 14 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 14 recites:
A computing device for providing data-insight for development of an automated driving system of a vehicle, the computing device comprising control circuitry configured to:
obtain sensor data pertaining to a driving scenario, the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle;
monitor a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest; and
in response to determining at least one of the one or more scenario triggers being fulfilled: generate, by a description generator network, a scenario description, based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains; and
store the generated scenario description.
The examiner submits that the foregoing bolded limitations constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitations in the human mind. For example, the limitation “obtaining sensor data pertaining to a driving scenario” in the context of this claim encompasses mentally obtaining (i.e., reading) sensor data. The limitation “monitoring a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest” encompasses mentally monitoring a fulfillment of a scenario trigger. Lastly, the limitation “in response to determining at least one of the one or more scenario triggers being fulfilled: generating…a scenario description, based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains” encompasses mentally generating a textual scenario description based on obtained sensor data.
Step 2A Prong 2: Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A computing device for providing data-insight for development of an automated driving system of a vehicle, the computing device comprising control circuitry configured to:
obtain sensor data pertaining to a driving scenario, the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle;
monitor a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest; and
in response to determining at least one of the one or more scenario triggers being fulfilled: generate, by a description generator network, a scenario description, based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains; and
store the generated scenario description.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
The additional limitation(s) “the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle” and “storing the generated scenario description” are recited at a high level of generality and amount to mere data gathering and data storage, which are a form of insignificant extra-solution activity. Lastly, the additional limitations “for providing data-insight for development of an automated driving system (ADS) of a vehicle”, “the computing device comprising control circuitry”, and “by a description generator network” merely describe how to generally “apply” the otherwise mental judgements in a generic vehicle control environment using generic computer components.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B: Regarding Step 2B of the 2019 PEG, representative independent claim 1 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements “for providing data-insight for development of an automated driving system (ADS) of a vehicle”, “the computing device comprising control circuitry”, and “by a description generator network” are recited at a high-level of generality and amount to nothing more than applying the exception to a technological environment using generic computer components. The examiner also submits that the additional limitations “the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle” and “storing the generated scenario description” are insignificant extra-solution activity.
Further, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field. The additional limitations “the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle” and “storing the generated scenario description” are well-understood, routine, and conventional activity in light of MPEP 2106.05(d)(II) and the cases cited therein, including Intellectual Ventures I, LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016), TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610 (Fed. Cir. 2016), and OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015), which indicate that mere collection or receipt of data over a network is a well‐understood, routine, and conventional function when it is claimed in a merely generic manner.
Therefore, claim 13 is ineligible under 35 U.S.C §101.
Regarding Independent Claim 15:
Step 1: Claim 15 is directed to a system (i.e., a machine). Therefore, claim 15 is within at least one of the four statutory categories.
Step 2A Prong 1: Regarding Prong 1 of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity and/or c) mental processes.
Independent claim 15 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 15 recites:
A system comprising:
a fleet of vehicles equipped with an automated driving system, ADS; and
a server communicatively connected to the fleet of vehicles;
wherein a vehicle of the fleet of vehicles comprises control circuitry configured to: obtain sensor data pertaining to a driving scenario, the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle;
monitor a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest; and
in response to determining at least one of the one or more scenario triggers being fulfilled: generate, by a description generator network, a scenario description based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains; and
transmit the generated scenario description to the server;
wherein the server comprises control circuitry configured to: receive the generated scenario description; and
store the generated scenario description.
The examiner submits that the foregoing bolded limitations constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitations in the human mind. For example, the limitation “obtaining sensor data pertaining to a driving scenario” in the context of this claim encompasses mentally obtaining (i.e., reading) sensor data. The limitation “monitoring a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest” encompasses mentally monitoring a fulfillment of a scenario trigger. Lastly, the limitation “in response to determining at least one of the one or more scenario triggers being fulfilled: generating…a scenario description, based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains” encompasses mentally generating a textual scenario description based on obtained sensor data.
Step 2A Prong 2: Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A system comprising:
a fleet of vehicles equipped with an automated driving system, ADS; and
a server communicatively connected to the fleet of vehicles;
wherein a vehicle of the fleet of vehicles comprises control circuitry configured to: obtain sensor data pertaining to a driving scenario, the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle;
monitor a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest; and
in response to determining at least one of the one or more scenario triggers being fulfilled: generate, by a description generator network, a scenario description based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains; and
transmit the generated scenario description to the server;
wherein the server comprises control circuitry configured to: receive the generated scenario description; and
store the generated scenario description.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
The additional limitation(s) “the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle”, “transmit the generated scenario description to the server”, “wherein the server comprises control circuitry configured to: receive the generated scenario description”, and “store the generated scenario description” are recited at a high level of generality and amount to mere data gathering and data storage, which are a form of insignificant extra-solution activity. Lastly, the additional limitations “a fleet of vehicles equipped with an automated driving system, ADS”, “a server communicatively connected to the fleet of vehicles”, “wherein a vehicle of the fleet of vehicles comprises control circuitry configured to”, “by a description generator network”, and “wherein the server comprises control circuitry configured to” merely describe how to generally “apply” the otherwise mental judgements in a generic vehicle control environment using generic computer components.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B: Regarding Step 2B of the 2019 PEG, representative independent claim 1 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements “a fleet of vehicles equipped with an automated driving system, ADS”, “a server communicatively connected to the fleet of vehicles”, “wherein a vehicle of the fleet of vehicles comprises control circuitry configured to”, “by a description generator network”, and “wherein the server comprises control circuitry configured to” are recited at a high-level of generality and amount to nothing more than applying the exception to a technological environment using generic computer components. The examiner also submits that the additional limitations “the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle”, “transmit the generated scenario description to the server”, “wherein the server comprises control circuitry configured to: receive the generated scenario description”, and “store the generated scenario description” are insignificant extra-solution activity.
Further, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field. The additional limitations “the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle”, “transmit the generated scenario description to the server”, “wherein the server comprises control circuitry configured to: receive the generated scenario description”, and “store the generated scenario description” are well-understood, routine, and conventional activity in light of MPEP 2106.05(d)(II) and the cases cited therein, including Intellectual Ventures I, LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016), TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610 (Fed. Cir. 2016), and OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015), which indicate that mere collection or receipt of data over a network is a well‐understood, routine, and conventional function when it is claimed in a merely generic manner.
Therefore, claim 15 is ineligible under 35 U.S.C §101.
Dependent Claims
Dependent claim(s) 7-12 and 14 do not recite any further limitations that cause the claims to be patent eligible. Rather, the limitations of the dependent claim(s) are directed toward additional aspects of the judicial exception. Dependent claim(s) 7 is further directed to transmitting the generated scenario description to a remote server, which is a form of insignificant extra-solution activity, dependent claim(s) 8 further describes the sensor data, dependent claims 9-10 are further directed to the abstract idea of determining the fulfillment of one or more scenario triggers, dependent claim 11 is further directed to obtaining further sensor data, which is a form of insignificant extra-solution activity, the abstract idea of generating a pre-scenario description or a post-scenario description based on the further sensor data, and storing the pre-scenario description or post-scenario description with the scenario description, which is a form of insignificant extra-solution activity. Dependent claims 12 and 14 are further directed to the additional elements of a non-transitory computer readable storage medium and a vehicle equipped with an automated driving system, sensors, and a computing device, which are recited at a high level of generality and merely “apply” the otherwise mental judgments in a technological environment using generic computer components. Therefore, dependent claim(s) 7-12 and 14 is/are not patent eligible under the same rationale as provided for in the rejection of claims 1 and 7.
Therefore, claim(s) 1 and 7-15 is/are ineligible under 35 U.S.C. §101.
Eligible Claims
Dependent claims 2 and 5-6 are found to have a practical application via generating sensor data embeddings by processing sensor data through an embedding network that outputs corresponding sensor data embeddings in a multi-dimensional space, as recited in claim 2, generating trigger embeddings by processing each fulfilled scenario trigger through a trigger embedding network that outputs corresponding trigger embeddings in the multi-dimensional space, as recited in claim 5, and generating ADS data embeddings by processing ADS data through an ADS data embedding network that outputs corresponding ADS data embeddings in the multi-dimensional space, as recited in claim 6, and are thus considered eligible. Therefore, claims 3-4, which depend on claim 2, are also considered eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 7-9, and 11-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gyllenhammar of WO 2022258203 A1, published 12/15/2022, hereinafter “Gyllenhammar”, in view of Schlict of DE 102018215351 A1, published 03/12/2020, hereinafter “Schlict”.
Regarding claim 1, Gyllenhammar teaches:
A computer-implemented method for providing data-insight for development of an automated driving system (ADS) of a vehicle, the method comprising: (See at least pg. 3, lines 1-4: “It is therefore an object of the present invention to provide solutions for facilitating development, testing, and/or validation of perception features or functions for autonomous and semi-autonomous vehicles in order to continuously be able to provide safer and more performant systems.”)
obtaining sensor data pertaining to a driving scenario, the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle; (See at least pgs. 12-13, lines 24-30 & 1-2: “A "perception system" as used herein, may be understood as software and/or hardware configured to acquire (raw) sensor data from one or more on-board sensors such as cameras, LIDARs and RADARs, ultrasonic sensors, and convert this (raw) sensor data into scene understanding including state estimations and/or predictions thereof (i.e. to a "worldview"). Accordingly, a perception system is configured to generate perception output/data that is indicative of one or more perceptive parameters (e.g. object position, object dimension, object classification, lane tracking, road geometry estimation, free-space estimation, etc.) based on one or more perception models (e.g. one or more neural networks) and sensor data serving as input…”)
monitoring a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest; and (See at least pg. 17, lines 18-27: “Further, the method S200 comprises, at the user-device, displaying S205 via the display apparatus, a graphical user interface comprising a graphical representation of at least a portion of the surrounding environment of the vehicle based on the filtered worldview. Then, a user annotation event is obtained S206, at the user-device 200, from the input device of the user-device 200. The obtained S206 user annotation event is indicative of a user interaction with the displayed graphical representation. Moreover, in some embodiments, the method S200 may further comprise a step of generating, at the user device 200, a prompter indicative of an instruction for a user of the user-device 200 to annotate the displayed graphical representation of the surrounding environment of the vehicle, i.e. to perform a user annotation event”)
in response to determining at least one of the one or more scenario triggers being fulfilled: generating, by a description generator network, a scenario description, based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains; and (See at least pgs. 17-18, lines 27-30 & 1-5: “Still further, the method S200 comprises forming S207, at the user device 200, an annotated worldview indicative of at least one annotated perceptive parameter in the first set of perception data (generated in the vehicle) based on the obtained user annotation event and the filtered worldview. It should be understood that the "annotated worldview" is indicative of at least one annotated perceptive parameter in the filtered worldview, which in turn is indicative of at least one annotated perceptive parameter in the first set of perception data, since the filtered worldview is a processed form of the first set of perception data. Then, the formed S207 annotated worldview is transmitted S208 from the user-device 200 to the vehicle 100” & pg. 27, lines 14-19: “Further, after the obtained user annotation event, the annotation engine 214 is configured to form an annotated worldview indicative of at least one annotated perceptive parameter in the first set of perception data generated by the perception system 111 of the vehicle based on the obtained user annotation event and the filtered worldview. The annotated worldview is then transmitted from the user-device 200 to the vehicle 100, where it is processed by an evaluation engine 116.”)
Gyllenhammar does not explicitly teach:
storing the generated scenario description.
Schlict teaches:
storing the generated scenario description. (See at least [0016]: “In general, vehicle sensor data and/or driving scenario data can be continuously recorded during a journey and saved, for example, at regular time intervals and/or at least when a change has been detected. It is possible to store the driving scenario data and/or vehicle sensor data at least temporarily in a vehicle-bound storage device and, for example, only subsequently transfer them to a vehicle-independent storage device…”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Gyllenhammar’s method with Schlict’s technique of storing the generated scenario description. Doing so would be obvious to “to validate and design automated driving functions based on these driving scenarios” (See [0052] of Schlict).
Regarding claim 7, Gyllenhammar and Schlict in combination teach all the limitations of claim 1 as discussed above.
Gyllenhammar additionally teaches:
further comprising transmitting the generated scenario description to a remote server. (See at least pgs. 15-16, lines 26-31 & 1-2: “…This annotated worldview may subsequently be transmitted to the vehicle 100 from the user-device 200, where it may be used to update S109 one or more model parameters of a perception model employed by the vehicle's 100 perception system. However, in some embodiments, the annotated worldview may not be suitable to be consumed directly by the in-vehicle training algorithm, but may instead be indicative of a rare scenario or edge-case. In such cases, the relevant dataset (sensor data, perception data, and annotated worldview) may instead be transmitted to a back-office for manual analysis…”)
Regarding claim 8, Gyllenhammar and Schlict in combination teach all the limitations of claim 1 as discussed above.
Gyllenhammar additionally teaches:
wherein the sensor data comprises at least two sensor data types. (See at least pg. 12, lines 24-27: “A "perception system" as used herein, may be understood as software and/or hardware 25 configured to acquire (raw) sensor data from one or more on-board sensors such as cameras, LIDARs and RADARs, ultrasonic sensors, and convert this (raw) sensor data into scene-understanding including state estimations and/or predictions thereof (i.e. to a "worldview")…”)
Regarding claim 9, Gyllenhammar and Schlict in combination teach all the limitations of claim 1 as discussed above.
Gyllenhammar additionally teaches:
wherein determining the fulfilment of the one or more scenario triggers of the driving scenario is based on the sensor data depicting at least part of the surrounding environment of the vehicle. (See at least pg. 20, lines 19-30: “However, in some embodiments, the method S200 further comprises (at least temporarily) 20 storing S203, in a memory device of the vehicle, the filtered worldview, and then transmitting S204 the stored filtered worldview to the user-device 200. Buffering or storing the filtered world-view may provide the advantage of being able to provide more relevant data for annotation (e.g. scenarios where the vehicle's perception system had a hard time to accurately make specific estimations/predictions, or scenarios deemed to be informative enough to be added to training dataset) which may further include the efficiency of the whole annotation process as described herein. Thus, one may store "more relevant" images for annotation for any suitable period of time as the vehicle's perception system encounters interesting scenarios, and then when suitable (e.g. when a connected user-device is present and has requested data for annotation) the stored filtered worldview (i.e. the "more relevant" images) is transmitted S204 to the user-device 200…”)
Regarding claim 11, Gyllenhammar and Schlict in combination teach all the limitations of claim 1 as discussed above.
Schlict additionally teaches:
further comprising: obtaining further sensor data depicting at least a portion of the surrounding environment of the vehicle at a time before and/or after a time of occurrence of the driving scenario; (See at least [0051]: “In the storage device 18, after the completion of a journey of the vehicle 2, there are preferably several individual data records relating to driving scenarios that have passed through. It is also possible that several vehicles 2 transmit data sets to the arrangement 3 and that the information collection stored there is thus composed of a plurality of driving scenario data from different vehicles 2. The information collection can be continuously supplemented over a longer period (for example, several weeks or months)…” & [0054]: “In an optional step S5, it is checked whether a predetermined evaluation condition is met (for example, the expiry of a predetermined time interval, the presence of a minimum number of driving scenario data records, or the presence of a predetermined user input). If this is not the case, the process returns to step S1 to determine further data records…”)
generating a pre-scenario description and/or a post-scenario description based on the further sensor data; and (See at least [0051]: “…Furthermore, the information collected can be statistically evaluated by the evaluation unit 20, in particular to determine frequently recurring driving scenarios or, more generally, a driving scenario frequency distribution. Additionally or alternatively, frequency distributions and/or frequent correlations of the individual driving scenario parameter values can also be determined…”)
storing the pre-scenario description and/or post-scenario description together with the scenario description. (See at least [0051]: “…The result of this evaluation can also be stored in storage device 18.”)
Regarding claim 12, Gyllenhammar and Schlict in combination teach all the limitations of claim 1 as discussed above.
Gyllenhammar additionally teaches:
A non-transitory computer readable storage medium storing instructions which, when executed by a computer, causes the computer to carry out the method according to claim 1. (See at least pg. 22, lines 21-22: “…The vehicle 100 and user-device 200 comprise control circuitry configured to perform the functions of the methods disclosed herein, where the functions may be included in a non-transitory computer-readable storage medium or other computer program product configured for execution by the control circuitry…”)
Regarding claim 13, Gyllenhammar teaches:
A computing device for providing data-insight for development of an automated driving system of a vehicle, the computing device comprising control circuitry configured to: (See at least pg. 3, lines 1-4: “It is therefore an object of the present invention to provide solutions for facilitating development, testing, and/or validation of perception features or functions for autonomous and semi-autonomous vehicles in order to continuously be able to provide safer and more performant systems.”)
obtain sensor data pertaining to a driving scenario, the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle; (See at least pgs. 12-13, lines 24-30 & 1-2: “A "perception system" as used herein, may be understood as software and/or hardware configured to acquire (raw) sensor data from one or more on-board sensors such as cameras, LIDARs and RADARs, ultrasonic sensors, and convert this (raw) sensor data into scene understanding including state estimations and/or predictions thereof (i.e. to a "worldview"). Accordingly, a perception system is configured to generate perception output/data that is indicative of one or more perceptive parameters (e.g. object position, object dimension, object classification, lane tracking, road geometry estimation, free-space estimation, etc.) based on one or more perception models (e.g. one or more neural networks) and sensor data serving as input…”)
monitor a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest; and (See at least pg. 17, lines 18-27: “Further, the method S200 comprises, at the user-device, displaying S205 via the display apparatus, a graphical user interface comprising a graphical representation of at least a portion of the surrounding environment of the vehicle based on the filtered worldview. Then, a user annotation event is obtained S206, at the user-device 200, from the input device of the user-device 200. The obtained S206 user annotation event is indicative of a user interaction with the displayed graphical representation. Moreover, in some embodiments, the method S200 may further comprise a step of generating, at the user device 200, a prompter indicative of an instruction for a user of the user-device 200 to annotate the displayed graphical representation of the surrounding environment of the vehicle, i.e. to perform a user annotation event”)
in response to determining at least one of the one or more scenario triggers being fulfilled: generate, by a description generator network, a scenario description, based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains; and (See at least pgs. 17-18, lines 27-30 & 1-5: “Still further, the method S200 comprises forming S207, at the user device 200, an annotated worldview indicative of at least one annotated perceptive parameter in the first set of perception data (generated in the vehicle) based on the obtained user annotation event and the filtered worldview. It should be understood that the "annotated worldview" is indicative of at least one annotated perceptive parameter in the filtered worldview, which in turn is indicative of at least one annotated perceptive parameter in the first set of perception data, since the filtered worldview is a processed form of the first set of perception data. Then, the formed S207 annotated worldview is transmitted S208 from the user-device 200 to the vehicle 100” & pg. 27, lines 14-19: “Further, after the obtained user annotation event, the annotation engine 214 is configured to form an annotated worldview indicative of at least one annotated perceptive parameter in the first set of perception data generated by the perception system 111 of the vehicle based on the obtained user annotation event and the filtered worldview. The annotated worldview is then transmitted from the user-device 200 to the vehicle 100, where it is processed by an evaluation engine 116.”)
Gyllenhammar does not explicitly teach:
store the generated scenario description.
Schlict teaches:
store the generated scenario description. (See at least [0016]: “In general, vehicle sensor data and/or driving scenario data can be continuously recorded during a journey and saved, for example, at regular time intervals and/or at least when a change has been detected. It is possible to store the driving scenario data and/or vehicle sensor data at least temporarily in a vehicle-bound storage device and, for example, only subsequently transfer them to a vehicle-independent storage device…”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Gyllenhammar’s method with Schlict’s technique of storing the generated scenario description. Doing so would be obvious to “to validate and design automated driving functions based on these driving scenarios” (See [0052] of Schlict).
Regarding claim 14, Gyllenhammar and Schlict in combination teach all the limitations of claim 13 as discussed above.
Gyllenhammar additionally teaches:
A vehicle equipped with an automated driving system comprising: one or more sensors; and a computing device according to claim 13. (See at least pg. 15, lines 13-17: “…In some embodiments, where the vehicle 100 is an ADS-equipped vehicle, the perception stack may be a part of the vehicle's 100 ADS. Once, the required data/information has been retrieved S104, the perception data is processed S105 in order to form a filtered worldview to be transmitted to the user-device 200 to elicit annotations from the user…”)
Regarding claim 15, Gyllenhammar teaches:
A system comprising: a fleet of vehicles equipped with an automated driving system, ADS; and a server communicatively connected to the fleet of vehicles; (See at least Abstract: “Disclosed herein are methods, apparatuses and systems that allow the passengers of an ADS- quipped vehicle to supply weak (i.e. "inaccurate") annotations to the vehicle-platform by streaming (or otherwise transmitting) perception data (e.g. images or various combinations of other sensor data) to the passenger's mobile devices to elicit annotations” & pg. 38, lines 16-21: “…Accordingly, the remote system 300 may form a set of globally updated parameters, and pushes a "global update" to the fleet of vehicles…”)
wherein a vehicle of the fleet of vehicles comprises control circuitry configured to: obtain sensor data pertaining to a driving scenario, the sensor data being captured by one or more sensors of the vehicle and depicting at least part of a surrounding environment of the vehicle; (See at least pgs. 12-13, lines 24-30 & 1-2: “A "perception system" as used herein, may be understood as software and/or hardware configured to acquire (raw) sensor data from one or more on-board sensors such as cameras, LIDARs and RADARs, ultrasonic sensors, and convert this (raw) sensor data into scene understanding including state estimations and/or predictions thereof (i.e. to a "worldview"). Accordingly, a perception system is configured to generate perception output/data that is indicative of one or more perceptive parameters (e.g. object position, object dimension, object classification, lane tracking, road geometry estimation, free-space estimation, etc.) based on one or more perception models (e.g. one or more neural networks) and sensor data serving as input…”)
monitor a fulfillment of one or more scenario triggers of the driving scenario, wherein fulfillment of the one or more scenario triggers is indicative of the driving scenario being a driving scenario of interest; and (See at least pg. 17, lines 18-27: “Further, the method S200 comprises, at the user-device, displaying S205 via the display apparatus, a graphical user interface comprising a graphical representation of at least a portion of the surrounding environment of the vehicle based on the filtered worldview. Then, a user annotation event is obtained S206, at the user-device 200, from the input device of the user-device 200. The obtained S206 user annotation event is indicative of a user interaction with the displayed graphical representation. Moreover, in some embodiments, the method S200 may further comprise a step of generating, at the user device 200, a prompter indicative of an instruction for a user of the user-device 200 to annotate the displayed graphical representation of the surrounding environment of the vehicle, i.e. to perform a user annotation event”)
in response to determining at least one of the one or more scenario triggers being fulfilled: generate, by a description generator network, a scenario description based at least on the obtained sensor data pertaining to the driving scenario and/or based on ADS data outputted from the ADS having processed the sensor data, wherein the scenario description comprises textual data about the driving scenario to which the obtained sensor data pertains; and (See at least pgs. 17-18, lines 27-30 & 1-5: “Still further, the method S200 comprises forming S207, at the user device 200, an annotated worldview indicative of at least one annotated perceptive parameter in the first set of perception data (generated in the vehicle) based on the obtained user annotation event and the filtered worldview. It should be understood that the "annotated worldview" is indicative of at least one annotated perceptive parameter in the filtered worldview, which in turn is indicative of at least one annotated perceptive parameter in the first set of perception data, since the filtered worldview is a processed form of the first set of perception data. Then, the formed S207 annotated worldview is transmitted S208 from the user-device 200 to the vehicle 100” & pg. 27, lines 14-19: “Further, after the obtained user annotation event, the annotation engine 214 is configured to form an annotated worldview indicative of at least one annotated perceptive parameter in the first set of perception data generated by the perception system 111 of the vehicle based on the obtained user annotation event and the filtered worldview. The annotated worldview is then transmitted from the user-device 200 to the vehicle 100, where it is processed by an evaluation engine 116.”)
transmit the generated scenario description to the server; (See at least pgs. 15-16, lines 26-31 & 1-2: “…This annotated worldview may subsequently be transmitted to the vehicle 100 from the user-device 200, where it may be used to update S109 one or more model parameters of a perception model employed by the vehicle's 100 perception system. However, in some embodiments, the annotated worldview may not be suitable to be consumed directly by the in-vehicle training algorithm, but may instead be indicative of a rare scenario or edge-case. In such cases, the relevant dataset (sensor data, perception data, and annotated worldview) may instead be transmitted to a back-office for manual analysis…”)
wherein the server comprises control circuitry configured to: receive the generated scenario description; and (See at least pgs. 32-33, lines 30 & 1-5: “…Then, if the determined level of matching is below a threshold (or a level of "mismatching" is above a threshold), the stored sensor data, the stored first set of perception data and the annotated worldview are transmitted from the vehicle to a remote entity 300. The transmitted data - which may be construed as edge-case data or otherwise important data for developing performant perception features - may then be manually analysed at a "back-office".” See also pg. 21, lines 16-20.)
Gyllenhammar does not explicitly teach:
store the generated scenario description.
Schlict teaches:
store the generated scenario description. (See at least [0016]: “In general, vehicle sensor data and/or driving scenario data can be continuously recorded during a journey and saved, for example, at regular time intervals and/or at least when a change has been detected. It is possible to store the driving scenario data and/or vehicle sensor data at least temporarily in a vehicle-bound storage device and, for example, only subsequently transfer them to a vehicle-independent storage device…”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Gyllenhammar’s method with Schlict’s technique of storing the generated scenario description. Doing so would be obvious to “to validate and design automated driving functions based on these driving scenarios” (See [0052] of Schlict).
Claim(s) 2 and 5-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gyllenhammar in view of Schlict and further in view of Zhao of US 20240367685 A1, filed 05/02/2023, hereinafter “Zhao”.
Regarding claim 2, Gyllenhammar and Schlict in combination teach all the limitations of claim 1 as discussed above.
Gyllenhammar and Schlict in combination do not explicitly teach:
wherein generating the scenario description comprises: generating one or more sensor data embeddings for the obtained sensor data pertaining to the driving scenario, wherein the sensor data embeddings are generated by processing the sensor data through one or more sensor data embedding networks that have been trained to process sensor data and to output a corresponding sensor data embedding in a multi-dimensional space; and
Zhao teaches:
wherein generating the scenario description comprises: generating one or more sensor data embeddings for the obtained sensor data pertaining to the driving scenario, wherein the sensor data embeddings are generated by processing the sensor data through one or more sensor data embedding networks that have been trained to process sensor data and to output a corresponding sensor data embedding in a multi-dimensional space; and (See at least [0048]: “In some implementations, the encoder neural network 220 can be configured through training to generate intermediate feature maps based on the sensor inputs 211. The intermediate feature maps can represent the sensor inputs 211 in any appropriate numerical format. The intermediate feature maps can take the form of embeddings in an embedding space. An “embedding” as used in this specification is a vector of numeric values, e.g., floating point values or other values, having a pre-determined dimensionality. The space of possible vectors having the pre-determined dimensionality is referred to as the “embedding space” & [0077]: “The REM query system 600 maintains a library of embeddings 630 at one or more data stores. Each embedding can be a vector of numeric values. The library of embeddings 630 include a plurality of embeddings generated by neural networks that correspond respectively to different types of rareness from processing the historical sensor inputs stored in a driving log 640…”)
inputting the one or more sensor data embeddings to the description generator network having been trained to process sensor data embeddings and output a corresponding scenario description for the driving scenario to which the sensor data pertains, wherein the description generator network has been trained in association with at least one of the one or more sensor data embedding networks so as to relate to the same multi-dimensional space. (See at least [0085]: “Additionally or alternatively, the REM query system 600 can receive a query that includes text 604 and use the selection engine 620 to retrieve one or more sensor inputs 624 based on the text 604. For example, the text 604 may include phrases of one or more terms that are descriptive of user-specified contents that should appear in sensor inputs. The REM query system 600 can generate a textual embedding for the text 604 and then select, for the textual embedding and from the library of embeddings 630, one or more embeddings that are similar to the textual embedding for the text 604. For each embedding, the system can identify from the driving log 640 a historical sensor input 624 (referred to below as a “retrieved historical sensor input”) from which the embedding has been generated. The textual embedding can be generated by a textual embedding neural network which has been jointly trained with the prediction neural networks to process text, e.g., the text in text-sensor input pairs where the text in each pair describes the contents of the sensor input in the pair, to generate textual embeddings, e.g., in a same embedding space as the plurality of embeddings from the library of embeddings 630.” See also [0113] regarding the textual embeddings being in the same embedding space as the historical sensor input embeddings.)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Gyllenhammar and Schlict’s method with the teachings of Zhao discussed above. Doing so would be obvious “to improve (or evaluate) the performance of a prediction neural network that is configured to generate agent behavior prediction data” (See [0065] of Zhao).
Regarding claim 5, Gyllenhammar and Schlict in combination teach all the limitations of claim 1 as discussed above.
Gyllenhammar and Schlict in combination do not explicitly teach:
wherein generating the scenario description further comprises: generating one or more trigger embeddings for the at least one scenario trigger being fulfilled, wherein the one or more trigger embeddings are generated by processing each fulfilled scenario trigger through a trigger embedding network having been trained to process scenario triggers and to output a corresponding trigger embedding in the multi-dimensional space, and wherein the trigger embedding network have been trained in association with at least one of the one or more sensor data embedding networks so as to relate to the same multi-dimensional space; and
inputting the one or more trigger embeddings to the description generator network.
Zhao teaches:
wherein generating the scenario description further comprises: generating one or more trigger embeddings for the at least one scenario trigger being fulfilled, wherein the one or more trigger embeddings are generated by processing each fulfilled scenario trigger through a trigger embedding network having been trained to process scenario triggers and to output a corresponding trigger embedding in the multi-dimensional space, and wherein the trigger embedding network have been trained in association with at least one of the one or more sensor data embedding networks so as to relate to the same multi-dimensional space; and (See at least [0084]: “One or more embeddings (referred to below as “query embeddings”) will be generated by the prediction neural networks during the processing of the sensor inputs 602 to output the rareness scores 612. The REM query system 600 thus can use a selection engine 620 to select, from the library of embeddings 630, one or more similar embeddings to the query embeddings generated from the sensor inputs 602 that are referenced in the query received by the system 600. Just like how the rareness scores are generated, the REM query system 600 can select, for each query embedding, one or more similar embeddings associated with each different type of rareness from the library of embeddings 630. For each similar embedding, the system can identify from the driving log 640 a historical sensor input 622 (referred to below as a “similar historical sensor input”) from which the similar embedding has been generated.” See also [0117-0118] regarding jointly training the textual embedding neural network and the prediction neural networks.)
inputting the one or more trigger embeddings to the description generator network. (See at least [0085]: “Additionally or alternatively, the REM query system 600 can receive a query that includes text 604 and use the selection engine 620 to retrieve one or more sensor inputs 624 based on the text 604. For example, the text 604 may include phrases of one or more terms that are descriptive of user-specified contents that should appear in sensor inputs. The REM query system 600 can generate a textual embedding for the text 604 and then select, for the textual embedding and from the library of embeddings 630, one or more embeddings that are similar to the textual embedding for the text 604. For each embedding, the system can identify from the driving log 640 a historical sensor input 624 (referred to below as a “retrieved historical sensor input”) from which the embedding has been generated. The textual embedding can be generated by a textual embedding neural network which has been jointly trained with the prediction neural networks to process text, e.g., the text in text-sensor input pairs where the text in each pair describes the contents of the sensor input in the pair, to generate textual embeddings, e.g., in a same embedding space as the plurality of embeddings from the library of embeddings 630.” See also [0113] regarding the textual embeddings being in the same embedding space as the historical sensor input embeddings.)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Gyllenhammar and Schlict’s method with the teachings of Zhao discussed above. Doing so would be obvious “to improve (or evaluate) the performance of a prediction neural network that is configured to generate agent behavior prediction data” (See [0065] of Zhao).
Regarding claim 6, Gyllenhammar and Schlict in combination teach all the limitations of claim 1 as discussed above.
Gyllenhammar and Schlict in combination do not explicitly teach:
wherein generating the scenario description comprises: generating one or more ADS data embeddings from the ADS data, wherein the one or more ADS data embeddings are generated by processing the ADS data through one or more ADS data embedding networks that have been trained to process ADS data and to output a corresponding ADS data embedding in a multi-dimensional space; and
inputting the one or more ADS data embeddings to the description generator network having been trained to process ADS data embeddings and output a corresponding scenario description for the driving scenario, wherein the description generator network has been trained in association with at least one of the one or more ADS data embedding networks so as to relate to the same multi-dimensional space.
Zhao teaches:
wherein generating the scenario description comprises: generating one or more ADS data embeddings from the ADS data, wherein the one or more ADS data embeddings are generated by processing the ADS data through one or more ADS data embedding networks that have been trained to process ADS data and to output a corresponding ADS data embedding in a multi-dimensional space; and (See at least [0048]: “In some implementations, the encoder neural network 220 can be configured through training to generate intermediate feature maps based on the sensor inputs 211. The intermediate feature maps can represent the sensor inputs 211 in any appropriate numerical format. The intermediate feature maps can take the form of embeddings in an embedding space. An “embedding” as used in this specification is a vector of numeric values, e.g., floating point values or other values, having a pre-determined dimensionality. The space of possible vectors having the pre-determined dimensionality is referred to as the “embedding space” & [0077]: “The REM query system 600 maintains a library of embeddings 630 at one or more data stores. Each embedding can be a vector of numeric values. The library of embeddings 630 include a plurality of embeddings generated by neural networks that correspond respectively to different types of rareness from processing the historical sensor inputs stored in a driving log 640…”)
inputting the one or more ADS data embeddings to the description generator network having been trained to process ADS data embeddings and output a corresponding scenario description for the driving scenario, wherein the description generator network has been trained in association with at least one of the one or more ADS data embedding networks so as to relate to the same multi-dimensional space. (See at least [0085]: “Additionally or alternatively, the REM query system 600 can receive a query that includes text 604 and use the selection engine 620 to retrieve one or more sensor inputs 624 based on the text 604. For example, the text 604 may include phrases of one or more terms that are descriptive of user-specified contents that should appear in sensor inputs. The REM query system 600 can generate a textual embedding for the text 604 and then select, for the textual embedding and from the library of embeddings 630, one or more embeddings that are similar to the textual embedding for the text 604. For each embedding, the system can identify from the driving log 640 a historical sensor input 624 (referred to below as a “retrieved historical sensor input”) from which the embedding has been generated. The textual embedding can be generated by a textual embedding neural network which has been jointly trained with the prediction neural networks to process text, e.g., the text in text-sensor input pairs where the text in each pair describes the contents of the sensor input in the pair, to generate textual embeddings, e.g., in a same embedding space as the plurality of embeddings from the library of embeddings 630.” See also [0113] regarding the textual embeddings being in the same embedding space as the historical sensor input embeddings.)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Gyllenhammar and Schlict’s method with the teachings of Zhao discussed above. Doing so would be obvious “to improve (or evaluate) the performance of a prediction neural network that is configured to generate agent behavior prediction data” (See [0065] of Zhao).
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gyllenhammar in view of Schlict and Zhao and further in view of Gao of US 20220164350 A1, filed 11/25/2020, hereinafter “Gao”.
Regarding claim 3, Gyllenhammar, Schlict, and Zhao in combination teach all the limitations of claim 2 as discussed above.
Zhao additionally teaches:
wherein the one or more sensor data embedding networks comprise a plurality of sensor data embedding networks including one sensor data embedding network for a corresponding sensor data type of the vehicle, (See at least [0041]: “To generate the various prediction data 114 from the sensor data 110, the prediction subsystem 112 implements trained neural networks that are each configured to process inputs derived from the sensor data 110 in accordance with trained parameters of the neural network to generate respective outputs that are included in the prediction data 114. For example, the neural networks can include one or more object detector or classifier neural networks that are configured to process the sensor data 110 to generate detection or classification outputs with respect to the objects depicted in the sensor data 110, one or more trajectory prediction neural networks that are configured to process the sensor data 110 to generate a respective predicted trajectory for an agent depicted in the sensor data 110, one or more agent intent prediction models that are configured to process the sensor data 110 to generate intent predictions for an agent depicted in the sensor data 110, and so on.”)
Gyllenhammar, Schlict, and Zhao in combination do not explicitly teach:
wherein the plurality of sensor data embedding networks comprises a first sensor data embedding network trained to process a first sensor data type and to output a corresponding sensor data embedding, and
a second sensor data embedding network trained to process a second sensor data type and to output a corresponding sensor data embedding, and
wherein the first sensor data embedding network has been trained in association with the second sensor data embedding network such that a sensor embedding generated by the first sensor data embedding network and a sensor embedding generated by the second sensor data embedding network point towards the same point within the multi-dimensional space when the two-sensor data embeddings are contextually, spatially and/or temporally related.
Gao teaches:
wherein the plurality of sensor data embedding networks comprises a first sensor data embedding network trained to process a first sensor data type and to output a corresponding sensor data embedding, and a second sensor data embedding network trained to process a second sensor data type and to output a corresponding sensor data embedding, and (See at least [0056-0057]: “In the embedding generation step 214, the system processes the extracted portion of the sensor sample using each of the one or more embedding neural networks to generate one or more embeddings of the sensor sample. As described above, when the set of embedding neural networks includes multiple neural networks, the different embeddings can represent different properties or characteristics of the sensor sample. For example, a vehicle intent neural network can include a vehicle appearance embedding neural network, a trajectory feature embedding neural network, and a context feature embedding neural network. The appearance embeddings of a vehicle can characterize an appearance of the vehicle as sensed by one or more sensors of a particular other vehicle in the environment. The appearance embeddings can include appearance feature maps generated from one or more camera images using a pretrained appearance embedding neural network. The appearance embeddings can include features of information identifying turn signal, heading, one or more tracked objects in the vicinity of the vehicle, object type, etc., of the vehicle. For example, the appearance embeddings can include features extracted from camera images that can indicate whether the left-turn signal light of a vehicle is currently on.”)
wherein the first sensor data embedding network has been trained in association with the second sensor data embedding network such that a sensor embedding generated by the first sensor data embedding network and a sensor embedding generated by the second sensor data embedding network point towards the same point within the multi-dimensional space when the two-sensor data embeddings are contextually, spatially and/or temporally related. (See at least [0093]: “For example, the system can generate and store a spatio-temporal embedding for each portion of the sensor samples in the sensor sample repository. A query portion of a query sensor sample can depict a scenario of passengers getting on a vehicle. The system can generate a spatio-temporal embedding of the query portion of the query sensor sample. The system can determine a subset of the sensor samples that depict interactive relationships between people and vehicles. The system can identify relevant sensor samples that have spatio-temporal embeddings that are closest to the query spatio-temporal embedding. The identified relevant sensor samples most likely also characterize scenarios of passengers getting on a vehicle.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Gyllenhammar, Schlict, and Zhao’s method with the teachings of Gao as discussed above. Doing so would be obvious “identify relevant sensor samples” (See [0093] of Gao).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gyllenhammar in view of Schlict and Zhao and further in view of Chiu of US 20230004797 A1, filed 02/11/2021, hereinafter “Chiu”.
Regarding claim 4, Gyllenhammar, Schlict, and Zhao in combination teach all the limitations of claim 2 as discussed above.
Gyllenhammar, Schlict, and Zhao in combination do not explicitly teach:
wherein the one or more sensor data embedding networks comprise a fused sensor data embedding network trained to process fused sensor data, wherein the fused sensor data comprises a fusion of at least two sensor data types.
Chiu teaches:
wherein the one or more sensor data embedding networks comprise a fused sensor data embedding network trained to process fused sensor data, wherein the fused sensor data comprises a fusion of at least two sensor data types. (See at least [0034]: “In the sensor data fusion system 100 of FIG. 1, images of a scene captured by at least two different types (e.g., different modalities) of sensors are communicated to a respective one of the at least one feature extraction module 110. In some embodiments, at the at least one feature extraction module 110, neural networks can be applied to the respective captured images of the at least two different types of sensors to extract the visual features of the images of the at least two different types of sensors…”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Gyllenhammar, Schlict, and Zhao’s method with Chiu’s fused sensor data embedding network trained to process fused sensor data of at least two sensor data types. Doing so would be obvious “to achieve more robust and accurate task performance” (See [0002] of Chiu).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gyllenhammar in view of Schlict and further in view of Harder of “Scenario2Vector: Scenario Description Language Based Embeddings for Traffic Situations”, published 05/19/2021, hereinafter “Harder”.
Regarding claim 10, Gyllenhammar and Schlict in combination teach all the limitations of claim 1 as discussed above.
Gyllenhammar and Schlict in combination do not explicitly teach:
wherein determining the fulfilment of the one or more scenario triggers of the driving scenario is based on one or more internal states of the vehicle at a time of occurrence of the driving scenario.
Harder teaches:
wherein determining the fulfilment of the one or more scenario triggers of the driving scenario is based on one or more internal states of the vehicle at a time of occurrence of the driving scenario. (See at least pg. 171: “…The Scenario2Vector SDL object is a tuple 𝑆 = ⟨𝐴𝑇, 𝐴𝐶, 𝑆𝐶⟩ with each element of𝑆 representing a list that contains information about the content of the scenario. The list 𝐴𝑇 contains the Actors present in the scenario, with elements such as “Ego” and “Light Ve hicle”. Similarly, 𝐴𝐶 contains temporal Actions such as “Accelerate”, “Brake”, ”Merge”, while 𝑆𝐶 contains Scene Elements such as “Stop Sign”, ”Intersection”, “Green Light”. Including Undefined values and variations of elements listed in Figure 2, there are 6 possible actor elements, 19 possible action elements, and 20 possible scene elements that are valid entries to these lists. These three lists taken together encapsulate the relevant information about the scenario.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Gyllenhammar and Schlict’s method with Harder’s technique of determining the fulfillment of the scenario triggers based on an internal state of the vehicle at a time of occurrence of the driving scenario. Doing so would be obvious to “encapsulate the relevant information about the scenario” (See pg. 171 of Harder).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20210403036 A1 is directed to receiving a search query and providing information describing an identified scenario in response to the search query.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Nikki Molina whose telephone number is (571) 272-5180. The examiner can normally be reached Monday - Thursday and alternate Fridays, 7:30-4:30 PT. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aniss Chad, can be reached on (571) 270-3832. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NIKKI MARIE M MOLINA/Examiner, Art Unit 3662
/ANISS CHAD/Supervisory Patent Examiner, Art Unit 3662
Prosecution Insights