DETAILED ACTION
This correspondence is responsive to the application filed on January 2, 2024. Claims 1-20 are pending in the case with claims 1, 8 and 14 in independent form.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Summary of Detailed Action
Claims 1-7 are rejected under 35 U.S.C. 112(b) as being indefinite
Claims 8-13 are rejected under 35 U.S.C. 112(b) as being indefinite
Claims 1-17 and 19-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1, 8 rejected under 35 U.S.C. 103 as being unpatentable over Acuna Marrero et al. (Acuna) in view of Englard et al.
Claims 2, 9 rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard, and further in view of Burlina et al.
Claim 3 rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard, and further in view of Venkatadri.
Claim 4 rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard and Venkatadri, and further in view of Al Faruque et al.
Claims 6, 13 rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard, and further in view of Kar et al.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard, and further in view of Burlina et al. and Mudalige et al.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard, and further in view of Bagnell et al.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard and Bagnell, and further in view of Al Faruque and Manoj Mangam.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Burlina et al. and Acuna Marrero et al.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Burlina and Acuna, and further in view of Choe et al.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Burlina and Acuna, and further in view of Venkatadri.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Burlina, Acuna and Choe, and further in view of Al Faruque et al.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-7 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 1 recites receive, by a machine learning model, a plurality of input datasets that are generated as part of a simulated scene within a simulation environment. It is not clear if the plurality of input datasets generated as part of a simulated scene within a simulation environment are actually simulated data or are any data that is part of or used as part of a simulated scene and include real-world data, parameters data, or features data or other data generated as part of the simulated scene within a simulation environment? It is further unclear what a simulation environment is or is not. For example, is a simulation environment any environment that adjusts data, or transforms data, or modifies or somehow alters data in anyway? Or is a simulation environment a virtual environment or modeling environment? Or is a simulation environment something else entirely? Thus, the boundaries of the claim are not clear and the claim is indefinite.
Claims 2-7 depend, directly or indirectly from independent claim 1 and are rejected for the same reasons discussed above with respect to claim 1. Applicant may cancel claims 1-7 or amend claims 1-7 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 8-13 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 8 recites a plurality of input datasets that are generated as part of a simulated scene within a simulation environment. It is not clear if the plurality of input datasets that are generated as part of a simulated scene within a simulation environment are actually simulated data or are any data that is part of or used as part of a simulated scene and include real-world data, parameters data, or features data or other data generated as part of the simulated scene within a simulation environment? It is further unclear what a simulation environment is or is not. For example, is a simulation environment any environment that adjusts data, or transforms data, or modifies or somehow alters data in anyway? Or is a simulation environment a virtual environment or modeling environment? Or is a simulation environment something else entirely? Thus, the boundaries of the claim are not clear and the claim is indefinite.
Claims 9-13 depend, directly or indirectly from independent claim 8 and are rejected for the same reasons discussed above with respect to claim 8. Applicant may cancel claims 8-13 or amend claims 8-13 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefore, subject to the conditions and requirements of this title.
Claims 1-17 and 19-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) recite(s) subject matter at a general, high-level to determine, a plurality of input classifiers for each of the plurality of outputs, wherein each input classifier from the plurality of input classifiers indicates whether input data corresponding to a respective output from the plurality of outputs is associated with simulated input data or real-world input data, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). This judicial exception is not integrated into a practical application and the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Claims 1-17 and 19-20 recite one of the four statutory categories of patent able subject matter and belong to the statutory class(es) of a process (method claims 8-13), a machine (system/apparatus claims 1-7, 14-20), and an article of manufacture (non-transitory computer readable media claims).
Claim 1 recites a system, thus a machine and one of the four statutory categories of patentable subject matter. However, claim 1 further recites to determine, a plurality of input classifiers for each of the plurality of outputs, wherein each input classifier from the plurality of input classifiers indicates whether input data corresponding to a respective output from the plurality of outputs is associated with simulated input data or real-world input data, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
A system comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
receive, by a machine learning model, a plurality of input datasets that are generated as part of a simulated scene within a simulation environment (An additional element of extra-solution activity that courts have identified is well understood, routine and conventional activity for receiving or transmitting data over a network, e.g., using the internet to gather data. See also, MPEP 2106.05(d)(II), MPEP 2106.05(g), 2019 Guidance, 84 FR 50 at 55, 2019 Guidance, 84 FR 50, footnote 31.).
generate, by the machine learning model, a plurality of outputs that are based on the plurality of input datasets (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
by a discriminator head of the machine learning model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more, and transmitting data over a network is well-understood, routine and conventional (MPEP 2106.05(d), and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim 2, dependent on claim 1, recites additional abstract ideas to identify at least one input feature within the plurality of input datasets that is used to classify the input data as simulated input data, wherein the at least one input feature reduces a simulation fidelity associated with the simulated scene, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the one or more processors are further configured to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
by the discriminator head (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 3, dependent on claim 1, recites additional abstract ideas to determine a simulation fidelity score for the simulated scene, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the one or more processors are further configured to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
wherein the simulation fidelity score is based on the plurality of input classifiers determined by the discriminator head (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 4, dependent on claim 3, only recites additional abstract ideas for wherein the simulation fidelity score is based on an area under a receiver operating characteristic curve, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, this recited subject matter is also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
Claim 5, dependent on claim 1, does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the discriminator head determines the plurality of input classifiers based on a feature vector generated by an intermediate layer of the machine learning model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 6, dependent on claim 1, does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the machine learning model corresponds to at least one of a perception stack of an autonomous vehicle (AV), a prediction stack of an AV, and a planning stack of an AV (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 7, dependent on claim 1, recites wherein the plurality of input datasets includes synthetic sensor data and synthetic object tracks that are associated with the simulated scene. The claim does not include additional elements because it further specifies elements already present in the parent claims. Instead, the claim further specifies that the received input data datasets includes synthetic sensor data and synthetic object tracks that are associated with the simulated scene. Accordingly, claim 7 is directed to unpatentable subject matter.
Claim 8 recites a method, thus a process and one of the four statutory categories of patentable subject matter. However, claim 8 further recites for determining a plurality of input classifiers for each of the plurality of outputs, wherein each input classifier from the plurality of input classifiers indicates whether input data corresponding to a respective output from the plurality of outputs is associated with simulated input data or real-world input data, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
computer-implemented (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
receiving, by a machine learning model, a plurality of input datasets that are generated as part of a simulated scene within a simulation environment (An additional element of extra-solution activity that courts have identified is well understood, routine and conventional activity for receiving or transmitting data over a network, e.g., using the internet to gather data. See also, MPEP 2106.05(d)(II), MPEP 2106.05(g), 2019 Guidance, 84 FR 50 at 55, 2019 Guidance, 84 FR 50, footnote 31.).
generating, by the machine learning model, a plurality of outputs that are based on the plurality of input datasets (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
by a discriminator head of the machine learning model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more, and transmitting data over a network is well-understood, routine and conventional (MPEP 2106.05(d), and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim 9, dependent on claim 8, recites additional abstract ideas for identifying at least one input feature within the plurality of input datasets that is to classify the input data as simulated input data, wherein the at least one input feature reduces a simulation fidelity associated with the simulated scene, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
by the discriminator head (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 10, dependent on claim 8, recites additional abstract ideas for determining a simulation fidelity score for the simulated scene and wherein the simulation fidelity score indicates whether the simulated scene is distinguishable from a corresponding real-world environment, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the simulation fidelity score is based on the plurality of input classifiers determined by the discriminator head (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 11, dependent on claim 10, only recites additional abstract ideas for wherein the simulation fidelity score is based on an area under a receiver operating characteristic curve, and wherein the simulation fidelity score has a value from zero to one, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, this recited subject matter is also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
Claim 12, dependent on claim 8, does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the discriminator head determines the plurality of input classifiers based on a feature vector generated by an intermediate layer of the machine learning model, wherein the feature vector includes compressed object features from the input dataset (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 13, dependent on claim 8, does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the machine learning model corresponds to at least one of a perception stack of an autonomous vehicle (AV), a prediction stack of an AV, and a planning stack of an AV (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 14 recites a system, thus a machine and one of the four statutory categories of patentable subject matter. However, claim 14 further recites that indicates whether input data to the machine learning model corresponds to real-world input data or simulated input data, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
A system comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
generate a first training dataset that is based on real-world data collected by an autonomous vehicle having a machine learning model (An additional element of extra-solution activity that courts have identified is well understood, routine and conventional activity for receiving or transmitting data over a network, e.g., using the internet to gather data, collect data. See also, MPEP 2106.05(d)(II), MPEP 2106.05(g), 2019 Guidance, 84 FR 50 at 55, 2019 Guidance, 84 FR 50, footnote 31. Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
generate a second training dataset that is based on simulation data, wherein the simulation data corresponds to the real-world data (An additional element of extra-solution activity that courts have identified is well understood, routine and conventional activity for receiving or transmitting data over a network, e.g., using the internet to gather data, collect data. See also, MPEP 2106.05(d)(II), MPEP 2106.05(g), 2019 Guidance, 84 FR 50 at 55, 2019 Guidance, 84 FR 50, footnote 31. Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
train a revised version of the machine learning model using the first training dataset and the second training dataset, wherein the revised version of the machine learning model includes a discriminator head that is configured to generate an input classifier (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 15, dependent on claim 14, recites additional abstract ideas to convert the real-world data into the simulation data, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, this recited subject matter is also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the one or more processors are further configured to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
Claim 16, dependent on claim 14, does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the discriminator head determines the input classifier based on a feature vector generated by an intermediate layer of the machine learning model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 17, dependent on claim 16, does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the intermediate layer corresponds to a combined embedding layer (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 19, dependent on claim 14, recites additional abstract ideas to determine a simulation fidelity score, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the one or more processors are further configured to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
that is based on the input classifier (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 20, dependent on claim 19, only recites additional abstract ideas for wherein the fidelity score is based on an area under a receiver operating characteristic curve, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, this recited subject matter is also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Acuna Marrero et al. (Pub. No. US 2022/0391766 A1, published December 8, 2022) hereinafter Acuna and Englard et al. (Pub. No. US 2020/0074230 A1, filed March 5, 2020) hereinafter Englard.
Regarding claim 1, Acuna teaches:
A system comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to (i.e., [0158] FIG. 7 is a block diagram of an example computing device(s) 700 suitable for use in implementing some embodiments of the present disclosure. Computing device 700 may include an interconnect system 702 that directly or indirectly couples the following devices: memory 704, one or more central processing units (CPUs) 706, … As such, a computing device(s) 700 may include discrete components (e.g., a full GPU dedicated to the computing device 700), virtual components (e.g., a portion of a GPU dedicated to the computing device 700), or a combination thereof. Acuna, Fig 7, para 157-158, 159-171):
receive, by a machine learning model, a plurality of input datasets that are generated as part of a simulated scene within a simulation environment;
Acuna teaches that, Figure 1 of Acuna illustrates and teaches receive, by a machine learning model (#112), input dataset (#108) that are generated as part of a simulated scene (#104) within a simulation environment. Acuna, Fig. 1, para 26, 31, 37, 6. In addition, to train models using synthetic data that still perform well in the real-world, the systems and methods of the present disclosure use a discriminator that allows a model to learn domain-invariant representations to minimize the divergence between the virtual world and the real-world in a latent space. Acuna, Fig 1, para 6, 26, 31, 37. [0026] Generalization capabilities of a machine learning model(s) 112 under distribution shifts may be learned by analyzing a corresponding binary classifier of the machine learning model(s) 112. As such, the process 100 may implement domain adaptation theory, and may assume the output domain be y={0, 1}, and restrict the mathematical analysis to the binary classification setting. A simulator 106 can automatically produce labels for a task, and these data samples obtained from the simulator 106 may be referred to as a synthetic (or simulated) dataset (S) (e.g., synthetic data 108) with labeled data points S={(x.sub.i.sup.s, y.sub.i.sup.s)}.sub.i=1.sup.n.sup.s. In some embodiments, a real-world dataset (T) (e.g., real-world data 110) with unlabeled examples T={(x.sub.i.sup.t)}.sub.i=1.sup.n.sup.t that are collected in the real-world may also be available. The goal of the process 100 is learn a model (hypothesis) h for a particular task (e.g., object detection, object tracking, free space analysis, etc.) using data from the labeled dataset S (e.g., the synthetic data 108) such that h performs well in the real-world. In embodiments, the unlabeled dataset T (e.g., the real-world data 110) may be incorporated in the learning process (receive, by a machine learning model, a plurality of input datasets (simulated dataset 108, real-world dataset 110) to help capture the real-world distribution. Acuna, Fig 1, para 26, 31, 37, 6.
Thus, Acuna teaches to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets. Acuna does not specifically disclose a plurality of input datasets generated as part of a simulated scene within a simulation environment.
However, Englard teaches in the field related to autonomous vehicles, and, more particularly, to generating feature training datasets, and/or other data, for use in real-world autonomous driving applications based on virtual environments. Englard, para 2. Englard, which is analogous to the claimed invention because Englard is directed to generating datasets for training machine learning models for autonomous vehicles, teaches that, a software architecture includes an automated training dataset generator that generates feature training datasets based on simulated or virtual environments. The feature training datasets may be used to train various machine learning models (receive, by a machine learning model,)for use in real-world autonomous driving applications, e.g., to control the maneuvering of autonomous vehicles. The feature training datasets may include virtual data based on photo-realistic scenes (e.g., simulated 2D image data), depth-map realistic scenes (e.g., simulated 3D image data), and/or environment-object data (e.g., simulated data defining how objects or surfaces interact), each corresponding to the same virtual environment (a plurality of input datasets generated as part of a simulated scene within a simulation environment). For example, the environment-object data for a particular vehicle in the virtual environment may relate to the vehicle's motion (e.g., position, velocity, acceleration, trajectory, etc.). In some embodiments, interactions between objects or surfaces within the virtual environment can affect the data outputted for the simulated environment, e.g., rough roads or potholes may affect measurements of a virtual inertial measurement unit (IMU) of a vehicle. ... More generally, environment-object data may broadly to refer to information about objects/surfaces within a virtual environment, e.g., interactions between objects or surfaces in the virtual environment and how those interactions effect the objects or surfaces in the virtual environment, e.g., a vehicle hitting a pothole. Englard, Figs 1-4B, para 34, 61-62, 66. In certain embodiments, one or more outputs of a machine learning model may be compared to ground truth value(s) (generate, by the machine learning model, a plurality of outputs that are based on the plurality of input datasets). In such embodiments, the ground truth value(s) may each include representations of vehicle action (e.g., from vehicles including vehicles 401, 451, 700, and/or 756 as described herein) and a corresponding safety parameter defining, e.g., a safety-related outcome, or a degree of safety that is associated with the vehicle action. In some embodiments, a machine learning model may be updated to choose vehicle actions that maximize a degree of safety across a plurality of ground truth values. However, in other embodiments, a machine learning model may be updated to choose vehicle actions that vary the degree of safety (e.g., risking driving to safe driving) across a plurality of ground truth values. Englard, Figs 1-4B, para 66, 34, 61-62.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment of Englard, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving applications. Englard, para 5-6, 4-6. This would have provided the advantages of improving training and performance of autonomous systems and applications.
generate, by the machine learning model, a plurality of outputs that are based on the plurality of input datasets; and
Acuna in view of Englard teaches the plurality of input data sets. Acuna teaches that, Using the synthetic data 108 generated by the simulator 106, in combination with the real-world data 110, in embodiments, the machine learning model(s) 112 may be trained (e.g., one or more parameters—such as weights and biases—of the machine learning model(s) 112 may be updated or adjusted using the data 108 and 110 as input) using a training engine 114 (generate, by the machine learning model (machine learning model 112, Figures 1, 4), a plurality of outputs (Figures 1, 4 illustrate ML model 112 outputs) that are based on the input dataset (an input dataset 108). For example, the parameters of the machine learning model(s) 112 may be updated until an acceptable level of accuracy or precision is reached (e.g., until convergence is achieved). Because the synthetic data 108 is generated using a sampling strategy that is simulator agnostic, any different simulator 106 may be used to generate the synthetic data 108 with enough diversity to minimize the distance between the label marginals. As observed based on theorem (1), above, in order to improve performance in the real-world, the distance between the input distributions P.sub.s(x) and P.sub.t(x) should be minimized. The training algorithm of the present disclosure accomplishes this by learning domain invariant representations and minimizing the divergence between the virtual and real-world in a latent space Z. Minimizing the divergence in a latent space allows domain adaptation algorithms to be sensor and architecture agnostic. Without using the sampling strategy of the present disclosure, effectively learning invariant representation through adversarial learning would be less effective. As such, the combination of the adversarial training algorithm and the sampling strategy to generate the synthetic data 108 allow for learning the domain invariant representations. In addition, in embodiments, pseudo-labels 116 may be used to further improve the training algorithms and increase the accuracy and precision of the machine learning model(s) 112. Acuna, Figs. 1, 4, para 40, 37-39.
As discussed above, Acuna does not specifically disclose the plurality of input datasets and plurality of outputs.
However as similarly discussed above Englard teaches the plurality of input data sets. England teaches that, In certain embodiments, one or more outputs of a machine learning model may be compared to ground truth value(s) (generate, by the machine learning model, a plurality of outputs that are based on the plurality of input datasets). In such embodiments, the ground truth value(s) may each include representations of vehicle action (e.g., from vehicles including vehicles 401, 451, 700, and/or 756 as described herein) and a corresponding safety parameter defining, e.g., a safety-related outcome, or a degree of safety that is associated with the vehicle action. In some embodiments, a machine learning model may be updated to choose vehicle actions that maximize a degree of safety across a plurality of ground truth values. However, in other embodiments, a machine learning model may be updated to choose vehicle actions that vary the degree of safety (e.g., risking driving to safe driving) across a plurality of ground truth values. Englard, Figs 1-4B, para 66, 34, 61-62.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment and a plurality of outputs that are based on the plurality of input datasets of Englard, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving applications. Englard, para 5-6, 4-6. This would have provided the advantages of improving training and performance of autonomous systems and applications.
determine, by a discriminator head of the machine learning model, a plurality of input classifiers for each of the plurality of outputs, wherein each input classifier from the plurality of input classifiers indicates whether input data corresponding to a respective output from the plurality of outputs is associated with simulated input data or real-world input data.
As discussed above, Acuna in view of Englard teaches the plurality of input datasets and plurality of outputs. Acuna teaches that, Different from prior implementations off-DAL Pearson, d.sub.st can be interpreted as a per-location domain classifier 410, as illustrated in FIG. 4. For example, the per-location domain classifier 410 may operate as a discriminator that identifies whether the data in a latent space corresponds to a real-world domain or a simulated or virtual domain (determine, by a discriminator head (h’, Figure 4) of the machine learning model, a plurality of input classifiers (Per-Location Domain Classifier 410 of the plurality of classifiers per location, Figure 4) for each of the plurality of outputs (z, Figure 4), wherein each input classifier from the plurality of input classifiers indicates whether input data corresponding to a respective output from the plurality of outputs is associated with simulated input data (simulated or virtual domain) or real-world input data (real-world domain, see also Figure 4 which illustrates machine learning model g outputs z to discriminator h’ head that determines a plurality of Per-Location Domain Classifiers #410 that identify whether each input data corresponding to a respective output from the plurality of outputs is associated with simulated domain or real-world domain). As such, when the values in the latent space are classified as belonging to the source domain (and not the target domain), this may be captured in the d.sub.st value, and used by the training engine 114 to update the parameters of the machine learning model(s) 112. In embodiments, a gradient reversal layer may be used to deal with min-max objective in a single forward-backward pass. Acuna, Figs. 1, 4, para 41, 42, 40, 47, 31, 33-34, 37-39.
[0042] As illustrated in FIG. 4, process 400 may be used to train the machine learning model(s) 112. For example, to compute the discrepancy term, d.sub.st, a per-location domain classifier ĥ′ is introduced before, in embodiments, the final up-sampling module of the model's bird's eye view encoder. The domain classifier may include two convolutional layers with LeakyRelu non-linearity that predicts whether a pixel in the H×W semantic map corresponds to either a source domain (e.g., the virtual or simulated world) or a target domain (e.g., the real-world). The other output, ĥ, predicts the output of the machine learning model(s) 112—which is a BEV segmentation map in this example. g constitutes a backbone of the machine learning model(s) 112, which may include a CamEncoder and a BevEncoder (up to the last up-sampling module). As discussed above, Acuna in view of Herman teaches the plurality of input data sets. Acuna, Figs. 1, 4, para 42, 41, 40, 47, 31, 33-34, 37-39.
Claim 8 recites a computer-implemented method that parallels the system of claim 1. Therefore, the analysis discussed above with respect to claim 1 also applies to claim 8. Accordingly, claim 8 is rejected based on substantially the same rationale as set forth above with respect to claim 1.
Claim(s) 2 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard as applied to claims 1 and 8 above, and further in view of Burlina et al. (Patent No. US 12,688,686 B1, filed November 22, 2023, hereinafter Burlina.
Regarding claim 2, which depends from claim 1, and further recites:
identify at least one input feature within the plurality of input datasets that is used by the discriminator head to classify the input data as simulated input data, wherein the at least one input feature reduces a simulation fidelity associated with the simulated scene.
Acuna in view of Englard teaches the system of claim 1 from which claim 2 depends, including the plurality of input datasets that is used by the discriminator head to classify the input data as simulated input data. Acuna in view of Englard does not specifically disclose identify at least one input feature, wherein the at least one input feature reduces a simulation fidelity associated with the simulated scene.
However, Burlina teaches in the field related to improving synthetic training data generation. Burlina, Abstract. Burlina, which is analogous to the claimed invention because Burlina is directed to improving synthetic training data generation for training object detection models, teaches that, (29) Various techniques may be used to further refine particular aspects of a synthetic training data generation model and/or the training data generated thereby (identify at least one input feature, wherein the at least one input feature reduces a simulation fidelity associated with the simulated scene (identify at least one particular aspect feature of the synthetic input training data to refine because it reduces a simulation fidelity associated with the scene)). For example, one or more masks may be used as input and/or a model parameter to a synthetic training data generation model to determine a location of an object of interest. For instance, if the system is attempting to refine the synthetic training data generation model's generation of prone pedestrians, an object mask may be used to determine the location of such pedestrians within individual images. This may help improve the realistic placement of such objects (e.g., placing pedestrians on sidewalks or crosswalks rather than in rooftops or awnings) (identify at least one input feature, wherein the at least one input feature reduces a simulation fidelity associated with the simulated scene (identify at least one particular aspect feature (pedestrian placement) of the synthetic input training data to refine because it (pedestrian placement in rooftops or awnings) reduces a simulation fidelity associated with the scene)), for example, where objects are augmented to real-world images or other data representing existing portions of an environment. Burlina, col 7:52-col 8:7.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment and a plurality of outputs that are based on the plurality of input datasets of Englard and to identify at least one input feature, wherein the at least one input feature reduces a simulation fidelity associated with the simulated scene of Burlina, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving applications and to provide training data including unusual scenarios with infrequently occurring objects and/or characteristic that may be challenging due to the rarity of occurrence of such scenarios in the real-world and decrease the difficulty of training models to accurately detect such scenarios. Englard, para 5-6, 4-6. Burlina, col 1:6-28. This would have provided the advantages of improving training and performance of autonomous systems and applications.
Claim 9 recites a computer-implemented method that parallels the system of claim 2. Therefore, the analysis discussed above with respect to claim 2 also applies to claim 9. Accordingly, claim 9 is rejected based on substantially the same rationale as set forth above with respect to claim 2.
Claim(s) 3 is rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard as applied to claims 1 above, and further in view of Venkatadri (Pub No. US 2020/0134494 A1, published April 30, 2020).
Regarding claim 3, which depends from claim 1, and further recites:
determine a simulation fidelity score for the simulated scene, wherein the simulation fidelity score is based on the plurality of input classifiers determined by the discriminator head.
Acuna in view of Englard teaches the system of claim 1 from which claim 3 depends, including the simulated scene and the plurality of input classifiers determined by the discriminator head. Acuna in view of Englard does not specifically disclose determine a simulation fidelity score.
However, Venkatadri teaches in the field related to devices, systems, and methods for generating artificial scenarios for autonomous vehicles. Venkatadri, para 2. Venkatadri, which is analogous to the claimed invention because Venkatadri is directed to generating artificial scenarios for autonomous vehicles, teaches that, As an example, the machine-learned discriminator model(s) can determine an authenticity associated with each parameter used to generate the artificial data and/or determine an overall authenticity associated with the artificial data ((determine a simulation fidelity score (artificial scenarios simulation authenticity fidelity score)) (e.g., authenticity of a simulated scenario, authenticity of a simulated log data set, etc.). The machine-learned discriminator model(s) can determine the authenticity data based on the authenticity for each parameter and/or the overall authenticity. Venkatadri, para 41, 47, 127, 5, 23.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment and a plurality of outputs that are based on the plurality of input datasets of Englard and to determine a simulation fidelity score of Venkatadri, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving applications and in order to provide vehicle simulations, and in particular to generating artificial scenarios for autonomous vehicle, where an artificial environment can be simulated to create a virtual world and allow for the measurement of vehicle system performance, and an artificial log can be used for the testing of vehicle system performance and to help improve the safety of passengers of an autonomous vehicle, improve the safety of the surroundings of the autonomous vehicle, improve the experience of the rider and/or operator of the autonomous vehicle. Englard, para 5-6, 4-6. Venkatadri, para 23, 5, 9. This would have provided the advantages of improving training and performance of autonomous systems and applications.
Claim(s) 4 is rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard and Venkatadri as applied to claims 3 above, and further in view of Al Faruque et al. (Pub No. US 2023/0230484 A1, published April 30, 2020) hereinafter Al Faruque.
Regarding claim 4, which depends from claim 3 and recites:
wherein the simulation fidelity score is based on an area under a receiver operating characteristic curve.
Acuna in view of Englard and Venkatadri teach the system of claim 3 from which claim 4 depends, including the simulation fidelity score. Acuna in view of Englard and Venkatadri does not specifically disclose based on an area under a receiver operating characteristic curve.
However, Al Faruque teaches in the field related to a spatiotemporal scene-graph embedding methodology that models scene-graphs and resolves safety-focused tasks for autonomous vehicles. Al Faruque, para 3. Al Faruque which is analogous to the claimed invention because Ref is directed to simulated datasets, scene-graphs, and autonomous vehicle applications, teaches that, Each model's performance was evaluated by measuring its classification accuracy and the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) for each dataset (based on an area under a receiver operating characteristic curve). The classification accuracy is the ratio of the number of correct predictions on the test set of a dataset to the total number of samples in the testing set. AUC, sometimes referred to as a balanced accuracy measure, measures the probability that a binary classifier ranks a positive sample more highly than a random negative sample. This was a more balanced measure for measuring accuracy, especially with imbalanced datasets. Al Faruque, Fig 17, para 165, 180,159.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment and a plurality of outputs that are based on the plurality of input datasets of Englard and to determine a simulation fidelity score of Venkatadri and based on an area under a receiver operating characteristic curve of Al Faruque, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving applications and in order to provide vehicle simulations, and in particular to generating artificial scenarios for autonomous vehicle, where an artificial environment can be simulated to create a virtual world and allow for the measurement of vehicle system performance, and an artificial log can be used for the testing of vehicle system performance and to help improve the safety of passengers of an autonomous vehicle, improve the safety of the surroundings of the autonomous vehicle, improve the experience of the rider and/or operator of the autonomous vehicle and to provide for the development of safe and robust AVs and a model that can transfer knowledge gained from a simulated training set to a real-world testing set effectively will likely perform better in unseen real-world scenarios. Englard, para 5-6, 4-6. Venkatadri, para 23, 5, 9. Al Faruque, para 4, 8, 165. This would have provided the advantages of improving models, training and performance of autonomous systems and applications.
Claim(s) 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard as applied to claims 1 and 8 above, and further in view of Kar et al. (Patent No. US 12,499,363 B2, filed March 20, 2023) hereinafter Kar.
Regarding claim 6, which depends from claim 1, and recites:
wherein the machine learning model corresponds to at least one of a perception stack of an autonomous vehicle (AV), a prediction stack of an AV, and a planning stack of an AV.
Acuna in view of Englard teaches the system of claim 1 from which claim 6 depends, including the machine learning model. Acuna teaches that the machine learning model corresponds to application of an autonomous vehicle. Acuna, para 20, 24. Acuna in view of Englard does not specifically disclose one of a perception stack of an autonomous vehicle (AV), a prediction stack of an AV, and a planning stack of an AV.
However, Kar teaches in the field related to generating synthetic datasets for training neural networks. Kar, Abstract, col 1:59-65, 20-55. Kar is analogous to the claimed invention because Kar is directed to a generative model used to synthesize datasets for use in training a downstream machine learning model teaches that, In addition, in some non-limiting embodiments, the task network may be used in a simulated or virtual environment (which may also be generated or rendered using the distribution transformer 108 and the transformed scene graphs 110, in embodiments) in order to test the performance of the task network prior to deploying the task network for use in a real-world environment. For example, the task network may be a part of an autonomous driving software stack—e.g., part of a perception layer of the stack, tasked with object detection. As such, a virtual or simulated vehicle may implement the autonomous driving software stack in a virtual environment, where the simulated vehicle may capture virtual image data (or other sensor data types) using virtual or simulated image sensors or cameras (or other sensor types). The virtual image data may then be applied to the task network to test the functionality of the task network within the autonomous driving stack in the virtual environment. Kar, Abstract, col 6:49-col 7:12, col 5:51-65.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment and a plurality of outputs that are based on the plurality of input datasets of Englard and one of a perception stack of an autonomous vehicle (AV), a prediction stack of an AV, and a planning stack of an AV of Kar, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving applications and in order to aid in bridging the domain gap and the content gap, with resulting in photo-realistic environments that mimic real-world scenes with an accuracy that enables downstream task networks to be trained on these synthetic datasets while being deployed for use in real-world applications using real-world data. Englard, para 5-6, 4-6. Kar, col 2:13-32. This would have provided the advantages of improving training and performance of autonomous systems and applications.
Claim 13 recites a computer-implemented method that parallels the system of claim 6. Therefore, the analysis discussed above with respect to claim 6 also applies to claim 13. Accordingly, claim 13 is rejected based on substantially the same rationale as set forth above with respect to claim 6.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard as applied to claim 1 above, and further in view of Burlina et al. (Patent No. US 12,688,686 B1, filed November 22, 2023, hereinafter Burlina and Mudalige et al. (Pub. No. US 2019/0235521 A1, published August 1, 2019) hereinafter Mudalige.
Regarding claim 7, which depends from claim 1, and recites:
wherein the plurality of input datasets includes synthetic sensor data and synthetic object tracks that are associated with the simulated scene.
Acuna in view of Englard teaches the system of claim 1 from which claim 7 depends, including the plurality of input datasets and simulated scene. Acuna in view of Englard does not specifically disclose includes synthetic sensor data and synthetic object tracks
However, Burlina teaches in the field related to improving synthetic training data generation. Burlina, Abstract. Burlina, which is analogous to the claimed invention because Burlina is directed to improving synthetic training data generation for training object detection models, teaches that, (12) In various examples, a system may train a machine-learned model to automatically, more accurately, and more efficiently detect objects and features in data collected in an environment, including objects in unusual circumstances that may typically inhibit accurate detection. The system may generate training data using a synthetic training data generation model. This generated synthetic training data may include synthetic data based on real-word scenarios and/or objects. The generated synthetic training data may take one or more forms, including synthetic sensor data (e.g., images, lidar data, radar data, sonar data, audio data, etc.) (includes synthetic sensor data) and synthetic object detection data (e.g., one or more data structures representing object detection data, such as object labels, attributes, properties, location, type, velocity, yaw, position, acceleration, etc.). The system may then process such generated synthetic training data using a critic network to determine if the critic network is able to identify the synthetic data as synthetic. For example, the critic network may evaluate the synthetic data associated with a particular scenario against real-world data associated with the same or a similar scenario to determine if the synthetic data is distinguishable from real-world data. Burlina, col 3:25-46; col 25:45 - col 26:17, col 8:39-54.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment of Englard and the included synthetic sensor data of Burlina, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving applications and to provide training data including unusual scenarios with infrequently occurring objects and/or characteristic that may be challenging due to the rarity of occurrence of such scenarios in the real-world and decrease the difficulty of training models to accurately detect such scenarios. Englard, para 5-6, 4-6. Burlina, col 1:6-28. This would have provided the advantages of improving training and performance of autonomous systems and applications.
Acuna in view of Englard and Burlina does not specifically disclose synthetic object tracks.
However, Mudalige teaches in the field related to automotive vehicles, and more particularly relates to systems and methods for developing and validating autonomous vehicle operation using real-world and virtual data sources. Mudalige, para 1. Mudalige, which is analogous to the claimed invention because Mudalige is directed to developing and validating autonomous vehicle operation using real-world and virtual data sources, teaches that, data from database 212 is synthesized and processed in fusion module 308 to represent the presence, location, classification, and/or path of objects (synthetic object tracks) and features of the environment of the vehicle 10 and of the scenes captured by sensors 304, 306. The fusion module 308 incorporates information from the multiple sensors in a register type synchronized form. For example, as shown in FIG. 4, data from the vehicle 10 may be used to reproduce a scene from the perspective of the vehicle as depicted in image 310. For example, a roadway 312, other vehicles 314, objects 316, and signs 318 may be represented. Data may also be included from sensor model emulator 320 using a simulated virtual sensor set modeling the sensors 40a-40n. This may include a model of the vehicle 10 with all sensors 40a-40n. Generation of data for various scenarios may be scripted or manually prompted to generate synthetic data. The sensor model emulator 320 may run in the validation system 200 or in another computer or computers. Scenarios may be created with a number of other actors including roadway variations, pedestrians, other vehicles and other objects. Data from the sensor model emulator 320 may be stored in the database 212 or may be supplied directly to the fusion module 308, where it is fused along with the real-world data. The sensor model emulator coordinates with a virtual world renderer 322, which creates 3-dimensional representations of roadways and objects using the virtually generated data from the sensor model emulator 320. For example, environmental aspects of the virtual world may include infrastructure details such as traffic signals, traffic marks, traffic signs, and others. In addition, object aspects of the virtual world may include the identification of the object and whether it moves or is stationary, along with a timestamp, location, size, speed, acceleration, heading, trajectory, surface reflectivity and material properties. Event information may be included, such as lane changes, speed changes, stops, turns, and others. Mudalige, Figs 4,5, para 46.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment of Englard and the included synthetic sensor data of Burlina and the objects synthetic object tracks of Mudalige, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving applications and to provide training data including unusual scenarios with infrequently occurring objects and/or characteristic that may be challenging due to the rarity of occurrence of such scenarios in the real-world and decrease the difficulty of training models to accurately detect such scenarios and to provide for evaluating and validating autonomous vehicle control and operation during product development. Englard, para 5-6, 4-6. Burlina, col 1:6-28. Mudalige, para4, 2-6, 46. This would have provided the advantages of improving training and performance of autonomous systems and applications.
Claim(s) 10 is rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard as applied to claim 8 above, and further in view of Bagnell et al. (Patent No. US 12,444,247 B1, filed August 12, 2022) hereinafter Bagnell.
Regarding claim 10, which depends from claim 8, and further recites:
determining a simulation fidelity score for the simulated scene, wherein the simulation fidelity score is based on the plurality of input classifiers determined by the discriminator head, and wherein the simulation fidelity score indicates whether the simulated scene is distinguishable from a corresponding real-world environment.
Acuna in view of Englard teaches the method of claim 8 from which claim 3 depends, including the simulated scene and the plurality of input classifiers determined by the discriminator head. Acuna in view of Englard does not specifically disclose determine a simulation fidelity score, and wherein the simulation fidelity score indicates whether the simulated scene is distinguishable from a corresponding real-world environment.
However, Bagnell teaches in the field related to evaluating the performance of an autonomous vehicle and determining a plurality of simulation scenarios. Bagnell, Abstract, col 1:24-53. Bagnell, which is analogous to the claimed invention because Bagnell is directed to estimating the performance metrics of autonomous vehicles in the real world based on their behavior measured in the simulated scenarios, teaches that the present disclosure is particularly advantageous for estimating the performance metrics of an autonomous vehicle because density ratio estimation facilitates determining how well the distribution of events covered by a representative set of simulation scenarios matches the distribution of events expected in real-world driving and reweighting the measurements from the corresponding simulation runs with respect to their exposure in the real-world driving. For example, the density ratio estimation approach may be used to estimate how over-represented a particular simulation scenario is with respect to the real world (determine a simulation fidelity score (density ratio simulation fidelity score), and wherein the simulation fidelity score indicates whether the simulated scene (simulation scenario) is distinguishable from a corresponding real-world environment). One implementation of the density ratio estimation approach is using maximum entropy modeling. Bagnell, col 1:24-53, col 15:47-col 16:17, col 15:47-col 17:2; col 17:61-col 18:23.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment and a plurality of outputs that are based on the plurality of input datasets of Englard and to determine a simulation fidelity score, and wherein the simulation fidelity score indicates whether the simulated scene is distinguishable from a corresponding real-world environment of Bagnell, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving and to provide a technique to estimate the performance metrics of autonomous vehicles in the real world based on their behavior measured in the simulated scenarios. Englard, para 5-6, 4-6. Bagnell, col 1:7-20; 24-53. This would have provided the advantages of improving training and performance of autonomous systems and applications.
Claim(s) 11 is rejected under 35 U.S.C. 103 as being unpatentable over Acuna in view of Englard and Bagnell as applied to claim 10 above, and further in view of Al Faruque and Manoj Mangam, AUC Decoded: Its Meaning & Intuition. Clearly Explained! Decoding the intuition behind AUROC (Area Under ROC Curve). May 27, 2023, retrieved from https://medium.com/@mangammanoj/auc-what-is-its-meaning-c6ccbfb99892.) hereinafter Mangam.
Regarding claim 11, which depends from claim 10, and recites:
wherein the simulation fidelity score is based on an area under a receiver operating characteristic curve, and wherein the simulation fidelity score has a value from zero to one.
Acuna in view of Englard and Bagnell teach the method of claim 10 from which claim 11 depends, including the simulation fidelity score. Acuna in view of Englard and Bagnell does not specifically disclose based on an area under a receiver operating characteristic curve, and wherein the simulation fidelity score has a value from zero to one.
However, Al Faruque teaches in the field related to a spatiotemporal scene-graph embedding methodology that models scene-graphs and resolves safety-focused tasks for autonomous vehicles. Al Faruque, para 3. Al Faruque which is analogous to the claimed invention because Ref is directed to simulated datasets, scene-graphs, and autonomous vehicle applications, teaches that, Each model's performance was evaluated by measuring its classification accuracy and the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) for each dataset (based on an area under a receiver operating characteristic curve value). The classification accuracy is the ratio of the number of correct predictions on the test set of a dataset to the total number of samples in the testing set. AUC, sometimes referred to as a balanced accuracy measure, measures the probability that a binary classifier ranks a positive sample more highly than a random negative sample. This was a more balanced measure for measuring accuracy, especially with imbalanced datasets. Al Faruque, Fig 17, para 165, 180,159.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment and a plurality of outputs that are based on the plurality of input datasets of Englard and to determine a simulation fidelity score, and wherein the simulation fidelity score indicates whether the simulated scene is distinguishable from a corresponding real-world environment of Bagnell and based on an area under a receiver operating characteristic curve value of Al Faruque, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving and to provide a technique to estimate the performance metrics of autonomous vehicles in the real world based on their behavior measured in the simulated scenarios and to provide for the development of safe and robust AVs and a model that can transfer knowledge gained from a simulated training set to a real-world testing set effectively will likely perform better in unseen real-world scenarios.. Englard, para 5-6, 4-6. Bagnell, col 1:7-20; 24-53. Al Faruque, para 4, 8, 165. This would have provided the advantages of improving models, training and performance of autonomous systems and applications.
Acuna in view of Englard, Bagnell and Al Faruque teach wherein the simulation fidelity score is based on an area under a receiver operating characteristic curve, and wherein the simulation fidelity score has a value. Acuna in view of Englard, Bagnell and Al Faruque does not specifically disclose has a value from zero to one.
However, Mangam teaches in the field related to machine learning, classification and area under the curve. Mangum, pages 1, 2. Mangam, which is analogous to the claimed invention because Mangum is directed to metrics used to quantify performance of a classifier, teaches that AUROC is simply the Area Under the ROC curve. Since the AUC is a portion of the area of the unit square, its value will always be between 0 and 1, with a higher value indicating better performance. Mangum, page 8.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the training models using synthetic data for autonomous systems including to receive, by a machine learning model, an input dataset generated as part of a simulated scene within a simulation environment and receive, by a machine learning model, a plurality of input datasets and applications of Acuna using the plurality of input datasets generated as part of a simulated scene within a simulation environment and a plurality of outputs that are based on the plurality of input datasets of Englard and to determine a simulation fidelity score, and wherein the simulation fidelity score indicates whether the simulated scene is distinguishable from a corresponding real-world environment of Bagnell and based on an area under a receiver operating characteristic curve value of Al Faruque and has a value from zero to one of Mangam, with a reasonable expectation of success, in order to provide systems and methods to generate feature training datasets for use in real-world autonomous driving applications, including simulated or virtual data that may be used to generate and/or obtain feature-rich and plentiful training datasets and to improve the efficiency and effectiveness of generating and/or collecting numerous autonomous driving datasets, and to also address safety concerns with respect to generating sufficient datasets in a non-dangerous and controlled manner when training autonomous vehicles in real-world driving and to provide a technique to estimate the performance metrics of autonomous vehicles in the real world based on their behavior measured in the simulated scenarios and to provide for the development of safe and robust AVs and a model that can transfer knowledge gained from a simulated training set to a real-world testing set effectively will likely perform better in unseen real-world scenarios and to provide classification metrics to quantify performance of a classifier. Englard, para 5-6, 4-6. Bagnell, col 1:7-20; 24-53. Al Faruque, para 4, 8, 165. Mangam, page 2, 8. This would have provided the advantages of improving models, training and performance of autonomous systems and applications.
Claim(s) 14 is rejected under 35 U.S.C. 103 as being unpatentable over Burlina et al. (Patent No. US 12,688,868 B1, filed November 22, 2023) hereinafter Burlina and Acuna Marrero et al. (Pub. No. US 2022/0391766 A1, published December 8, 2022) hereinafter Acuna.
Regarding claim 14, Burlina teaches:
A system comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to (i.e., (97) The vehicle computing device 504 can include one or more processors 516 and memory 518 communicatively coupled with the one or more processors 516. In the illustrated example, the vehicle 502 is an autonomous vehicle; however, the vehicle 502 could be any other type of vehicle. In the illustrated example, the memory 518 of the vehicle computing device 504 stores a localization component 520, a perception component 522, an object detection component 524, a planning component 528, one or more system controllers 530, one or more maps 532, and a prediction component 534. Though depicted in FIG. 5 as residing in memory 518 for illustrative purposes, it is contemplated that any one or more of the localization component 520, the perception component 522, the object detection component 524, the planning component 528, the one or more system controllers 530, the one or more maps 532, and the prediction component 534 can additionally or alternatively be accessible to the vehicle 502 (e.g., stored remotely).Burlina, Fig 5, col 24:66-col 25:26.):
generate a first training dataset that is based on real-world data collected by an autonomous vehicle having a machine learning model;
generate a second training dataset that is based on simulation data, wherein the simulation data corresponds to the real-world data; and
(i.e., This generated synthetic training data may include synthetic data based on real-word scenarios and/or objects (generate a second training dataset that is based on simulation data, wherein the simulation data corresponds to the real-world data). The generated synthetic training data may take one or more forms, including synthetic sensor data (e.g., images, lidar data, radar data, sonar data, audio data, etc.) and synthetic object detection data (e.g., one or more data structures representing object detection data, such as object labels, attributes, properties, location, type, velocity, yaw, position, acceleration, etc.). The system may then process such generated synthetic training data using a critic network to determine if the critic network is able to identify the synthetic data as synthetic. For example, the critic network may evaluate the synthetic data associated with a particular scenario against real-world data associated with the same or a similar scenario to determine if the synthetic data is distinguishable from real-world data. Burlina, col 3:25-46.
In examples, data of various types (e.g., beyond images) may be used to refine a model that may be configured to generate training data. For example, one or more models may be configured at a vehicle computing system as part of one or more perception components that may interact with individual sensors of a variety of types (e.g., lidar, sonar, radar, vision, audio, time of flight, etc.). Such models may be trained using training data generated by a refined synthetic training data generation model as described herein (generate a second training dataset that is based on simulation data, wherein the simulation data corresponds to the real-world data). For instance, real-world lidar data for a particular environment may be used by an exemplary model training to refine a synthetic lidar training data generation model (generate a first training dataset that is based on real-world data collected by an autonomous vehicle having a machine learning model). The refined synthetic lidar training data generation model may then be used to generate training data that may be used to train a lidar-based object detection model that may be configured as a perception component at a vehicle computing system. In various examples, a machine-learned model trained as described herein may be executed by one or more of various components that may be configured in an autonomous vehicle, including perception components and/or individual sensors (e.g., lidar, sonar, radar, vision, audio, time of flight, etc.), and/or one or more associated components. Such a model may be used to determine data that may be combined with or otherwise used in conjunction with other data (e.g., map data) to determine a location of a vehicle, a vehicle trajectory, a vehicle route, one or more vehicle controls, and/or any other data that may include or make use of object detection data. Burlina, col 7:15-42.
80) Real-world data 402 may be received and/or provided to a synthetic training data generation model refinement system 404 (generate a second training dataset that is based on simulation data, wherein the simulation data corresponds to the real-world data). Real-world data 402 may represent sensor data collected or otherwise determined from a real-world environment. The data 402 may be based on unimodal or multimodal data and/or may be represented by one or more data structures of any type. For example, the real-world data 402 may include and/or represent images (e.g., represented as two-dimensional frames) based on camera data and/or multi-modal sensor data (e.g., RGB data combined with depth data determined from lidar point clouds paired with images). The real-world data 402 may further include embeddings and/or other data augmented with sensor data, such as labels, classifications, etc. In various examples, the data 402 may originate at a vehicle, for example, generated by sensors capturing data in an environment as the vehicle travels within the environment. In various examples, the data 402 may be determined based on one or more multichannel data structures representing data associated with various types of sensors, such as image sensors (e.g., cameras), lidar sensors, radar sensors, audio sensors, sonar sensors, etc. that may capture sensor data representing an environment. The data 402 may include data associated with objects detected in an environment and/or one or more condition indications. Burlina, col 21:19-42.)
train a revised version of the machine learning model using the first training dataset and the second training dataset, wherein the revised version of the machine learning model includes a discriminator head that is configured to generate an input classifier that indicates whether input data to the machine learning model corresponds to real-world input data or simulated input data.
As discussed above, Burlina the first training dataset and the second training dataset. As similarly discussed above, Burlina teaches that, In examples, data of various types (e.g., beyond images) may be used to refine a model that may be configured to generate training data. For example, one or more models may be configured at a vehicle computing system as part of one or more perception components that may interact with individual sensors of a variety of types (e.g., lidar, sonar, radar, vision, audio, time of flight, etc.). Such models may be trained using training data (training a machine learning model using second training dataset) generated by a refined synthetic training data generation model as described herein. For instance, real-world lidar data for a particular environment may be used by an exemplary model training system (training a machine learning model using first training dataset) to refine a synthetic lidar training data generation model. The refined synthetic lidar training data generation model may then be used to generate training data that may be used to train a lidar-based object detection model that may be configured as a perception component at a vehicle computing system. In various examples, a machine-learned model trained as described herein may be executed by one or more of various components that may be configured in an autonomous vehicle, including perception components and/or individual sensors (e.g., lidar, sonar, radar, vision, audio, time of flight, etc.), and/or one or more associated components. Such a model may be used to determine data that may be combined with or otherwise used in conjunction with other data (e.g., map data) to determine a location of a vehicle, a vehicle trajectory, a vehicle route, one or more vehicle controls, and/or any other data that may include or make use of object detection data. Burlina, col 7:15-42.
Burlina does not specifically disclose train a revised version of the machine learning model using the first training dataset and the second training dataset, wherein the revised version of the machine learning model includes a discriminator head that is configured to generate an input classifier that indicates whether input data to the machine learning model corresponds to real-world input data or simulated input data
However, Acuna teaches in the field related to training perception models using synthetic data for autonomous systems and applications. Acuna para 6. Acuna, which is analogous to the claimed invention because Acuna is directed to minimizing the reality gap between simulated and real-world domains, teaches that Different from prior implementations off-DAL Pearson, d.sub.st can be interpreted as a per-location domain classifier 410, as illustrated in FIG. 4. For example, the per-location domain classifier 410 may operate as a discriminator that identifies whether the data in a latent space corresponds to a real-world domain or a simulated or virtual domain (train a revised version of the machine learning model (training engine 114 to update the parameters of the machine learning model(s)112) using the first training dataset and the second training dataset (real-world, simulated datasets), wherein the revised version of the machine learning model includes a discriminator head (h’, Figure 4) that is configured to generate an input classifier (Per-Location Domain Classifier 410, Figure 4) that indicates whether input data to the machine learning model corresponds to real-world input data or simulated input data (using first training dataset and second training dataset to train revised models 112 with discriminator generated input per-location classifier 410 that indicates whether the input data to the machine learning model corresponds to the real-world domain input data or simulated domain input data). As such, when the values in the latent space are classified as belonging to the source domain (and not the target domain), this may be captured in the d.sub.st value, and used by the training engine 114 to update the parameters of the machine learning model(s) 112. In embodiments, a gradient reversal layer may be used to deal with min-max objective in a single forward-backward pass. Acuna, Figs. 1, 4, para 41, 42, 40, 47, 31, 33-34, 37-39.
[0042] As illustrated in FIG. 4, process 400 may be used to train the machine learning model(s) 112. For example, to compute the discrepancy term, d.sub.st, a per-location domain classifier ĥ′ is introduced before, in embodiments, the final up-sampling module of the model's bird's eye view encoder. The domain classifier may include two convolutional layers with LeakyRelu non-linearity that predicts whether a pixel in the H×W semantic map corresponds to either a source domain (e.g., the virtual or simulated world) or a target domain (e.g., the real-world). The other output, ĥ, predicts the output of the machine learning model(s) 112—which is a BEV segmentation map in this example. g constitutes a backbone of the machine learning model(s) 112, which may include a CamEncoder and a BevEncoder (up to the last up-sampling module). Acuna, Figs. 1, 4, para 42, 41, 40, 47, 31, 33-34, 37-39. As discussed above, Acuna in view of Burlina teaches the first training dataset and second training dataset. Burlina, col 3:25-46; col 7:15-42; col 21:19-42.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the system for generating perception data using critic network, including a first training dataset that is based on real-world data collected by an autonomous vehicle having a machine learning model generate a second training dataset that is based on simulation data, wherein the simulation data corresponds to the real-world data of Burlina using the train a revised version of the machine learning model using the first training dataset and the second training dataset, wherein the revised version of the machine learning model includes a discriminator head that is configured to generate an input classifier that indicates whether input data to the machine learning model corresponds to real-world input data or simulated input data of Acuna, with a reasonable expectation of success, in order to provide to minimize the reality gap between simulated and real-world domains .Acuna, para 5-6, 2-4. This would have provided the advantages of improving training and performance of autonomous systems and applications.
Claim(s) 15 is rejected under 35 U.S.C. 103 as being unpatentable over Burlina and Acuna as applied to claim 14 above, and further in view of Choe et al. (Pub. No. US 2021/0309248 A1, published October 7, 2021) hereinafter Choe.
Regarding claim 15, which depends from claim 14, and further recites:
convert the real-world data into the simulation data.
Burlina in view of Acuna teaches system of claim 14 from which claim 15 depends, including the real-world data and the simulation data. Burlina in view of Acuna does not specifically disclose convert the real-world data into the simulation data.
However, Choe teaches in the field related to autonomous vehicles and machine learning. Choe, para 2. Choe, which is analogous to the claimed invention because Choe is directed to training machine learning models to detect objects using real-world images augmented with simulated objects, teaches that, In various examples, systems and methods are disclosed that preserve rich, detail-centric information from a real-world image by augmenting the real-world image with simulated objects (convert the real-world data into the simulation data) to train a machine learning model to detect objects in an input image. Choe, abstract, para 4, 62. [0062] Now referring to FIG. 7, FIG. 7 is data flow diagram illustrating an example process 700 for training a machine learning model to detect road debris, in accordance with some embodiments of the present disclosure. Training image data 702 may include image data, such as real-world images augmented with simulated objects (e.g., road debris, road signs) (convert the real-world data into the simulation data) that satisfy a set of constraints. As described herein, FIG. 1 describes the process of augmenting a real-world image with augmented objects (convert the real-world data into the simulation data) and FIG. 6 is an example of a training image. The training image data 702 may also include images that contain objects that the machine learning model was not trained with. For example, the machine learning model may be previously trained with images containing cardboard boxes and deceased animals. The training image data 702, however, may also include images that have objects not present in the prior training images, such as construction cones and mattresses in order to implement zero-shot learning. Choe, Abstract, para 62, 4.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the system for generating perception data using critic network, including a first training dataset that is based on real-world data collected by an autonomous vehicle having a machine learning model generate a second training dataset that is based on simulation data, wherein the simulation data corresponds to the real-world data of Burlina using the train a revised version of the machine learning model using the first training dataset and the second training dataset, wherein the revised version of the machine learning model includes a discriminator head that is configured to generate an input classifier that indicates whether input data to the machine learning model corresponds to real-world input data or simulated input data of Acuna and the convert the real-world data into the simulation data of Choe, with a reasonable expectation of success, in order to provide to minimize the reality gap between simulated and real-world domains and to help provide for generating a practical, sound, and reliable autonomous driving system. Acuna, para 5-6, 2-4. Choe, para 3, 2-5. This would have provided the advantages of improving training and performance of autonomous systems and applications.
Claim(s) 19 is rejected under 35 U.S.C. 103 as being unpatentable over Burlina and Acuna as applied to claim 14 above, and further in view of Venkatadri.
Regarding claim 19, which depends from claim 14, and further recites:
determine a fidelity score that is based on the input classifier.
Burlina in view of Acuna teaches the system of claim 14 from which claim 19 depends. Burlina in view of Acuna does not specifically disclose determine a fidelity score that is based on the input classifier.
However, Venkatadri teaches in the field related to devices, systems, and methods for generating artificial scenarios for autonomous vehicles. Venkatadri, para 2. Venkatadri, which is analogous to the claimed invention because Venkatadri is directed to generating artificial scenarios for autonomous vehicles, teaches that, As an example, the machine-learned discriminator model(s) can determine an authenticity associated with each parameter used to generate the artificial data and/or determine an overall authenticity associated with the artificial data (determine a fidelity score that is based on the input classifier (determine an authenticity fidelity score that is based on the input discriminator classifer)) (e.g., authenticity of a simulated scenario, authenticity of a simulated log data set, etc.). The machine-learned discriminator model(s) can determine the authenticity data based on the authenticity for each parameter and/or the overall authenticity. Venkatadri, para 41, 47, 127, 5, 23.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the system for generating perception data using critic network, including a first training dataset that is based on real-world data collected by an autonomous vehicle having a machine learning model generate a second training dataset that is based on simulation data, wherein the simulation data corresponds to the real-world data of Burlina using the train a revised version of the machine learning model using the first training dataset and the second training dataset, wherein the revised version of the machine learning model includes a discriminator head that is configured to generate an input classifier that indicates whether input data to the machine learning model corresponds to real-world input data or simulated input data of Acuna and the determine a fidelity score that is based on the input classifier of Venkatadri, with a reasonable expectation of success, in order to provide to minimize the reality gap between simulated and real-world domains and to help improve the safety of passengers of an autonomous vehicle, improve the safety of the surroundings of the autonomous vehicle, improve the experience of the rider and/or operator of the autonomous vehicle. Acuna, para 5-6, 2-4. Venkatadri, para 23, 5, 9. This would have provided the advantages of improving training and performance of autonomous systems and applications.
Claim(s) 20 is rejected under 35 U.S.C. 103 as being unpatentable over Burlina, Acuna and Choe as applied to claim 19 above, and further in view of Al Faruque et al. (Pub No. US 2023/0230484 A1, published April 30, 2020) hereinafter Al Faruque.
Regarding claim 20, which depends from claim 19, and recites:
wherein the fidelity score is based on an area under a receiver operating characteristic curve.
Burlina in view of Acuna and Venkatadri teaches the system of claim 19 from which claim 20 depends, including the fidelity score. Burlina in view of Acuna and Venkatadri does not specifically disclose based on an area under a receiver operating characteristic curve.
However, Al Faruque teaches in the field related to a spatiotemporal scene-graph embedding methodology that models scene-graphs and resolves safety-focused tasks for autonomous vehicles. Al Faruque, para 3. Al Faruque which is analogous to the claimed invention because Ref is directed to simulated datasets, scene-graphs, and autonomous vehicle applications, teaches that, Each model's performance was evaluated by measuring its classification accuracy and the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) for each dataset (based on an area under a receiver operating characteristic curve). The classification accuracy is the ratio of the number of correct predictions on the test set of a dataset to the total number of samples in the testing set. AUC, sometimes referred to as a balanced accuracy measure, measures the probability that a binary classifier ranks a positive sample more highly than a random negative sample. This was a more balanced measure for measuring accuracy, especially with imbalanced datasets. Al Faruque, Fig 17, para 165, 180,159.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the system for generating perception data using critic network, including a first training dataset that is based on real-world data collected by an autonomous vehicle having a machine learning model generate a second training dataset that is based on simulation data, wherein the simulation data corresponds to the real-world data of Burlina using the train a revised version of the machine learning model using the first training dataset and the second training dataset, wherein the revised version of the machine learning model includes a discriminator head that is configured to generate an input classifier that indicates whether input data to the machine learning model corresponds to real-world input data or simulated input data of Acuna and the determine a fidelity score that is based on the input classifier of Venkatadri and based on an area under a receiver operating characteristic curve of Al Faruque, with a reasonable expectation of success, in order to provide to minimize the reality gap between simulated and real-world domains and to help improve the safety of passengers of an autonomous vehicle, improve the safety of the surroundings of the autonomous vehicle, improve the experience of the rider and/or operator of the autonomous vehicle and to provide for the development of safe and robust AVs and a model that can transfer knowledge gained from a simulated training set to a real-world testing set effectively will likely perform better in unseen real-world scenarios. Acuna, para 5-6, 2-4. Venkatadri, para 23, 5, 9. Al Faruque, para 4, 8, 165. This would have provided the advantages of improving training and performance of autonomous systems and applications.
Allowable Subject Matter
Claims 5, 12, 16-17 would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and if the rejections as being indefinite are overcome and if the rejections as being directed to an abstract idea are overcome.
Claim 18 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US-20210073584-A1, US-20220196839-A1, US-11410388-B1, US-20230063601-A1, US-11734786-B2, US-20230377324-A1, US-20240127048-A1, US-20210166474-A1.
H. Hu, Z. Qiao, M. Cheng, Z. Liu and H. Wang, "DASGIL: Domain Adaptation for Semantic and Geometric-Aware Image-Based Localization," in IEEE Transactions on Image Processing, vol. 30, pp. 1342-1353, 2021, doi: 10.1109/TIP.2020.3043875.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BARBARA LEVEL whose telephone number is (303)297-4748. The examiner can normally be reached Monday through Friday 8:00 AM - 5:00 PM MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BARBARA M LEVEL/ Examiner, Art Unit 2142
Prosecution Insights