DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Responsive to the communication dated 07/07/2023
Claims 1-20 are presented for examination
Information Disclosure Statement
The IDS dated 07/07/2023, 11/22/2023, and 09/25/2026 has been reviewed. See attached.
Drawings
The drawings dated 07/07/2023 have been reviewed. They are accepted.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Abstract
The abstract dated 07/07/2023 has been reviewed. It has 111 words, and contains no legal phraseology. It is accepted.
Claim Objections
Claims 2, 4-7, 9 -11, 12-14, 16, and 18-19 are objected to because of the following informalities:
Claim 2 recites “receiving the sensor data transmitted by the simulation device” as no transmitted sensor data was previously introduced, to avoid potential issues with antecedent basis it is recommended to amend the claim to instead read “receiving the rendered sensor data from the simulation device…”
Claim 4 should be amended to read “…to represent the physical sensor, and the computing device is communicatively…” to improve readability.
Claim 6 recites “a physical scene when a fault of the fault type occurs in the sensor during the road test,” As it is clear this fault during the road test is meant to refer to a fault in the physical sensor, it is recommended to amend this claim to read “a physical scene when a fault of the fault type occurs in the physical sensor during the road test” to improve readability and avoid potential issues with antecedent basis.
Claim 6 recites “triggering, … the fault to start to occur.” As it is clear that this triggered fault is in the simulation, and not the physical fault that caused the trigger in the first place, it is recommended to amend the claim to instead read “triggering, … a fault to start to occur in the simulation scene.” to avoid potential issues with antecedent basis
Claim 7 recites “acquiring, in response to the data source types being the database source…” As only a single “data source type” was previously introduced, it is recommended to amend the claim to instead read “acquiring, in response to the data source type being the database source…” to avoid potential issues with antecedent basis.
Claim 9 recites “the fault type of the image acquisition apparatus comprises at least one of the followings: …” To improve readability, it is suggested to replace “one of the followings:” with “one of the following:”
Claim 9 recites “an image appearing a puzzle, an image appearing a stripe, an image appearing chromatic aberration, an image appearing a flare, an image appearing a ghost, image exposure abnormality, a lens attached by a foreign body, camera out of focus, camera looseness, or camera powered off.” It is recommended to reword these limitations to make them more clear and avoid potential issues with antecedent basis by adding proper articles where needed.
For example “image exposure abnormality” should instead read “an image exposure abnormality,” “camera powered off” should instead read “a camera powered off,” etc. Please ensure that these changes are made where required to all the elements listed in claim 9.
“a lens attached by a foreign body” should instead read “a foreign body attached to a lens”
Claim 10 recites “the fault type of the point cloud acquisition apparatus comprises at least one of the followings: …the fault type of the integrated navigation device comprises at least one of the followings:” To improve readability, it is suggested to replace “one of the followings:” with “one of the following:”
Additionally, for claim 10, it is suggested to replace “comprises” with “further comprises” when discussing the integrated navigation device, so the limitation reads “wherein the sensor further comprises an integrated navigation device…”
Claim 10 recites “a point cloud having no discrimination, excessive noise points, a point cloud appearing a ghost, a point cloud appearing a mirror image, point cloud points loss, communication interruption, sensor looseness, or a sensor surface attached by a foreign body; wherein the sensor comprises an integrated navigation device, and the fault type of the integrated navigation device comprises at least one of the followings: orientation angle error, positioning accuracy deviation, communication interruption, or sensor looseness.” It is recommended to reword these limitations to make them more clear and avoid potential issues with antecedent basis by adding proper articles where needed.
For example “orientation angle error” should instead read “an orientation angle error,” “positioning accuracy deviation” should instead read “a positioning accuracy deviation,” etc. Please ensure that these changes are made where required to all the elements listed in claim 10.
“a sensor surface attached by a foreign body;” should instead read “a foreign body attached to a sensor surface”
Claim 12 recites “to obtain the sensor data of the frame…” As no frame-specific sensor data was previously introduced, to avoid potential issues with antecedent basis, it is recommended to amend the claim to instead read “to obtain sensor data of the frame…”
Claim 16 recites “to obtain the sensor data of the frame…” As no frame-specific sensor data was previously introduced, to avoid potential issues with antecedent basis, it is recommended to amend the claim to instead read “to obtain sensor data of the frame…”
Claim 18 recites “receiving the sensor data transmitted by the simulation device” as no transmitted sensor data was previously introduced, to avoid potential issues with antecedent basis it is recommended to amend the claim to instead read “receiving the rendered sensor data from the simulation device…”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 9 and 10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “an image appearing a puzzle” in claim 9 is a relative term which renders the claim indefinite. The term “an image appearing a puzzle” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. It is not clear what exactly an image “appearing a puzzle” is meant to describe, and additional detail is not given in the rest of the disclosure. What defines something as being puzzle-like and the line that divides an image that “looks like a puzzle” and a image that “does not look like a puzzle” is a matter of subjective opinion.
The term “an image appearing a stripe” in claim 9 is a relative term which renders the claim indefinite. The term “an image appearing a stripe” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. What defines an image as being sufficiently striped to be generally described as “appearing a stripe” is a matter of subjective opinion.
The term “an image appearing chromatic aberration” in claim 9 is a relative term which renders the claim indefinite. The term “an image appearing chromatic aberration” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. What defines an image as having sufficient aberration to be generally described as “appearing chromatic aberration” is a matter of subjective opinion.
The term “an image appearing a flare” in claim 9 is a relative term which renders the claim indefinite. The term “an image appearing a flare” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. What defines an image as having sufficient flare-like qualities to be generally described as “appearing a flare” is a matter of subjective opinion.
The term “an image appearing a ghost” in claim 9 is a relative term which renders the claim indefinite. The term “an image appearing a ghost” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. What defines an image as having sufficient ghost-like qualities to be generally described as “appearing a ghost” is a matter of subjective opinion.
The term “a point cloud appearing a ghost” in claim 10 is a relative term which renders the claim indefinite. The term “a point cloud appearing a ghost” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. What defines an point cloud as having sufficient ghost-like qualities to be generally described as “appearing a ghost” is a matter of subjective opinion.
The term “a point cloud appearing a mirror image” in claim 10 is a relative term which renders the claim indefinite. The term “a point cloud appearing a mirror” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. What defines an point cloud as having sufficient mirrored elements to be generally described as “appearing a mirror image” is a matter of subjective opinion. Does the entire point cloud have to be mirrored? 50% of the point cloud? A single point mirrored from its correct position?
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because they are directed to an abstract idea without significantly more.
Claim 1 (Statutory Category – Process)
Step 2A – Prong 1: Judicial Exception Recited?
Yes, the claim recites a mental process, specifically:
MPEP 2106.04(a)(2)(Ill): “Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, Judgments, and opinions.”
Further, the MPEP recites “The courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation.”
A sensor data generation method, adapted to be executed in a computing device, the method comprising: receiving information about a fault type of a sensor; generating an instruction corresponding to the fault type of the sensor, the instruction comprising a fault parameter, and the fault parameter comprising an injection mode of the fault type and a fault occurrence probability; and generating target sensor data based on the injection mode of the fault type and the fault occurrence probability.
Receiving the information about the fault type, when recited at such a high level of generality, merely consists of collection of information and understanding that information, which has been recognized by the courts as an example of a mental process (See the 2024 Guidance Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence section (III)(A)(c) “Additional examples of mental processes are:
A claim to a method of “(1) receiving user information; (2) providing a polling question; (3) receiving and storing an answer; (4) comparing that answer to generate a `likelihood of match' with other users; and (5) displaying certain user profiles based on that likelihood” could practically be performed in the human mind (i.e., “[a] human mind could review people's answers to questions and identify matches based on those answers”), Trinity Info Media, LLC v. Covalent, Inc., 72 F.4th 1355, 1362 (Fed. Cir. 2023).
A claim to “the collection of information from various sources (a Federal database, a State database, and a case worker) and understanding the meaning of that information (determining whether a person is receiving SSDI benefits and determining whether they are eligible for benefits under the law),” where “ `[t]hese steps can be performed by a human, using “observation, evaluation, judgment, [and] opinion,” because they involve making determinations and identifications, which are mental tasks humans routinely do,' ” and thus can practically be performed in the human mind, In re Killian, 45 F.4th 1373, 1379 (Fed. Cir. 2022).”)
Generating such simulated sensor data based on a particular injection mode, fault type, and fault probability is a mental process. Firstly, the values for fault type, injection mode and probability could be arbitrarily judged mentally, for example a person might decide that, for an image sensor, the fault type might be that the camera view is entirely obscured, the injection mode will be the processing of road data, and the probability 100%. With this in mind, the person could generate a representation of the sensor output with a pencil and paper, i.e. a solid black box representative of a camera with a completely obscured view. Other sensor types can be similarly depicted by drawing or writing out representations (e.g. writing coordinates for a GPS, a series of points to represent LIDAR data, etc.)
Further, a person could arbitrarily mentally decide a probability, say 50%, and determine outputs based on that probability by randomly coming up with numbers in the range of 0-100% and judging whether they were above or below that probability.
Should it be found that receiving information about the fault type is not a mental process, it is also an example of mere data gathering.
Additionally, performing this process on a generic computer amounts to no more than mere instructions to apply.
Step 2A – Prong 2: Integrated into a Practical Solution?
Insignificant Extra-Solution Activity (MPEP 2106.05(g)) has found mere data gathering and
post solution activity to be insignificant extra-solution activity.
Data gathering:
receiving information about a fault type of a sensor;
Generically receiving this information, without any specificity as to how this receiving process is actually performed, is simply the act of gathering that data, and therefore amounts to no more than mere data gathering.
Moreover, Mere Instructions To Apply An Exception (MPEP 2106.05(f)) has found that simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. In light of this, the additional generic computer component elements of “A sensor data generation method, adapted to be executed in a computing device, the method comprising…” are not sufficient to integrate a judicial exception into a practical application nor provide evidence of an inventive concept.
Step 2B: Claim provides an Inventive Concept?
No, as discussed with respect to Step 2A, the additional limitations are Insignificant Extra-Solution Activity and Mere Instructions To Apply An Exception and do not impose any meaningful limits on practicing the abstract idea and therefore the claim does not provide an inventive concept in Step 2B.
Insignificant Extra-Solution Activity (MPEP 2106.05(g)) has found mere data gathering and
post solution activity to be insignificant extra-solution activity.
Data gathering:
receiving information about a fault type of a sensor;
Generically receiving this information, without any specificity as to how this receiving process is actually performed, is simply the act of gathering that data, and therefore amounts to no more than mere data gathering.
A claim element that amounts to merely gathering data is not indicative of integration into a practical solution nor evidence that the claim provides an inventive concept or significantly more, as exemplified by ((MPEP 2106.05)(g)(Mere Data Gathering) i. Performing clinical tests on individuals to obtain input for an equation, In re Grams, 888 F.2d 835, 839-40; 12 USPQ2d 1824, 1827-28 (Fed. Cir. 1989); iv. Obtaining information about transactions using the Internet to verify credit card transactions, CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011);
Moreover, Mere Instructions To Apply An Exception (MPEP 2106.05(f)) has found that simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. In light of this, the additional generic computer component elements of “A sensor data generation method, adapted to be executed in a computing device, the method comprising…” are not sufficient to integrate a judicial exception into a practical application nor provide evidence of an inventive concept.
The additional elements have been considered both individually and as an ordered combination in the consideration of whether they constitute significantly more, and have been determined not to constitute such.
The claim is ineligible.
Claim 15 The elements of claim 15 are substantially the same as those of claim 1. Therefore, the elements of claim 15 are rejected due to the same reasons as outlined above for claim 1. Further, as to the elements present in claim 15 but not claim 1:
sending the sensor data to the computing device.
The courts have explicitly recognizing sending and transmitting data in a generic manner as an example of mere instructions to apply. (MPEP 2106.05(f)(2): Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, "claiming the improved speed or efficiency inherent with applying the abstract idea on a computer" does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015).)
Further, such generic data transmission, as well as storing and retrieving information in memory, are also explicitly recognized by the courts as examples of well-understood, routine, conventional activity. (MPEP 2106.05(d)(II): The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
iii. Electronic recordkeeping, Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 225, 110 USPQ2d 1984 (2014) (creating and maintaining “shadow accounts”); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log);
iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;)
Moreover, Mere Instructions To Apply An Exception (MPEP 2106.05(f)) has found that simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. In light of this, the additional generic computer component elements of “A sensor data generation method, adapted to be executed in a simulation device, the method comprising: receiving an instruction sent by a computing device… sending the sensor data to the computing device.” are not sufficient to integrate a judicial exception into a practical application nor provide evidence of an inventive concept.
Claim 17. The elements of claim 17 are substantially the same as those of claim 1. Therefore, the elements of claim 17 are rejected due to the same reasons as outlined above for claim 1.
Moreover, Mere Instructions To Apply An Exception (MPEP 2106.05(f)) has found that simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. In light of this, the additional generic computer component elements of “A computing device, comprising a storage, and one or more processors communicatively connected with the storage; wherein the storage stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors to cause the one or more processors to implement a method comprising:..” are not sufficient to integrate a judicial exception into a practical application nor provide evidence of an inventive concept.
Claim 2 recites “wherein the computing device is communicatively connected with a simulation device in which a virtual vehicle is simulated, and
Applying a computer to perform a generic simulation at a high level of generality is simply the act of instructing a computer to perform generic functions to perform that simulation, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the virtual vehicle is “simulated” without reciting how this simulation is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations;
The courts have found that such mere instructions to apply are not indicative of integration into a practical application nor recitation of significantly more than the judicial exception (MPEP 2106.05(f) “Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do "‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’". Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983”)
Additionally, specifying that the simulation device is connected to the computing device merely clarifies the physical configuration of the generic computing setup, and therefore is merely an extension of the mere instructions to apply.
generating the target sensor data based on the injection mode of the fault type and the fault occurrence probability comprises: sending the instruction to the simulation device so as to enable the simulation device to render sensor data based on the injection mode of the fault type and the fault occurrence probability from a perspective of the sensor in the simulation device; and
Generating such simulated sensor data based on a particular injection mode, fault type, and fault probability is a mental process. Firstly, the values for fault type, injection mode and probability could be arbitrarily judged mentally, for example a person might decide that, for an image sensor, the fault type might be that the camera view is entirely obscured, the injection mode will be the processing of road data, and the probability 100%. With this in mind, the person could generate a representation of the sensor output with a pencil and paper, i.e. a solid black box representative of a camera with a completely obscured view.
Further, a person could arbitrarily mentally decide a probability, say 50%, and determine outputs based on that probability by randomly coming up with numbers in the range of 0-100% and judging whether they were above or below that probability.
Additionally, performing this process on a generic computer amounts to no more than mere instructions to apply.
receiving the sensor data transmitted by the simulation device as the target sensor data.”
Generically receiving this data, without any specificity as to how this receiving process is actually performed, is simply the act of gathering that data, and therefore amounts to no more than mere data gathering.
A claim element that amounts to merely gathering data is not indicative of integration into a practical solution nor evidence that the claim provides an inventive concept or significantly more, as exemplified by ((MPEP 2106.05)(g)(Mere Data Gathering) i. Performing clinical tests on individuals to obtain input for an equation, In re Grams, 888 F.2d 835, 839-40; 12 USPQ2d 1824, 1827-28 (Fed. Cir. 1989); iv. Obtaining information about transactions using the Internet to verify credit card transactions, CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011);
Claim 3 recites “wherein the injection mode of the fault type comprises a mode of adjusting at least one of an internal parameter or an external parameter of the sensor in a simulation device.”
This merely clarifies how the fault injection is represented within the computer implementation of the mental process of generating the fault-effected sensor data, and is therefore merely an extension of the mental process and mere instructions to apply.
Claim 4 recites “wherein the sensor is a virtual sensor in a simulation device, the virtual sensor has a same parameter as a physical sensor to represent the physical sensor, the computing device is communicatively connected with a database storing road test data collected by the physical sensor during a road test.”
Clarifying that the sensor is a virtual sensor in a simulation device and that it has the same configuration as a physical sensor merely further specifies how the sensor data generation mental process is implemented on a computer, and is therefore merely an extension of the mental process and mere instructions to apply.
Further, clarifying that the system is connected with a database merely further specifies the physical setup of the generic computing components, and therefore is merely an extension of the mere instructions to apply.
Claim 5 recites “wherein the injection mode of the fault type comprises a mode of processing the road test data, and generating the target sensor data based on the injection mode of the fault type and the fault occurrence probability comprises: acquiring the road test data of the physical sensor from the database; and processing, according to the injection mode of the fault type and the fault occurrence probability in the instruction, the road test data to obtain the target sensor data.”
Receiving data from a database has been recognized by the courts as an example of a mental process (See the 2024 Guidance Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence section (III)(A)(c) “Additional examples of mental processes are:
A claim to a method of “(1) receiving user information; (2) providing a polling question; (3) receiving and storing an answer; (4) comparing that answer to generate a `likelihood of match' with other users; and (5) displaying certain user profiles based on that likelihood” could practically be performed in the human mind (i.e., “[a] human mind could review people's answers to questions and identify matches based on those answers”), Trinity Info Media, LLC v. Covalent, Inc., 72 F.4th 1355, 1362 (Fed. Cir. 2023).
A claim to “the collection of information from various sources (a Federal database, a State database, and a case worker) and understanding the meaning of that information (determining whether a person is receiving SSDI benefits and determining whether they are eligible for benefits under the law),” where “ `[t]hese steps can be performed by a human, using “observation, evaluation, judgment, [and] opinion,” because they involve making determinations and identifications, which are mental tasks humans routinely do,' ” and thus can practically be performed in the human mind,In re Killian, 45 F.4th 1373, 1379 (Fed. Cir. 2022).”)
Additionally, should it be found that obtaining this data from that database is not a mental process, it is also an example of mere data gathering, mere instructions to apply, and WURC.
Mere data gathering:
Generically “acquiring” road test data from a database, without any specificity as to how this acquisition is accomplished, amounts to no more than gathering that data, and therefore equates to mere data gathering.
Mere instructions to apply:
The courts have explicitly recognizing sending and transmitting data in a generic manner as an example of mere instructions to apply. (MPEP 2106.05(f)(2): Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, "claiming the improved speed or efficiency inherent with applying the abstract idea on a computer" does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015).)
WURC:
Further, such generic data acquisition, as well as storing and retrieving information in memory, are also explicitly recognized by the courts as examples of well-understood, routine, conventional activity. (MPEP 2106.05(d)(II): The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
iii. Electronic recordkeeping, Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 225, 110 USPQ2d 1984 (2014) (creating and maintaining “shadow accounts”); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log);
iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;)
Generating such simulated sensor data based on a physical sensor data, fault type, and fault probability is a mental process. Firstly, the sensor data from the database could be observed; in the case of image data, this would merely be the act of looking at a picture. Next, the values for fault type and probability could be arbitrarily judged mentally, for example a person might decide that, the fault type might be that the camera view is obscured by a large circle in the center, and the probability 100%. With this in mind, the person could generate a representation of the target sensor output with a pencil and paper by first drawing a representation of the observed sensor data, e.g. recreating the picture, and then altering it as per the fault type, e.g. drawing a large circle in the center obscuring the original content.
Further, a person could arbitrarily mentally decide a probability, say 50%, and determine outputs based on that probability by randomly coming up with numbers in the range of 0-100% and judging whether they were above or below that probability.
Performing this process on a generic computer amounts to no more than mere instructions to apply.
Claim 6 recites “wherein the database further stores information of a physical scene when a fault of the fault type occurs in the sensor during the road test, the method further comprising: calculating a similarity between a simulation scene in the simulation device and the physical scene; and triggering, in response to the similarity being greater than or equal to a preset threshold, the fault to start to occur.”
Clarifying that the database stores information about when a fault occurs merely clarifies the contents of the database and is therefore merely an extension of the mental process, mere data gathering, mere instructions to apply, and WURC.
Calculating a numeric similarity metric is a mathematic calculation and therefore a mathematic concept.
Comparing a number to a threshold value is a mental process equivalent to observing both numbers and judging whether the first number is at or surpasses the threshold value. Using the occurrence of this to decide when to start generating the fault-effected sensor data merely clarifies when that mental process is performed, and is therefore mere an extension of the mental process, mere data gathering, mere instructions to apply, and WURC.
Claim 7 recites “wherein the fault parameter further comprises a data source type comprising at least one of a simulation device source or a database source, and generating the target sensor data based on the injection mode of the fault type and the fault occurrence probability comprises: sending, in response to the data source type being the simulation device source, the instruction to the simulation device to receive the sensor data from the simulation device as the target sensor data; and acquiring, in response to the data source types being the database source, the road test data from the database, and processing, according to the instruction, the road test data to obtain the target sensor data.”
Choosing whether to use data from a real sensor or a purely simulated data is a mental process equivalent to making a judgement. If the purely simulated data is chosen, it can be generated as described for claim 1. If the road data is chosen, the fault-effected data can be generated as described for claim 5.
Claim 8 recites “wherein the fault type of a fault comprises at least one of a data quality fault, a sensor location fault, a sensor hardware fault, or a data transmission fault; the data transmission fault comprising at least one of a data frame loss, a data frame drop, or a data frame error; and the fault parameter further comprises an occurrence duration of the fault.”
This merely clarifies various conditions as to how the mental process of generating the sensor data is performed, and is therefore merely an extension of the mental process, and mere instructions to apply of generating said fault-effected data.
Claim 9 recites “wherein the sensor comprises an image acquisition apparatus, and the fault type of the image acquisition apparatus comprises at least one of the followings: an image appearing a puzzle, an image appearing a stripe, an image appearing chromatic aberration, an image appearing a flare, an image appearing a ghost, image exposure abnormality, a lens attached by a foreign body, camera out of focus, camera looseness, or camera powered off.”
This merely further clarifies the content of the drawn/written fault-effected sensor data, and is therefore merely an extension of the mental process, and mere instructions to apply of generating said fault-effected data.
Claim 10 recites “wherein the sensor comprises a point cloud acquisition apparatus, and the fault type of the point cloud acquisition apparatus comprises at least one of the followings: a point cloud having no discrimination, excessive noise points, a point cloud appearing a ghost, a point cloud appearing a mirror image, point cloud points loss, communication interruption, sensor looseness, or a sensor surface attached by a foreign body; wherein the sensor comprises an integrated navigation device, and the fault type of the integrated navigation device comprises at least one of the followings: orientation angle error, positioning accuracy deviation, communication interruption, or sensor looseness.”
This merely further clarifies the content of the drawn/written fault-effected sensor data, and is therefore merely an extension of the mental process and mere instructions to apply of generating said fault-effected data.
Claim 11 recites “wherein the mode of processing the road test data comprises at least one of: a mode of processing image data, a mode of processing point cloud data, or a mode of processing positioning data.”
This merely clarifies the type of sensor data that the fault-effected sensor data is generated from, and therefore is merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Claim 12 recites “further comprising: sending a sensor model of the sensor to the simulation device
Firstly, sending data in a generic manner has been recognized by the as an example of mere instructions to apply. (MPEP 2106.05(f)(2): Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, "claiming the improved speed or efficiency inherent with applying the abstract idea on a computer" does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015).)
WURC:
Further, such generic data transmission, as well as storing and retrieving information in memory, are also explicitly recognized by the courts as examples of well-understood, routine, conventional activity. (MPEP 2106.05(d)(II): The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
iii. Electronic recordkeeping, Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 225, 110 USPQ2d 1984 (2014) (creating and maintaining “shadow accounts”); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log);
iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;)
to enable the simulation device to determine, based on a vehicle motion model, poses of the virtual vehicle at different moments in an acquisition period of a frame, and to render environmental data at the different moments based on the sensor model to obtain the sensor data of the frame,
Creating such an frame of sensor data based on a vehicle moving during the acquisition time of the frame is equivalent to a mental process of drawing an image that is warped by motion, as by bending certain elements of the image as they would be in an image taken with a rolling shutter.
Further, doing this within a generic computer simulation amounts to no more than mere instructions to apply.
Applying a computer to perform a generic simulation at a high level of generality is simply the act of instructing a computer to perform generic functions to perform that simulation, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the virtual vehicle is “simulated” and data at different moments is “rendered” without reciting how this simulation or rendering is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations.
wherein the sensor model comprises a rendering scheme of the sensor data and an initial pose of the sensor relative to the virtual vehicle.”
Specifying such a model is a mental process equivalent to judging where the sensor is on the vehicle and how the representation of the sensor data is to be produced.
Claim 13 recites “transmitting the target sensor data to an autonomous driving software system,
Firstly, sending data in a generic manner has been recognized by the as an example of mere instructions to apply. (MPEP 2106.05(f)(2): Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, "claiming the improved speed or efficiency inherent with applying the abstract idea on a computer" does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015).)
WURC:
Further, such generic data transmission, as well as storing and retrieving information in memory, are also explicitly recognized by the courts as examples of well-understood, routine, conventional activity. (MPEP 2106.05(d)(II): The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
iii. Electronic recordkeeping, Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 225, 110 USPQ2d 1984 (2014) (creating and maintaining “shadow accounts”); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log);
iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;)
so as to enable the autonomous driving software system to generate a traveling strategy based on the target sensor data, and the virtual vehicle in the simulation device to perform fault simulation based on the traveling strategy, wherein the fault occurrence probability comprises a data transmission fault probability, and transmitting the target sensor data to the autonomous driving software system comprises: transmitting, according to the data transmission fault probability, the target sensor data to the autonomous driving software system.
Generating a travelling strategy based on sensor data, such as a camera view from the front of a car, is a mental process equivalent to observing the road and making decisions as to how the car should be driven. This kind of mental process is performed by millions of people every day driving their vehicle.
Generating a data transmission fault is equivalent to drawing a representation of the sensor data when such a fault occurs. For example, if the fault is that a camera sensor is disconnected, the camera view can be represented by drawing a blank camera frame to indicate that no camera data was transmitted.
Performing this process on a generic computer amounts to no more than mere instructions to apply.
Further, Applying a computer to perform a generic simulation at a high level of generality is simply the act of instructing a computer to perform generic functions to perform that simulation, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the virtual vehicle is “simulated” without reciting how this simulation is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations.
Finally, as mentioned above, transmitting data between components, either over a network or by storing and retrieving data in memory, is recognized by the courts as an example of mere instructions to apply and WURC.
Claim 14 recites “wherein transmitting the target sensor data to the autonomous driving software system comprises: converting a format of the target sensor data into a format of data acquired by the sensor during a road test; and transmitting the format-converted data to the autonomous driving software system.”
Converting data between formats is a mathematic concept explicitly recognized by the courts; see (MPEP 2106.04(a)(2)(I): Examples of mathematical relationships recited in a claim include: … ii. a conversion between binary coded decimal and pure binary, Benson, 409 U.S. at 64, 175 USPQ at 674;)
Performing such a conversion on a generic computing system amounts to no more than mere instructions to apply.
Transmitting data in a generic manner has been recognized by the as an example of mere instructions to apply. (MPEP 2106.05(f)(2): Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, "claiming the improved speed or efficiency inherent with applying the abstract idea on a computer" does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015).)
WURC:
Further, such generic data transmission, as well as storing and retrieving information in memory, are also explicitly recognized by the courts as examples of well-understood, routine, conventional activity. (MPEP 2106.05(d)(II): The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
iii. Electronic recordkeeping, Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 225, 110 USPQ2d 1984 (2014) (creating and maintaining “shadow accounts”); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log);
iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;)
Claim 16 recites “wherein a virtual vehicle is simulated in the simulation device,
Applying a computer to perform a generic simulation at a high level of generality is simply the act of instructing a computer to perform generic functions to perform that simulation, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the virtual vehicle is “simulated” without reciting how this simulation is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations;
The courts have found that such mere instructions to apply are not indicative of integration into a practical application nor recitation of significantly more than the judicial exception (MPEP 2106.05(f) “Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do "‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’". Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983”)
the method further comprising: receiving a sensor model of the sensor, the sensor model comprising a rendering scheme of the sensor data and an initial pose of the sensor relative to the virtual vehicle;
Generically receiving such model data, without any specificity as to how the receiving process is actually carried out, amounts to no more than mere data gathering.
wherein rendering the sensor data according to the injection mode of the fault type and the fault occurrence probability comprises: predicting poses of the virtual vehicle at different moments in an acquisition period of a frame; and rendering environmental data at the different moments based on the sensor model, to obtain the sensor data of the frame.”
Creating such an frame of sensor data based on a vehicle moving during the acquisition time of the frame is equivalent to a mental process of drawing an image that is warped by motion, as by bending certain elements of the image as they would be in an image taken with a rolling shutter.
Further, doing this within a generic computer simulation amounts to no more than mere instructions to apply.
Applying a computer to perform a generic simulation at a high level of generality is simply the act of instructing a computer to perform generic functions to perform that simulation, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the virtual vehicle is “simulated” and data at different moments is “rendered” without reciting how this simulation or rendering is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations.
Claim 18 The elements of claim 18 are substantially the same as those of claim 2. Therefore, the elements of claim 18 are rejected due to the same reasons as outlined above for claim 2.
Claim 19 recites “wherein the injection mode of the fault type comprises at least one of: a mode of adjusting at least one of an internal parameter or an external parameter of the sensor in the simulation device; or
This merely clarifies how the fault injection is represented within the computer implementation of the mental process of generating the fault-effected sensor data, and is therefore merely an extension of the mental process and mere instructions to apply.
a mode of processing road test data of the sensor acquired from a database, the mode of processing road test data comprises at least one of: a mode of processing image data, a mode of processing point cloud data, or a mode of processing positioning data.
This merely clarifies the type of sensor data that the fault-effected sensor data is generated from, and therefore is merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Claim 20 recites “A non-transitory computer-readable storage medium, comprising programs or instructions, wherein the programs or the instructions, when operated on a computer, implement a method according to claim 1.”
This claim merely specifies generic computer hardware on which to perform the process of claim 1, and therefore amounts to no more than mere instructions to apply.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
(1) Claims 1-3, 8-9, 13, 15 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hardware-in-the-Loop-Based Real-Time Fault Injection Framework for Dynamic Behavior Analysis of Automotive Software Systems (Hereinafter Abboush) in view of Assessment and Assurance of Autonomous Vehicle Safety (Hereinafter Cyriac)
Claim 1. Abboush teaches A sensor data generation method, adapted to be executed in a computing device, ([Abstract] “In this study, a real-time FI framework is proposed based on a hardware-in-the-loop (HiL) simulation platform and a real-time electronic control unit (ECU) prototype. The framework is modelled in the MATLAB/Simulink environment and implemented in the HiL simulation to enable the analysis process in real time during the V-cycle development process. With the objective of covering most of the potential faults, nine different types of sensor and actuator control signal faults are injected programmatically into the HiL system as single and multiple faults without changing the original system model.”) the method comprising: receiving information about a fault type of a sensor; ([Figure 2.] Shows the input of fault data from a fault library as part of the fault injection framework) generating an instruction corresponding to the fault type of the sensor, the instruction comprising a fault parameter, and the fault parameter comprising an injection mode of the fault type and ([Page 10 Par 2-3] “The main objective of the FI GUI is to support the tester in the HiL user environment during run-time FI. From the GUI, a tester can perform various actions. Some of them are: selecting the fault type from the fault library and the fault values associated with the selected fault type. In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space. According to the literature, there are several types of potential faults in the time series data, such as gain, offset/bias, noise, hard-over, spike, stuck-at, packet loss, delay and drift faults [59–61]. By analyzing the signal features that characterize automotive fault behavior, the authors of [62] categorized the common faults in the vehicle’s PCM signals, such as abnormal magnitudes, rolling, noise and dependency faults. In addition, they also named some direct and indirect causes of the defined faults. Dirty or deteriorated sensors, faulty vehicle components, vibrations caused by dangerous internal motors, and sudden acceleration are good examples of direct sources that can cause these types of faults. Conversely, bumpy roads and uncertainties in driving behavior can cause anomalies in signal behavior without necessarily provoking faults. Equation (1) and Table 2 are used to summarize the mathematical representation of the aforementioned types: {EQUATION} where f(t) is faulty or manipulated signal value, dv represents the gain value and h(t) is the healthy or standard signal value. ov represents the offset/bias value.”) hardware.”)
PNG
media_image1.png
875
828
media_image1.png
Greyscale
Abboush does not explicitly teach a fault occurrence probability; generating target sensor data based on the fault occurrence probability.
Cyriac makes obvious a fault occurrence probability; generating target sensor data based on the fault occurrence probability. ([Page 62 Par 1] “This chapter presents AVFI, a fault injection based methodology for assessing the end-to-end resilience and safety of AV systems. AVFI assesses the robustness of AVs empirically by injecting faults into various components of the AV system which recreate situations that may be rarely tested. The tool uses a state-of-the-art AV simulation framework [64] as a platform to perform fault injections in sensor inputs (e.g., camera, LIDAR), in the neural networks that make driving decisions for the AV and in the hardware/software components.” [Page 65 Par 2 – Page 67 Par 1] “The fault injection campaign in AVFI is performed in two stages. First, the location of where the fault should be injected is decided (i.e., input, neural network, output). Then, a fault model for the injection is chosen. The various fault models implemented in AVFI are discussed below. There are three classes of faults that AVFI is capable of injecting: 1) data faults, 2) hardware faults and 3) timing faults. 5.2.1 Data Faults AVFI injects data faults by modifying sensor measurements such as camera images, LIDAR and GPS data or world measurements like weather type and car speed captured by the AV system. A data fault in sensor measurements can occur due to changes in the external environment, a malfunctioning sensor or unseen perturbations of the images (e.g., broken road sign posts). In the current implementation, AVFI intercepts the RGB camera sensor data from the server, modifies it to a sensor-specific fault model and finally sends it to IL-CNN ADA. The various fault models explored for the camera sensor are Gaussian noise, salt-and-pepper noise, solid occlusion, translucent occlusion and water droplets. Figure 12 shows the various fault models for the camera sensor. Gaussian noise in camera images occurs due to sensor limitations during image acquisition. There is inherent noise produced by the camera sensor in low-light situations or unfavorable temperatures of the sensor [66]. Furthermore, the electric circuits operating the sensor can also inject electronic circuit noise [67]. Salt-and-pepper noise is generated when there are malfunctioning sensor pixels, analog-to-digital converter errors [68] or bit errors in transmission [69]. Solid and translucent occlusions can occur when a physical object or liquid gets stuck on the lens of the camera. Finally the water droplet model aims to simulate the effect of rain drops on camera sensors. AVFI allows for the configuration of each type of fault model. For example, users can specify different noise parameters for the Gaussian and salt-and-pepper noise. For the occlusion fault models, the size and location are randomized every frame and maximum and minimum bounds can be specified. Furthermore, the probability that any given frame is injected with a fault can also be configured. Finally the water droplet model allows the user to specify the number, size and speed of water droplets dripping down in the frame.”)
Cyriac is analogous art because it is within the field of autonomous vehicle testing systems. It would have been obvious to one of ordinary skill in the art to combine it with Abboush before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to expand the testing capabilities of the system introduced in Abboush to further systems. Cyriac notes that a major issue in the field of autonomous vehicle fault injection is the lack of systems that are capable of full end-to-end testing, with the majority of systems being limited to specific components or sets of components ([Page 62 Par 1] “Although there is substantial prior work in assessing the safety and resilience of individual components of AVs (e.g., neural networks [60], [61], sensors [62], mechanical components [63]), there are very few publicly available methods for holistic end-to-end safety assessment of these AV systems.”) It should be note that Abboush in particular only contemplates the use of internal sensors in its implementation examples, such as engine RPM sensors, gas pedal position, etc. ([Page 14 Par 2 – Page 15 Par 1] “In the case study, several sensor signals are available, such as the crank angle sensor, battery voltage, accelerator pedal position, ignition and starter demand, EGR mass flow, engine speed, intake and exhaust manifold pressure, fuel pressure, coolant temperature and railbar. The selected actuator control signals, on the other hand, are the control of the fuel metering unit per cylinder, the injection angle per cylinder, the injection time for direct injection and the control switches. Total information about the selected location of fault occurrence is shown in Table 3.”) To this end, Cyriac presents the AVFI fault injection system that greatly expands the capabilities of the fault injection system including handling of more complex sensors and injections into the decision making control systems themselves ([Page 62 Par 1] “This chapter presents AVFI, a fault injection based methodology for assessing the end-to-end resilience and safety of AV systems. AVFI assesses the robustness of AVs empirically by injecting faults into various components of the AV system which recreate situations that may be rarely tested. The tool uses a state-of-the-art AV simulation framework [64] as a platform to perform fault injections in sensor inputs (e.g., camera, LIDAR), in the neural networks that make driving decisions for the AV and in the hardware/software components. Detailed discussion in this chapter will be restricted to fault injections in the sensor inputs and output actuators. Furthermore, AVFI quantifies domain-specific failure metrics such as mission success rate, number of traffic violations per kilometer driven and time to traffic violation. Using these metrics jointly, together with lifelike simulations of the world describing pedestrians, cars and the environment, the safety of the AV can be evaluated.” [Page 65 Par 2] “The fault injection campaign in AVFI is performed in two stages. First, the location of where the fault should be injected is decided (i.e., input, neural network, output). Then, a fault model for the injection is chosen. The various fault models implemented in AVFI are discussed below. There are three classes of faults that AVFI is capable of injecting: 1) data faults, 2) hardware faults and 3) timing faults.”) Overall, one of ordinary skill in the art would have recognized that combining Cyriac with Abboush would greatly increase the fault injection capabilities of the system, enabling a considerably wider range of testing scenarios.
Claim 15. The elements of claim 15 are substantially the same as those of claim 1. Therefore, the elements of claim 15 are rejected due to the same reasons as outlined above for claim 1. Further, as to the elements present in claim 15 but not claim 1:
Cyriac teaches sending sensor data to the computing device. ([Page 62 Par 2- Page 65 Par 1] “The safety assessment approach uses three key components, namely 1) the urban driving simulator CARLA [64], 2) the imitation learning based convolutional neural network (IL-CNN) [65] as the autonomous driving agent (ADA) and 3) AVFI as the fault injection-based assessment for the ADA. The overall architecture of the simulation and fault injection framework of our approach is shown in Figure 10. … CARLA is an open source open world driving simulator which has an inbuilt library of urban roads, environments, pedestrians, vehicles and weather conditions (e.g., clear, rainy, sunset). It also provides the flexibility to users to create their own maps and assets so that they can recreate any scenario or environment they want to test. CARLA also provides an autonomous driving sensor suite which includes a variety of sensors like RGB cameras, LIDAR and GPS that can be ingested by ADAs for simulations. Figure 11 shows a screenshot of the CARLA simulator. CARLA operates by running two components. The server is responsible for generating and rendering the virtual environments while the client functions as an ADA, which receives sensor inputs from the server and sends control commands to the server (as shown in Figure 10). The server leverages Unreal Engine (popularly used in making video games) as the rendering and physics engine. The ADA under test uses [65] as the autonomous controller, which in turn employs an IL-CNN for the perception, planning and localization systems. In the test environment used for this experiment, the client is fed with RGB images from the forward facing camera sensor on the hood of the AV. The server sends the camera images together with other measurements of the car like speed and location to the client. The neural network model in [65] takes the sensor data from the server, perceives the image and produces a driving action for the AV. These driving decisions are then sent from the client to server, which services those commands and applies them to the AV’s actuators. This enables the AV to complete missions like navigating from one point to another in the simulated world.” [Page 65 Par 3] “AVFI injects data faults by modifying sensor measurements such as camera images, LIDAR and GPS data or world measurements like weather type and car speed captured by the AV system.”)
Claim 17. The elements of claim 17 are substantially the same as those of claim 1. Therefore, the elements of claim 17 are rejected due to the same reasons as outlined above for claim 1. Further, as to the elements present in claim 17 but not claim 1:
Abboush teaches A computing device, comprising a storage, and one or more processors communicatively connected with the storage; wherein the storage stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors to cause the one or more processors to implement a method comprising: ([Page 9 Par 3] “In HiL user environment, the tester can analyze recorded data and control HiL system in real time using HiL tools. Besides, the target system can be tested based on test cases with the list of sensors, actuator control signals and driving scenarios. HiL software tools can control HiL system from a computer machine. These tools are capable of configuring HiL system in a real-time environment. Generally, a computer and HiL system is connected with an Ethernet cable to each other for controlling and transmitting data.”)
Claim 2. Abboush teaches wherein the computing device is communicatively connected with a simulation device in which a virtual vehicle is simulated, and ([Figure 2] Shows the system setup and the fault injection framework connected to the HiL vehicle simulation system. Note that while the system is HiL, the vehicle is simulated and can be considered virtual. [Page 12 Par 5] “The behavior of the system under test SUT in the presence of faults can be accurately analyzed by considering vehicle dynamics and incorporating models of the environment. To be specific, the controller model is deployed and executed in the real ECU, and the dynamic models of the engine, transmission, environment and vehicle dynamics are executed on real-time processor hardware.”)
generating the target sensor data based on the injection mode of the fault type and ([Figure 4] Shows examples of healthy sensor data compared with data generated with an injected fault [Page 12 Par 5] “Real-time FI has been proposed by us to simulate various anomalies or faults in the overall system and to analyze their effects at the vehicle level, taking into account the real time constraints. In this way, not only sensor faults but also control and communication faults can be simulated. Besides, the robustness of the safety mechanism can also be evaluated in real time. The behavior of the system under test SUT in the presence of faults can be accurately analyzed by considering vehicle dynamics and incorporating models of the environment. To be specific, the controller model is deployed and executed in the real ECU, and the dynamic models of the engine, transmission, environment and vehicle dynamics are executed on real-time processor hardware.” [Page 13 Par 2] “To demonstrate the applicability of the proposed framework, the ASM gasoline engine model from dSPACE [65] is used as a case study. Figure 6 shows the system architecture of the gasoline engine designed in the MATLAB/Simulink environment. It consists of several systems and subsystems, i.e., SoftECU, gasoline engine, powertrain, vehicle dynamics and environment. In addition, there are I/O blocks for simulation and management of signals to the real hardware ECU. SoftECU provides the simulation of controller algorithms. Thus, the plant model can either contact the real target ECU (online mode) or receive controller commands from SoftECU at the simulation level (offline mode). The gasoline engine has been modeled in detail to include all physical characteristics of the engine. Air path system, fuel system, piston engine system, exhaust system and cooler system are the main parts.”)
PNG
media_image1.png
875
828
media_image1.png
Greyscale
Cyriac makes obvious teaches generating the target sensor data based on the fault occurrence probability comprises: sending an instruction to the simulation device so as to enable the simulation device to render sensor data based on the injection mode of the fault type and the fault occurrence probability from a perspective of the sensor in the simulation device; ([Figure 10] Shows a diagram of the system including the use of the CARLA vehicle simulation system [Page 63 Par 1 – Page 65 Par 1] “CARLA is an open source open world driving simulator which has an inbuilt library of urban roads, environments, pedestrians, vehicles and weather conditions (e.g., clear, rainy, sunset). It also provides the flexibility to users to create their own maps and assets so that they can recreate any scenario or environment they want to test. CARLA also provides an autonomous driving sensor suite which includes a variety of sensors like RGB cameras, LIDAR and GPS that can be ingested by ADAs for simulations. Figure 11 shows a screenshot of the CARLA simulator. CARLA operates by running two components. The server is responsible for generating and rendering the virtual environments while the client functions as an ADA, which receives sensor inputs from the server and sends control commands to the server (as shown in Figure 10). … The ADA under test uses [65] as the autonomous controller, which in turn employs an IL-CNN for the perception, planning and localization systems. In the test environment used for this experiment, the client is fed with RGB images from the forward facing camera sensor on the hood of the AV. The server sends the camera images together with other measurements of the car like speed and location to the client. The neural network model in [65] takes the sensor data from the server, perceives the image and produces a driving action for the AV. These driving decisions are then sent from the client to server, which services those commands and applies them to the AV’s actuators. This enables the AV to complete missions like navigating from one point to another in the simulated world.” [Page 62 Par 1] “This chapter presents AVFI, a fault injection based methodology for assessing the end-to-end resilience and safety of AV systems. AVFI assesses the robustness of AVs empirically by injecting faults into various components of the AV system which recreate situations that may be rarely tested. The tool uses a state-of-the-art AV simulation framework [64] as a platform to perform fault injections in sensor inputs (e.g., camera, LIDAR), in the neural networks that make driving decisions for the AV and in the hardware/software components.” [Page 65 Par 2 – Page 67 Par 1] “The fault injection campaign in AVFI is performed in two stages. First, the location of where the fault should be injected is decided (i.e., input, neural network, output). Then, a fault model for the injection is chosen. The various fault models implemented in AVFI are discussed below. There are three classes of faults that AVFI is capable of injecting: 1) data faults, 2) hardware faults and 3) timing faults. 5.2.1 Data Faults AVFI injects data faults by modifying sensor measurements such as camera images, LIDAR and GPS data or world measurements like weather type and car speed captured by the AV system. A data fault in sensor measurements can occur due to changes in the external environment, a malfunctioning sensor or unseen perturbations of the images (e.g., broken road sign posts). In the current implementation, AVFI intercepts the RGB camera sensor data from the server, modifies it to a sensor-specific fault model and finally sends it to IL-CNN ADA. The various fault models explored for the camera sensor are Gaussian noise, salt-and-pepper noise, solid occlusion, translucent occlusion and water droplets. Figure 12 shows the various fault models for the camera sensor. Gaussian noise in camera images occurs due to sensor limitations during image acquisition. There is inherent noise produced by the camera sensor in low-light situations or unfavorable temperatures of the sensor [66]. Furthermore, the electric circuits operating the sensor can also inject electronic circuit noise [67]. Salt-and-pepper noise is generated when there are malfunctioning sensor pixels, analog-to-digital converter errors [68] or bit errors in transmission [69]. Solid and translucent occlusions can occur when a physical object or liquid gets stuck on the lens of the camera. Finally the water droplet model aims to simulate the effect of rain drops on camera sensors. AVFI allows for the configuration of each type of fault model. For example, users can specify different noise parameters for the Gaussian and salt-and-pepper noise. For the occlusion fault models, the size and location are randomized every frame and maximum and minimum bounds can be specified. Furthermore, the probability that any given frame is injected with a fault can also be configured. Finally the water droplet model allows the user to specify the number, size and speed of water droplets dripping down in the frame.” [Figure 12] Shows examples of sensor data renderings affected by various injected faults)
PNG
media_image2.png
402
672
media_image2.png
Greyscale
PNG
media_image3.png
401
637
media_image3.png
Greyscale
Claim 18. The elements of claim 18 are substantially the same as those of claim 2. Therefore, the elements of claim 18 are rejected due to the same reasons as outlined above for claim 2.
Claim 3. Abboush teaches wherein the injection mode of the fault type comprises a mode of adjusting at least one of an internal parameter or an external parameter of the sensor in a simulation device. ([Figure 4] Shows examples of healthy sensor data compared with data generated with an injected fault [Page 10 Par 2-3] “The main objective of the FI GUI is to support the tester in the HiL user environment during run-time FI. From the GUI, a tester can perform various actions. Some of them are: selecting the fault type from the fault library and the fault values associated with the selected fault type. In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space. According to the literature, there are several types of potential faults in the time series data, such as gain, offset/bias, noise, hard-over, spike, stuck-at, packet loss, delay and drift faults [59–61]. By analyzing the signal features that characterize automotive fault behavior, the authors of [62] categorized the common faults in the vehicle’s PCM signals, such as abnormal magnitudes, rolling, noise and dependency faults. In addition, they also named some direct and indirect causes of the defined faults. Dirty or deteriorated sensors, faulty vehicle components, vibrations caused by dangerous internal motors, and sudden acceleration are good examples of direct sources that can cause these types of faults. Conversely, bumpy roads and uncertainties in driving behavior can cause anomalies in signal behavior without necessarily provoking faults. Equation (1) and Table 2 are used to summarize the mathematical representation of the aforementioned types: {EQUATION} where f(t) is faulty or manipulated signal value, dv represents the gain value and h(t) is the healthy or standard signal value. ov represents the offset/bias value.” [Table 3] Show the various sensor and control signal values effected by the injected fault)
Claim 8. Abboush teaches wherein the fault type of a fault comprises at least one of a data quality fault, a sensor location fault, a sensor hardware fault, or a data transmission fault; the data transmission fault comprising at least one of a data frame loss, a data frame drop, or a data frame error; and the fault parameter further comprises an occurrence duration of the fault. ([Page 10 Par 2-Page 12 Par 2] “The main objective of the FI GUI is to support the tester in the HiL user environment during run-time FI. From the GUI, a tester can perform various actions. Some of them are: selecting the fault type from the fault library and the fault values associated with the selected fault type. In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space. According to the literature, there are several types of potential faults in the time series data, such as gain, offset/bias, noise, hard-over, spike, stuck-at, packet loss, delay and drift faults [59–61]. By analyzing the signal features that characterize automotive fault behavior, the authors of [62] categorized the common faults in the vehicle’s PCM signals, such as abnormal magnitudes, rolling, noise and dependency faults. In addition, they also named some direct and indirect causes of the defined faults. Dirty or deteriorated sensors, faulty vehicle components, vibrations caused by dangerous internal motors, and sudden acceleration are good examples of direct sources that can cause these types of faults. Conversely, bumpy roads and uncertainties in driving behavior can cause anomalies in signal behavior without necessarily provoking faults. Equation (1) and Table 2 are used to summarize the mathematical representation of the aforementioned types: {EQUATION} where f(t) is faulty or manipulated signal value, dv represents the gain value and h(t) is the healthy or standard signal value. ov represents the offset/bias value. … The FI time at which the fault is to be injected can be defined on the basis of the driving cycle. The duration of the occurrence of the fault can be used to distinguish between permanent/persistent faults and transient faults [63]. Transient faults are short-term faults that occur for a certain duration and disappear again after a short time. In contrast, permanent faults occur over a longer period of time and remain until fault correction is accomplished. On the other hand, the fault location indicates the place where the fault is to be injected, e.g., the fault within an element, function or communication between components and the subsystem. The primary potential points of the fault in the vehicle have been explored in [64]. The authors classified the potential fault locations in an in-vehicle network into function specifications, network, sensors, actuators, control devices, gateways, a power supply, vehicle subsystems and a data acquisition system. In this study, the scope of fault location will cover the potential location of fault occurrence in the closed loop system, i.e., sensors, the control signal and the communication between them.”)
Claim 9. Cyriac teaches wherein the sensor comprises an image acquisition apparatus, and the fault type of the image acquisition apparatus comprises at least one of the followings: an image appearing a puzzle, an image appearing a stripe, an image appearing chromatic aberration, an image appearing a flare, an image appearing a ghost, image exposure abnormality, a lens attached by a foreign body, camera out of focus, camera looseness, or camera powered off. [Page 63 Par 1 – Page 65 Par 1] “CARLA is an open source open world driving simulator which has an inbuilt library of urban roads, environments, pedestrians, vehicles and weather conditions (e.g., clear, rainy, sunset). It also provides the flexibility to users to create their own maps and assets so that they can recreate any scenario or environment they want to test. CARLA also provides an autonomous driving sensor suite which includes a variety of sensors like RGB cameras, LIDAR and GPS that can be ingested by ADAs for simulations. Figure 11 shows a screenshot of the CARLA simulator. CARLA operates by running two components. The server is responsible for generating and rendering the virtual environments while the client functions as an ADA, which receives sensor inputs from the server and sends control commands to the server (as shown in Figure 10). … The ADA under test uses [65] as the autonomous controller, which in turn employs an IL-CNN for the perception, planning and localization systems. In the test environment used for this experiment, the client is fed with RGB images from the forward facing camera sensor on the hood of the AV. The server sends the camera images together with other measurements of the car like speed and location to the client. The neural network model in [65] takes the sensor data from the server, perceives the image and produces a driving action for the AV. These driving decisions are then sent from the client to server, which services those commands and applies them to the AV’s actuators. This enables the AV to complete missions like navigating from one point to another in the simulated world.” [Page 62 Par 1] “This chapter presents AVFI, a fault injection based methodology for assessing the end-to-end resilience and safety of AV systems. AVFI assesses the robustness of AVs empirically by injecting faults into various components of the AV system which recreate situations that may be rarely tested. The tool uses a state-of-the-art AV simulation framework [64] as a platform to perform fault injections in sensor inputs (e.g., camera, LIDAR), in the neural networks that make driving decisions for the AV and in the hardware/software components.” [Page 65 Par 2 – Page 67 Par 1] “The fault injection campaign in AVFI is performed in two stages. First, the location of where the fault should be injected is decided (i.e., input, neural network, output). Then, a fault model for the injection is chosen. The various fault models implemented in AVFI are discussed below. There are three classes of faults that AVFI is capable of injecting: 1) data faults, 2) hardware faults and 3) timing faults. 5.2.1 Data Faults AVFI injects data faults by modifying sensor measurements such as camera images, LIDAR and GPS data or world measurements like weather type and car speed captured by the AV system. A data fault in sensor measurements can occur due to changes in the external environment, a malfunctioning sensor or unseen perturbations of the images (e.g., broken road sign posts). In the current implementation, AVFI intercepts the RGB camera sensor data from the server, modifies it to a sensor-specific fault model and finally sends it to IL-CNN ADA. The various fault models explored for the camera sensor are Gaussian noise, salt-and-pepper noise, solid occlusion, translucent occlusion and water droplets. Figure 12 shows the various fault models for the camera sensor. Gaussian noise in camera images occurs due to sensor limitations during image acquisition. There is inherent noise produced by the camera sensor in low-light situations or unfavorable temperatures of the sensor [66]. Furthermore, the electric circuits operating the sensor can also inject electronic circuit noise [67]. Salt-and-pepper noise is generated when there are malfunctioning sensor pixels, analog-to-digital converter errors [68] or bit errors in transmission [69]. Solid and translucent occlusions can occur when a physical object or liquid gets stuck on the lens of the camera. Finally the water droplet model aims to simulate the effect of rain drops on camera sensors. AVFI allows for the configuration of each type of fault model. For example, users can specify different noise parameters for the Gaussian and salt-and-pepper noise. For the occlusion fault models, the size and location are randomized every frame and maximum and minimum bounds can be specified. Furthermore, the probability that any given frame is injected with a fault can also be configured. Finally the water droplet model allows the user to specify the number, size and speed of water droplets dripping down in the frame.” [Figure 12] Shows examples of sensor data renderings affected by various injected faults. The occlusion faults, particularly the translucent occlusion, is interpreted to read on the camera image “appearing a ghost.” Further, based on [Par 58] of the specification, the occlusion faults and water droplet fault are understood as reading on “a lens attached by a foreign body” which, based on the aforementioned paragraph, is interpreted as blocking or distorting part of the image to simulate an object being attached to the camera lens.)
Claim 13. Abboush teaches ([Page 10 Par 3] “In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space. According to the literature, there are several types of potential faults in the time series data, such as gain, offset/bias, noise, hard-over, spike, stuck-at, packet loss, delay and drift faults [59–61]” [Figure 4h] Shows an example of a packet loss fault [Figure 4i]) Shows an example of a delay fault”) ([Page 10 Par 3] “In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space. According to the literature, there are several types of potential faults in the time series data, such as gain, offset/bias, noise, hard-over, spike, stuck-at, packet loss, delay and drift faults [59–61]” [Figure 4h] Shows an example of a packet loss fault [Figure 4i]) Shows an example of a delay fault)
Cyriac makes obvious further comprising: transmitting the target sensor data to an autonomous driving software system, so as to enable the autonomous driving software system to generate a traveling strategy based on the target sensor data, and the virtual vehicle in the simulation device to perform fault simulation based on the traveling strategy, wherein the fault occurrence probability comprises a data fault probability, and transmitting the target sensor data to the autonomous driving software system comprises: transmitting, according to the data fault probability, the target sensor data to the autonomous driving software system. ([Page 62 Par 2- Page 67 Par 1] “The safety assessment approach uses three key components, namely 1) the urban driving simulator CARLA [64], 2) the imitation learning based convolutional neural network (IL-CNN) [65] as the autonomous driving agent (ADA) and 3) AVFI as the fault injection-based assessment for the ADA. The overall architecture of the simulation and fault injection framework of our approach is shown in Figure 10. … CARLA is an open source open world driving simulator which has an inbuilt library of urban roads, environments, pedestrians, vehicles and weather conditions (e.g., clear, rainy, sunset). It also provides the flexibility to users to create their own maps and assets so that they can recreate any scenario or environment they want to test. CARLA also provides an autonomous driving sensor suite which includes a variety of sensors like RGB cameras, LIDAR and GPS that can be ingested by ADAs for simulations. Figure 11 shows a screenshot of the CARLA simulator. CARLA operates by running two components. The server is responsible for generating and rendering the virtual environments while the client functions as an ADA, which receives sensor inputs from the server and sends control commands to the server (as shown in Figure 10). The server leverages Unreal Engine (popularly used in making video games) as the rendering and physics engine. The ADA under test uses [65] as the autonomous controller, which in turn employs an IL-CNN for the perception, planning and localization systems. In the test environment used for this experiment, the client is fed with RGB images from the forward facing camera sensor on the hood of the AV. The server sends the camera images together with other measurements of the car like speed and location to the client. The neural network model in [65] takes the sensor data from the server, perceives the image and produces a driving action for the AV. These driving decisions are then sent from the client to server, which services those commands and applies them to the AV’s actuators. This enables the AV to complete missions like navigating from one point to another in the simulated world. … AVFI injects data faults by modifying sensor measurements such as camera images, LIDAR and GPS data or world measurements like weather type and car speed captured by the AV system. A data fault in sensor measurements can occur due to changes in the external environment, a malfunctioning sensor or unseen perturbations of the images (e.g., broken road sign posts). In the current implementation, AVFI intercepts the RGB camera sensor data from the server, modifies it to a sensor-specific fault model and finally sends it to IL-CNN ADA. … AVFI allows for the configuration of each type of fault model. For example, users can specify different noise parameters for the Gaussian and salt-and-pepper noise. For the occlusion fault models, the size and location are randomized every frame and maximum and minimum bounds can be specified. Furthermore, the probability that any given frame is injected with a fault can also be configured. … AVFI injects hardware faults by injecting a single bit, multiple bits and stuck-at faults in the hardware components of autonomous systems such as processor sensors, software and communication sensors. … Hardware faults can be simulated as data faults and timing faults. … AVFI is capable of injecting timing faults into the communication paths in the AV network which cause either a delay in the flow of data or the loss of data. Figure 13 shows an example of a timing fault injection on the steering angle control command from the ADA to the AV actuator.” [Table 7] Shows available settings including input and output fault injection probability)
Claim 19. Abboush teaches wherein the injection mode of the fault type comprises at least one of: a mode of adjusting at least one of an internal parameter or an external parameter of the sensor in the simulation device; ([Figure 4] Shows examples of healthy sensor data compared with data generated with an injected fault [Page 10 Par 2-3] “The main objective of the FI GUI is to support the tester in the HiL user environment during run-time FI. From the GUI, a tester can perform various actions. Some of them are: selecting the fault type from the fault library and the fault values associated with the selected fault type. In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space. According to the literature, there are several types of potential faults in the time series data, such as gain, offset/bias, noise, hard-over, spike, stuck-at, packet loss, delay and drift faults [59–61]. By analyzing the signal features that characterize automotive fault behavior, the authors of [62] categorized the common faults in the vehicle’s PCM signals, such as abnormal magnitudes, rolling, noise and dependency faults. In addition, they also named some direct and indirect causes of the defined faults. Dirty or deteriorated sensors, faulty vehicle components, vibrations caused by dangerous internal motors, and sudden acceleration are good examples of direct sources that can cause these types of faults. Conversely, bumpy roads and uncertainties in driving behavior can cause anomalies in signal behavior without necessarily provoking faults. Equation (1) and Table 2 are used to summarize the mathematical representation of the aforementioned types: {EQUATION} where f(t) is faulty or manipulated signal value, dv represents the gain value and h(t) is the healthy or standard signal value. ov represents the offset/bias value.” [Table 3] Show the various sensor and control signal values effected by the injected fault) or a mode of processing road test data of the sensor acquired from a database, the mode of processing road test data comprises at least one of: a mode of processing image data, a mode of processing point cloud data, or a mode of processing positioning data. ([Examiner’s note: this limitation is written in the alternative, and therefore, for the purpose of this examination, the unmapped elements are not given patentable weight])
Claim 20. Abboush teaches A non-transitory computer-readable storage medium, comprising programs or instructions, wherein the programs or the instructions, when operated on a computer, implement a method according to claim 1. ([Figure 2] Shows the system setup and the fault injection framework connected to the HiL vehicle simulation system. [Page 9 Par 3] “In HiL user environment, the tester can analyze recorded data and control HiL system in real time using HiL tools. Besides, the target system can be tested based on test cases with the list of sensors, actuator control signals and driving scenarios. HiL software tools can control HiL system from a computer machine. These tools are capable of configuring HiL system in a real-time environment. Generally, a computer and HiL system is connected with an Ethernet cable to each other for controlling and transmitting data.” [Page 12 Par 5] “The behavior of the system under test SUT in the presence of faults can be accurately analyzed by considering vehicle dynamics and incorporating models of the environment. To be specific, the controller model is deployed and executed in the real ECU, and the dynamic models of the engine, transmission, environment and vehicle dynamics are executed on real-time processor hardware.”)
(2) Claims 4-5 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Hardware-in-the-Loop-Based Real-Time Fault Injection Framework for Dynamic Behavior Analysis of Automotive Software Systems (Hereinafter Abboush) in view of Assessment and Assurance of Autonomous Vehicle Safety (Hereinafter Cyriac) in further view of Experimental Resilience Assessment of An Open-Source Driving Agent (Hereinafter Rubaiyat)
Claim 4. Abboush teaches wherein the sensor is a virtual sensor in a simulation device, the virtual sensor has a same parameter as a physical sensor to represent the physical sensor, the computing device ([Page 12 Par 4] “Utilizing the real sensors connected to the HiL simulator or simulated sensor model, the real controller takes sensor data from various sources and sends the actuation command to the simulated actuator in the HiL simulator so that the action is performed according to the control logic.”)
The combination of Abboush and Cyriac does not explicitly teach the device is communicatively connected with a database storing road test data collected by the physical sensor during a road test.
Rubaiyat makes obvious the device is communicatively connected with a database storing road test data collected by the physical sensor during a road test. ([Page 55 Col 2 Par 4] “Vision Module: Our simulated vision module takes RGB road images (video frames) as input and generates the positions of the left and right lane markers on the road which is used by the openpilot LKAS system. The lane marker detection algorithm exploits histogram of oriented gradient (HOG) [14] and neural networks (NN) [15] to process the pre-collected images of the road. In our experiments, we used the video frames from the Caltech lanes dataset [16], but any other dataset is applicable here.” [Fig. 3] Shows a diagram of the system including an image library taken from real-world vehicles)
Rubaiyat is analogous art because it is within the field of autonomous vehicle fault injection. It would have been obvious to one of ordinary skill in the art to combine it with Abboush and Cyriac before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to further expand the system’s coverage of potential hazard and fault scenarios, home in on particularly dangerous scenarios, and reduce costs. Rubaiyat notes how current autonomous vehicle testing systems suffer from high costs to perform ([Page 54 Col 1 Par 3 – Col 2 Par 1] “Current practice in testing and safety validation of AVs involves simulation testing using realistic scenarios in virtual environments or real-world road testing. However, given the high costs of developing testbeds and running experiments, a question that arises is what constitutes an adequate testing experiment or how much testing is good enough? The safety critical faults causing hazards and incidents are rare and it might take forever for an AV to experience them during road test experiments. To ensure resilience against safety-critical faults, we need techniques for specifically testing the influence of such faults using fault injection.”) To this end, Rubaiyat presents a system for strategic fault injection that increases the coverage of hazard scenarios while simultaneously reducing the number of scenarios that must be run to effectively test the driving system ([Abstract] “growing concerns regarding the safety and reliability of AVs. This paper presents a Systems-Theoretic Process Analysis (STPA) based fault injection framework to assess the resilience of an open-source driving agent, called openpilot, under different environmental conditions and faults affecting sensor data. To increase the coverage of unsafe scenarios during testing, we use a strategic software fault-injection approach where the triggers for injecting the faults are derived from the unsafe scenarios identified during the high-level hazard analysis of the system. The experimental results show that the proposed strategic fault injection approach increases the hazard coverage compared to random fault injection and, thus, can help with more effective simulation of safety-critical faults and testing of AVs. In addition, the paper provides insights on the performance of openpilot safety mechanisms and its ability in timely detection and recovery from faulty inputs.” [Page 54 Col 2 Par 4 – Page 55 Col 1 Par 1] “To reduce the test space and increase the probability of generation of unsafe scenarios, we present a strategic software fault injection approach where the triggers for injecting the faults are driven by the high-level hazard analysis of the system. Specifically, we use the Systems-Theoretic Process Analysis (STPA) [10] hazard analysis technique to identify the potential unsafe scenarios in an AV with ACC and LKAS mechanisms. The identified unsafe scenarios are then translated into fault injection campaigns where the potentially unsafe system contexts (defined by the unique combinations of state variables) are used as conditions under which the faults are injected into the control software. We compare the performance of the STPA guided fault injection approach in covering hazardous scenarios to a random fault injection approach.”) Overall, one of ordinary skill in the art would have recognized that combining the strategic fault injection system of Rubaiyat with the systems of Abboush and Cyriac would result in a fault injection and testing system that less expensive, both monetarily and computationally, as well as providing greater coverage of hazardous situations.
Claim 5. Abboush teaches wherein the injection mode of the fault type ([Page 10 Par 2-3] “The main objective of the FI GUI is to support the tester in the HiL user environment during run-time FI. From the GUI, a tester can perform various actions. Some of them are: selecting the fault type from the fault library and the fault values associated with the selected fault type. In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space. According to the literature, there are several types of potential faults in the time series data, such as gain, offset/bias, noise, hard-over, spike, stuck-at, packet loss, delay and drift faults [59–61]. By analyzing the signal features that characterize automotive fault behavior, the authors of [62] categorized the common faults in the vehicle’s PCM signals, such as abnormal magnitudes, rolling, noise and dependency faults. In addition, they also named some direct and indirect causes of the defined faults. Dirty or deteriorated sensors, faulty vehicle components, vibrations caused by dangerous internal motors, and sudden acceleration are good examples of direct sources that can cause these types of faults. Conversely, bumpy roads and uncertainties in driving behavior can cause anomalies in signal behavior without necessarily provoking faults. Equation (1) and Table 2 are used to summarize the mathematical representation of the aforementioned types: {EQUATION} where f(t) is faulty or manipulated signal value, dv represents the gain value and h(t) is the healthy or standard signal value. ov represents the offset/bias value.” [Figure 4] Shows examples of healthy sensor data compared with data generated with an injected fault [Page 12 Par 5] “Real-time FI has been proposed by us to simulate various anomalies or faults in the overall system and to analyze their effects at the vehicle level, taking into account the real time constraints. In this way, not only sensor faults but also control and communication faults can be simulated. Besides, the robustness of the safety mechanism can also be evaluated in real time. The behavior of the system under test SUT in the presence of faults can be accurately analyzed by considering vehicle dynamics and incorporating models of the environment. To be specific, the controller model is deployed and executed in the real ECU, and the dynamic models of the engine, transmission, environment and vehicle dynamics are executed on real-time processor hardware.”) ([Page 10 Par 2-3] “The main objective of the FI GUI is to support the tester in the HiL user environment during run-time FI. From the GUI, a tester can perform various actions. Some of them are: selecting the fault type from the fault library and the fault values associated with the selected fault type. In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space. According to the literature, there are several types of potential faults in the time series data, such as gain, offset/bias, noise, hard-over, spike, stuck-at, packet loss, delay and drift faults [59–61]. By analyzing the signal features that characterize automotive fault behavior, the authors of [62] categorized the common faults in the vehicle’s PCM signals, such as abnormal magnitudes, rolling, noise and dependency faults. In addition, they also named some direct and indirect causes of the defined faults. Dirty or deteriorated sensors, faulty vehicle components, vibrations caused by dangerous internal motors, and sudden acceleration are good examples of direct sources that can cause these types of faults. Conversely, bumpy roads and uncertainties in driving behavior can cause anomalies in signal behavior without necessarily provoking faults. Equation (1) and Table 2 are used to summarize the mathematical representation of the aforementioned types: {EQUATION} where f(t) is faulty or manipulated signal value, dv represents the gain value and h(t) is the healthy or standard signal value. ov represents the offset/bias value.”)
Cyriac makes obvious ([Page 62 Par 1] “This chapter presents AVFI, a fault injection based methodology for assessing the end-to-end resilience and safety of AV systems. AVFI assesses the robustness of AVs empirically by injecting faults into various components of the AV system which recreate situations that may be rarely tested. The tool uses a state-of-the-art AV simulation framework [64] as a platform to perform fault injections in sensor inputs (e.g., camera, LIDAR), in the neural networks that make driving decisions for the AV and in the hardware/software components.” [Page 65 Par 2 – Page 67 Par 1] “The fault injection campaign in AVFI is performed in two stages. First, the location of where the fault should be injected is decided (i.e., input, neural network, output). Then, a fault model for the injection is chosen. The various fault models implemented in AVFI are discussed below. There are three classes of faults that AVFI is capable of injecting: 1) data faults, 2) hardware faults and 3) timing faults. 5.2.1 Data Faults AVFI injects data faults by modifying sensor measurements such as camera images, LIDAR and GPS data or world measurements like weather type and car speed captured by the AV system. A data fault in sensor measurements can occur due to changes in the external environment, a malfunctioning sensor or unseen perturbations of the images (e.g., broken road sign posts). In the current implementation, AVFI intercepts the RGB camera sensor data from the server, modifies it to a sensor-specific fault model and finally sends it to IL-CNN ADA. The various fault models explored for the camera sensor are Gaussian noise, salt-and-pepper noise, solid occlusion, translucent occlusion and water droplets. Figure 12 shows the various fault models for the camera sensor. Gaussian noise in camera images occurs due to sensor limitations during image acquisition. There is inherent noise produced by the camera sensor in low-light situations or unfavorable temperatures of the sensor [66]. Furthermore, the electric circuits operating the sensor can also inject electronic circuit noise [67]. Salt-and-pepper noise is generated when there are malfunctioning sensor pixels, analog-to-digital converter errors [68] or bit errors in transmission [69]. Solid and translucent occlusions can occur when a physical object or liquid gets stuck on the lens of the camera. Finally the water droplet model aims to simulate the effect of rain drops on camera sensors. AVFI allows for the configuration of each type of fault model. For example, users can specify different noise parameters for the Gaussian and salt-and-pepper noise. For the occlusion fault models, the size and location are randomized every frame and maximum and minimum bounds can be specified. Furthermore, the probability that any given frame is injected with a fault can also be configured. Finally the water droplet model allows the user to specify the number, size and speed of water droplets dripping down in the frame.”)
Rubaiyat makes obvious wherein the injection mode comprises a mode of processing the road test data, and generating sensor data comprises acquiring the road test data of the physical sensor from the database; and processing, according to the injection mode, the road test data to obtain the target sensor data. ([Page 55 Col 2 Par 4] “Vision Module: Our simulated vision module takes RGB road images (video frames) as input and generates the positions of the left and right lane markers on the road which is used by the openpilot LKAS system. The lane marker detection algorithm exploits histogram of oriented gradient (HOG) [14] and neural networks (NN) [15] to process the pre-collected images of the road. In our experiments, we used the video frames from the Caltech lanes dataset [16], but any other dataset is applicable here.” [Page 58 Col 1 Par 3 – Col 2 Par 6] “The fault models and the parameters used for injecting faults into different modules are described next. 1) Vision Module: In the vision module, faults are injected into the input frames as well as the outputs, including the left and right lane positions and the predicted path of the vehicle. By injecting faults to input images, we measure the resilience of the ML lane detection algorithm. By injecting faults to the outputs we evaluate the response of the LKAS system if the vision module produces erroneous outputs. To implement these faults, random values are added/subtracted to/from the path model variables. It is also possible that the camera becomes unavailable or the communication between vision data and control thread gets disconnected. To simulate these faults, the vision data is made unavailable after a certain time (after the trigger) of the simulation. Faults injected to the input images of the vision module simulate real-world environmental conditions such as rain, fog, snow, and occlusion created by mud/snow on camera. We also use other transformations like contrast change, brightness change, and adding blur to the images to evaluate the resilience of LKAS. More details on adding these effects to the input images are described below: Rain: To add the effect of rain, rain streaks from the database [21] are used. Rain streaks are added in random positions. For varying the density of the streaks, a ’thickness’ parameter is used. The value of ’thickness’ varies from 0 to 10, 0 means no rain and 10 indicates maximum density (10,000 streaks per image). The angle of the streaks can also be varied. To adjust with the background, the contrast of the rain streak is reduced and motion blur is added to the streaks. Besides adding the streaks, rain has also other effects (e.g., cloudy, gloomy) on the environment. To add those effects, we apply Gaussian blur and contrast reduction (varies with thickness) to the background of image. The values for the number of streaks, standard deviation (σ) of Gaussian blur, and contrast gain (α) for the background are set empirically. Fog: For creating the foggy effect, a uniformly distributed random noise is added to the images. Also a Gaussian blur is applied to the noise to add the blurriness effect. Density of the added noise varies with ’thickness’ parameter. To create the haziness, contrast of the background image is varied. Snow: Small blocks (size 2x2) of white pixels are added to the random locations of the image to simulate the effect of falling snow. Similar to rain and fog, number of snow varies with ’thickness’. Gaussian and motion blur are applied to the white blocks. Snow-fall also creates the haziness like the fog. So, the procedure of adding fog (described previously) is applied on the background images first, and then falling snows are added on top. Occlusion: In real environment, there are many factors that can cause occlusion. In this work, we consider the occlusions created by mud/snow stuck on the camera. To create the effects of occlusions, circular blobs (dark) are added in random locations of the image. Number of the blobs can be varied and the radius of the blob is randomly chosen from the range [5,50] pixels. To make the blobs blurry, Gaussian blur is applied. Other Effects: Besides the faults described above, we also apply some other transformations from [6] to the input images. These are changes in contrast, brightness, and blur (averaging, Gaussian, median). Table II summarizes the parameters used to inject the mentioned effects to the input images. The last column shows the faulty images along with the lanes detected by the lane marker detection algorithm.”)
PNG
media_image4.png
692
438
media_image4.png
Greyscale
Claim 11. Rubaiyat teaches wherein the mode of processing the road test data comprises at least one of: a mode of processing image data, ([Page 55 Col 2 Par 4] “Vision Module: Our simulated vision module takes RGB road images (video frames) as input and generates the positions of the left and right lane markers on the road which is used by the openpilot LKAS system. The lane marker detection algorithm exploits histogram of oriented gradient (HOG) [14] and neural networks (NN) [15] to process the pre-collected images of the road. In our experiments, we used the video frames from the Caltech lanes dataset [16], but any other dataset is applicable here.” [Page 58 Col 1 Par 3 – Col 2 Par 6] “The fault models and the parameters used for injecting faults into different modules are described next. 1) Vision Module: In the vision module, faults are injected into the input frames as well as the outputs, including the left and right lane positions and the predicted path of the vehicle. By injecting faults to input images, we measure the resilience of the ML lane detection algorithm. By injecting faults to the outputs we evaluate the response of the LKAS system if the vision module produces erroneous outputs. To implement these faults, random values are added/subtracted to/from the path model variables. It is also possible that the camera becomes unavailable or the communication between vision data and control thread gets disconnected. To simulate these faults, the vision data is made unavailable after a certain time (after the trigger) of the simulation. Faults injected to the input images of the vision module simulate real-world environmental conditions such as rain, fog, snow, and occlusion created by mud/snow on camera. We also use other transformations like contrast change, brightness change, and adding blur to the images to evaluate the resilience of LKAS. More details on adding these effects to the input images are described below: Rain: To add the effect of rain, rain streaks from the database [21] are used. Rain streaks are added in random positions. For varying the density of the streaks, a ’thickness’ parameter is used. The value of ’thickness’ varies from 0 to 10, 0 means no rain and 10 indicates maximum density (10,000 streaks per image). The angle of the streaks can also be varied. To adjust with the background, the contrast of the rain streak is reduced and motion blur is added to the streaks. Besides adding the streaks, rain has also other effects (e.g., cloudy, gloomy) on the environment. To add those effects, we apply Gaussian blur and contrast reduction (varies with thickness) to the background of image. The values for the number of streaks, standard deviation (σ) of Gaussian blur, and contrast gain (α) for the background are set empirically. Fog: For creating the foggy effect, a uniformly distributed random noise is added to the images. Also a Gaussian blur is applied to the noise to add the blurriness effect. Density of the added noise varies with ’thickness’ parameter. To create the haziness, contrast of the background image is varied. Snow: Small blocks (size 2x2) of white pixels are added to the random locations of the image to simulate the effect of falling snow. Similar to rain and fog, number of snow varies with ’thickness’. Gaussian and motion blur are applied to the white blocks. Snow-fall also creates the haziness like the fog. So, the procedure of adding fog (described previously) is applied on the background images first, and then falling snows are added on top. Occlusion: In real environment, there are many factors that can cause occlusion. In this work, we consider the occlusions created by mud/snow stuck on the camera. To create the effects of occlusions, circular blobs (dark) are added in random locations of the image. Number of the blobs can be varied and the radius of the blob is randomly chosen from the range [5,50] pixels. To make the blobs blurry, Gaussian blur is applied. Other Effects: Besides the faults described above, we also apply some other transformations from [6] to the input images. These are changes in contrast, brightness, and blur (averaging, Gaussian, median). Table II summarizes the parameters used to inject the mentioned effects to the input images. The last column shows the faulty images along with the lanes detected by the lane marker detection algorithm.”) a mode of processing point cloud data, or a mode of processing positioning data.
(3) Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Hardware-in-the-Loop-Based Real-Time Fault Injection Framework for Dynamic Behavior Analysis of Automotive Software Systems (Hereinafter Abboush) in view of Assessment and Assurance of Autonomous Vehicle Safety (Hereinafter Cyriac) in further view of Experimental Resilience Assessment of An Open-Source Driving Agent (Hereinafter Rubaiyat) in further view of Duenas (US 20230035780 A1)
Claim 6. Rubaiyat teaches wherein the database further stores information of a physical scene ([Page 55 Col 2 Par 4] “Vision Module: Our simulated vision module takes RGB road images (video frames) as input and generates the positions of the left and right lane markers on the road which is used by the openpilot LKAS system. The lane marker detection algorithm exploits histogram of oriented gradient (HOG) [14] and neural networks (NN) [15] to process the pre-collected images of the road. In our experiments, we used the video frames from the Caltech lanes dataset [16], but any other dataset is applicable here.” [Fig. 3] Shows a diagram of the system including an image library taken from real-world vehicles) ([Page 54 Col 2 Par 4 – Page 55 Col 1 Par 5] “To reduce the test space and increase the probability of generation of unsafe scenarios, we present a strategic software fault injection approach where the triggers for injecting the faults are driven by the high-level hazard analysis of the system. Specifically, we use the Systems-Theoretic Process Analysis (STPA) [10] hazard analysis technique to identify the potential unsafe scenarios in an AV with ACC and LKAS mechanisms. The identified unsafe scenarios are then translated into fault injection campaigns where the potentially unsafe system contexts (defined by the unique combinations of state variables) are used as conditions under which the faults are injected into the control software. We compare the performance of the STPA guided fault injection approach in covering hazardous scenarios to a random fault injection approach. … STPA based hazard analysis technique is applied to identify the unsafe control actions that may cause safety hazards in ACC and LKAS of AVs. … A strategic software fault injection framework based on STPA is developed where the locations and triggers for injecting the faults are driven by the unsafe control actions and critical system context identified during the hazard analysis process. We show that using this technique, the probability of generation of unsafe scenarios and hazard coverage is increased and, thus, the fault injection space is reduced.”)
The combination of Abboush, Cyriac, and Rubaiyat does not explicitly teach data corresponding to when a fault of the fault type occurs in the sensor during the road test, the method further comprising: calculating a similarity between a simulation scene in the simulation device and the physical scene; and, in response to the similarity being greater than or equal to a preset threshold, generating data.
Duenas makes obvious data corresponding to when a fault of the fault type occurs in the sensor during the road test, ([Par 16] “For example, historical log data associated with vehicles under control of one or more vehicle controllers may be accessed and analyzed to determine portions of the log data representing detections (e.g., landmarks) having a high variance. In some examples, this may be the result of an obstructed and/or obscured field-of-view associated with one or more sensors of the vehicle, a systematic fault associated with the vehicle control system, and/or an unaccounted change to the physical environment (e.g., a construction zone where streetlights and/or lane markings may be shifted, rotated, added, and/or removed).”) the method further comprising: calculating a similarity between a simulation scene in the simulation device and the physical scene; and, in response to the similarity being greater than or equal to a preset threshold, generating data. ([Par 46-48] “An example 142 illustrates how the evaluation computing system may compare an actual position 144 of the vehicle 106 at an actual point in time traversing the physical environment 108 according to the original trajectory 130 to a simulated position 146 of the simulated vehicle 132 at a simulated point in time corresponding to the actual point in time while the simulated vehicle 132 is navigating the simulated change 118. The evaluation computing system may determine a difference 148 of the simulated position 146 of the simulated vehicle 132 as compared to the actual position 144 of the vehicle 106 traversing the environment 108 without the simulated change 118 to evaluate how the simulated vehicle controller detects and/or mitigates the simulated change 118 between the original map data 116 and the simulated environment 122. … In some examples, the evaluation computing system may determine that the difference between the actual position and the simulated position satisfies the threshold distance (e.g., is within a threshold range of the threshold distance), then the evaluation computing system may determine that the vehicle controller has handled the simulated change in a sufficient manner, and may determine that the vehicle controller is valid. Alternatively, the evaluation computing system may determine that the difference between the actual position and the simulated position does not satisfy the threshold distance (e.g., is outside of a threshold range of the threshold distance), then the evaluation computing system may determine that the vehicle controller has not handled the simulated change in a sufficient manner, and may determine that the vehicle controller is invalid.”)
Duenas is analogous art because it is within the field of vehicle sensor fault processing. It would have been obvious to one of ordinary skill in the art to combine Duenas with Abboush, Cyriac, and Rubaiyat before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to enable the system to better handle real-world scenarios. Duenas notes how even with an abundance of simulated testing, the actual translation of better AV performance from the realm of simulation to actual road driving can be difficult to achieve ([Par 1] “Vehicles are increasingly supplementing or replacing manual functionality with automatic controls. Autonomous and/or semi-autonomous driving, however, can require use of computing systems capable of making split-second decisions to respond to myriad events and scenarios. The computing systems, such as the software and/or hardware associated therewith, may be updated, such as to optimize performance of vehicle and/or computing system operations. Prior to incorporating an update into a vehicle, the efficacy of the updated system may be confirmed via simulations, such as to evaluate a performance of the updated system as compared to a previous version. These simulations may test whether the updated system performs differently than the previous version. However, merely determining a difference in performance does not necessarily indicate whether the updated system is configured to detect and mitigate faults that may arise in the operation of the autonomous or semi-autonomous system.”) To this end, Duenas presents a method for better fault detection and handling for autonomous vehicles ([Par 28-30] “The techniques discussed herein may improve the functioning of an autonomous and/or semi-autonomous vehicle in many ways. For example, the techniques discussed herein may provide simulations for testing a performance of a component of a vehicle controller (e.g., one or more components of a localization system, including a semantic-based localization component), to ensure that the vehicle controller may safely operate in a dynamic environment. In some examples, the vehicle controller may be configured as a monitor operating on an autonomous vehicle, and following validation, the vehicle controller (or component thereof) may be enabled for control of the autonomous vehicle. Additionally, or alternatively, the techniques described herein may provide simulations for testing systematic errors occurring in simulations (e.g., a portion of the environment including landmarks representing a similar error) and evaluating a vehicle controllers ability to detect and/or mitigate the systematic error. As such, the techniques described herein may allow for evaluation of how a vehicle controller responds to unpredicted systematic errors. As may be understood, testing scenarios representing unpredicted systematic errors may improve safety outcomes, particularly in the context of vehicles and autonomous vehicles. Thus, the techniques described herein may improve a functioning of a computing device as well as improve safety outcomes. Additionally, once a fault, such as, for example, a systematic error, has been detected, the outputs from a component of the system, such as, the semantic based localization component of the vehicle controller, may be utilized to further tune and/or adjust the operation of additional components, such that the vehicle controller may take different reactions (e.g., more favorable reactions) in future scenarios to a similar fault. For example, based on the output, one or more additional outputs from one or more additional components may be ignored and/or prioritized over another, such that the system may determine a reaction, differing from the initial reaction when the fault was first encountered. Furthermore, the techniques described herein can provide a means by which the vehicle controller handles a simulated environment which deviates from a provided map of the environment (e.g., a construction zone altering landmarks). Testing of such scenarios allow the vehicle controller to be validated for handling such situations, resulting in increased accuracy while navigating construction zones, which are often unmarked and encountered without warning, resulting in an improvement to vehicle safety of autonomous vehicle control systems.”) Overall one of ordinary skill in the art would have recognized that combining with the extensive testing capabilities provided by Abboush, Cyriac, and Rubaiyat with the system of Duenas would result in vehicles that were better capable of detecting and handling a wider range of faults and ultimately were significantly safer.
(4) Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Hardware-in-the-Loop-Based Real-Time Fault Injection Framework for Dynamic Behavior Analysis of Automotive Software Systems (Hereinafter Abboush) in view of Assessment and Assurance of Autonomous Vehicle Safety (Hereinafter Cyriac) in further view of Experimental Resilience Assessment of An Open-Source Driving Agent (Hereinafter Rubaiyat) in further view of Farabet (US 11436484 B2)
Claim 7. Abboush teaches wherein the fault parameter([Page 10 Par 2-3] “From the GUI, a tester can perform various actions. Some of them are: selecting the fault type from the fault library and the fault values associated with the selected fault type. In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space.”) ([Page 10 Par 2-3] “The main objective of the FI GUI is to support the tester in the HiL user environment during run-time FI. From the GUI, a tester can perform various actions. Some of them are: selecting the fault type from the fault library and the fault values associated with the selected fault type. In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space. According to the literature, there are several types of potential faults in the time series data, such as gain, offset/bias, noise, hard-over, spike, stuck-at, packet loss, delay and drift faults [59–61]. By analyzing the signal features that characterize automotive fault behavior, the authors of [62] categorized the common faults in the vehicle’s PCM signals, such as abnormal magnitudes, rolling, noise and dependency faults. In addition, they also named some direct and indirect causes of the defined faults. Dirty or deteriorated sensors, faulty vehicle components, vibrations caused by dangerous internal motors, and sudden acceleration are good examples of direct sources that can cause these types of faults. Conversely, bumpy roads and uncertainties in driving behavior can cause anomalies in signal behavior without necessarily provoking faults. Equation (1) and Table 2 are used to summarize the mathematical representation of the aforementioned types: {EQUATION} where f(t) is faulty or manipulated signal value, dv represents the gain value and h(t) is the healthy or standard signal value. ov represents the offset/bias value.” [Figure 4] Shows examples of healthy sensor data compared with data generated with an injected fault [Page 12 Par 5] “Real-time FI has been proposed by us to simulate various anomalies or faults in the overall system and to analyze their effects at the vehicle level, taking into account the real time constraints. In this way, not only sensor faults but also control and communication faults can be simulated. Besides, the robustness of the safety mechanism can also be evaluated in real time. The behavior of the system under test SUT in the presence of faults can be accurately analyzed by considering vehicle dynamics and incorporating models of the environment. To be specific, the controller model is deployed and executed in the real ECU, and the dynamic models of the engine, transmission, environment and vehicle dynamics are executed on real-time processor hardware.”)
sending, ([Page 10 Par 2-3] “The main objective of the FI GUI is to support the tester in the HiL user environment during run-time FI. From the GUI, a tester can perform various actions. Some of them are: selecting the fault type from the fault library and the fault values associated with the selected fault type. In order to implement the FI method, three main configuration fault dimensions must be defined, i.e., fault type, fault location and injection time, which, together, form a fault space. According to the literature, there are several types of potential faults in the time series data, such as gain, offset/bias, noise, hard-over, spike, stuck-at, packet loss, delay and drift faults [59–61]. By analyzing the signal features that characterize automotive fault behavior, the authors of [62] categorized the common faults in the vehicle’s PCM signals, such as abnormal magnitudes, rolling, noise and dependency faults. In addition, they also named some direct and indirect causes of the defined faults. Dirty or deteriorated sensors, faulty vehicle components, vibrations caused by dangerous internal motors, and sudden acceleration are good examples of direct sources that can cause these types of faults. Conversely, bumpy roads and uncertainties in driving behavior can cause anomalies in signal behavior without necessarily provoking faults. Equation (1) and Table 2 are used to summarize the mathematical representation of the aforementioned types: {EQUATION} where f(t) is faulty or manipulated signal value, dv represents the gain value and h(t) is the healthy or standard signal value. ov represents the offset/bias value.” [Figure 4] Shows examples of healthy sensor data compared with data generated with an injected fault [Page 12 Par 5] “Real-time FI has been proposed by us to simulate various anomalies or faults in the overall system and to analyze their effects at the vehicle level, taking into account the real time constraints. In this way, not only sensor faults but also control and communication faults can be simulated. Besides, the robustness of the safety mechanism can also be evaluated in real time. The behavior of the system under test SUT in the presence of faults can be accurately analyzed by considering vehicle dynamics and incorporating models of the environment. To be specific, the controller model is deployed and executed in the real ECU, and the dynamic models of the engine, transmission, environment and vehicle dynamics are executed on real-time processor hardware.”)
Cyriac makes obvious
sending, ([Page 62 Par 1] “This chapter presents AVFI, a fault injection based methodology for assessing the end-to-end resilience and safety of AV systems. AVFI assesses the robustness of AVs empirically by injecting faults into various components of the AV system which recreate situations that may be rarely tested. The tool uses a state-of-the-art AV simulation framework [64] as a platform to perform fault injections in sensor inputs (e.g., camera, LIDAR), in the neural networks that make driving decisions for the AV and in the hardware/software components.” [Page 65 Par 2 – Page 67 Par 1] “The fault injection campaign in AVFI is performed in two stages. First, the location of where the fault should be injected is decided (i.e., input, neural network, output). Then, a fault model for the injection is chosen. The various fault models implemented in AVFI are discussed below. There are three classes of faults that AVFI is capable of injecting: 1) data faults, 2) hardware faults and 3) timing faults. 5.2.1 Data Faults AVFI injects data faults by modifying sensor measurements such as camera images, LIDAR and GPS data or world measurements like weather type and car speed captured by the AV system. A data fault in sensor measurements can occur due to changes in the external environment, a malfunctioning sensor or unseen perturbations of the images (e.g., broken road sign posts). In the current implementation, AVFI intercepts the RGB camera sensor data from the server, modifies it to a sensor-specific fault model and finally sends it to IL-CNN ADA. The various fault models explored for the camera sensor are Gaussian noise, salt-and-pepper noise, solid occlusion, translucent occlusion and water droplets. Figure 12 shows the various fault models for the camera sensor. Gaussian noise in camera images occurs due to sensor limitations during image acquisition. There is inherent noise produced by the camera sensor in low-light situations or unfavorable temperatures of the sensor [66]. Furthermore, the electric circuits operating the sensor can also inject electronic circuit noise [67]. Salt-and-pepper noise is generated when there are malfunctioning sensor pixels, analog-to-digital converter errors [68] or bit errors in transmission [69]. Solid and translucent occlusions can occur when a physical object or liquid gets stuck on the lens of the camera. Finally the water droplet model aims to simulate the effect of rain drops on camera sensors. AVFI allows for the configuration of each type of fault model. For example, users can specify different noise parameters for the Gaussian and salt-and-pepper noise. For the occlusion fault models, the size and location are randomized every frame and maximum and minimum bounds can be specified. Furthermore, the probability that any given frame is injected with a fault can also be configured. Finally the water droplet model allows the user to specify the number, size and speed of water droplets dripping down in the frame.”)
Rubaiyat makes obvious ([Page 55 Col 2 Par 4] “Vision Module: Our simulated vision module takes RGB road images (video frames) as input and generates the positions of the left and right lane markers on the road which is used by the openpilot LKAS system. The lane marker detection algorithm exploits histogram of oriented gradient (HOG) [14] and neural networks (NN) [15] to process the pre-collected images of the road. In our experiments, we used the video frames from the Caltech lanes dataset [16], but any other dataset is applicable here.” [Page 58 Col 1 Par 3 – Col 2 Par 6] “The fault models and the parameters used for injecting faults into different modules are described next. 1) Vision Module: In the vision module, faults are injected into the input frames as well as the outputs, including the left and right lane positions and the predicted path of the vehicle. By injecting faults to input images, we measure the resilience of the ML lane detection algorithm. By injecting faults to the outputs we evaluate the response of the LKAS system if the vision module produces erroneous outputs. To implement these faults, random values are added/subtracted to/from the path model variables. It is also possible that the camera becomes unavailable or the communication between vision data and control thread gets disconnected. To simulate these faults, the vision data is made unavailable after a certain time (after the trigger) of the simulation. Faults injected to the input images of the vision module simulate real-world environmental conditions such as rain, fog, snow, and occlusion created by mud/snow on camera. We also use other transformations like contrast change, brightness change, and adding blur to the images to evaluate the resilience of LKAS. More details on adding these effects to the input images are described below: Rain: To add the effect of rain, rain streaks from the database [21] are used. Rain streaks are added in random positions. For varying the density of the streaks, a ’thickness’ parameter is used. The value of ’thickness’ varies from 0 to 10, 0 means no rain and 10 indicates maximum density (10,000 streaks per image). The angle of the streaks can also be varied. To adjust with the background, the contrast of the rain streak is reduced and motion blur is added to the streaks. Besides adding the streaks, rain has also other effects (e.g., cloudy, gloomy) on the environment. To add those effects, we apply Gaussian blur and contrast reduction (varies with thickness) to the background of image. The values for the number of streaks, standard deviation (σ) of Gaussian blur, and contrast gain (α) for the background are set empirically. Fog: For creating the foggy effect, a uniformly distributed random noise is added to the images. Also a Gaussian blur is applied to the noise to add the blurriness effect. Density of the added noise varies with ’thickness’ parameter. To create the haziness, contrast of the background image is varied. Snow: Small blocks (size 2x2) of white pixels are added to the random locations of the image to simulate the effect of falling snow. Similar to rain and fog, number of snow varies with ’thickness’. Gaussian and motion blur are applied to the white blocks. Snow-fall also creates the haziness like the fog. So, the procedure of adding fog (described previously) is applied on the background images first, and then falling snows are added on top. Occlusion: In real environment, there are many factors that can cause occlusion. In this work, we consider the occlusions created by mud/snow stuck on the camera. To create the effects of occlusions, circular blobs (dark) are added in random locations of the image. Number of the blobs can be varied and the radius of the blob is randomly chosen from the range [5,50] pixels. To make the blobs blurry, Gaussian blur is applied. Other Effects: Besides the faults described above, we also apply some other transformations from [6] to the input images. These are changes in contrast, brightness, and blur (averaging, Gaussian, median). Table II summarizes the parameters used to inject the mentioned effects to the input images. The last column shows the faulty images along with the lanes detected by the lane marker detection algorithm.”)
The combination of Abboush, Cyriac, and Rubaiyat does not explicitly teach wherein the configuration comprises a data source type comprising at least one of a simulation device source or a database source, and generating the target sensor data comprises: sending, in response to the data source type being the simulation device source, simulation data; acquiring, in response to the data source types being the database source, road test data;
Farabet teaches wherein the configuration comprises a data source type comprising at least one of a simulation device source or a database source, and generating the target sensor data comprises: sending, in response to the data source type being the simulation device source, simulation data; acquiring, in response to the data source types being the database source, road test data; ([Col 6 line 11-18] “The data store(s) 120 may store sensor data and/or virtual sensor data generated by one or more real-world sensors of one or more vehicle(s) 102 and/or virtual sensors of one or more virtual vehicles, respectively. Data ingestion 122 may include generating and/or recording the data output by the vehicle(s) 102 autonomous vehicle platform (e.g., the vehicle hardware 104 and/or the software stack(s) 116).” [Col 11 line 23-35] “The simulated environment 410 may be generated using virtual data, real-world data, or a combination thereof. For example, the simulated environment may include real-world data augmented or changed using virtual data to generate combined data that may be used to simulate certain scenarios or situations with different and/or added elements (e.g., additional AI objects, environmental features, weather conditions, etc.). For example, pre-recorded video may be augmented or changed to include additional pedestrians, obstacles, and/or the like, such that the virtual objects (e.g., executing the software stack(s) 116 as HIL objects and/or SIL objects) may be tested against variations in the real-world data.”)
Farabet is analogous art because it is within the field of vehicle sensor simulation. It would have been obvious to one of ordinary skill in the art to combine Farabet with Abboush, Cyriac, and Rubaiyat before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to generate more accurate sensor data. Farabet notes how typical systems that generate sensor data for vehicle system testing and training struggle with accuracy, and the tradeoffs between more accurate real-world data and safer, easier to produce synthetic data ([Col 1 line 25-53] “Autonomous vehicles and semi-autonomous vehicles rely on machine learning, and specifically deep neural networks (DNNs), for performing any number of operations for operating, piloting, and navigating the vehicle. For example, DNNs may be used for object detection, lane and road boundary detection, safety analysis, drivable free-space analysis, control generation during vehicle maneuvers, and the like. However, in order to verify and validate the usage of the DNNs in autonomous or semi-autonomous vehicles, the DNNs need to be trained on a large amount of data, which requires vast amounts of time and effort, and yet still does not always guarantee universally accurate or usable results. For example, conventional systems often rely on data generated by physical vehicles navigating real-world environments to train the DNNs prior to deployment in working models. This approach has several limitations, however. For example, vehicles can only navigate to so many places, recreating dangerous or unique situations is difficult in the real-world environment, and testing the DNNs in these real-world environments may be dangerous. For example, especially where a DNN is used for obstacle avoidance or other safety measures, testing the DNNs in real-world environments may not be practical. On the other hand, an automaker likely will not deploy an autonomous vehicle into the real-world until an acceptable level of safety has been achieved. As a result, these competing interests make generating a sound, safe, accurate, and reliable autonomous driving system increasingly difficult.”) To this end, Farabet presents a vehicle testing system capable of utilizing synthetic data, real-word data, and combinations of both ([Col 1 line 63- Col 2 line 24] “In contrast to conventional systems, such as those described above, the systems of the present disclosure leverage a simulated environment to test one or more autonomous driving software stacks that include a multitude of DNNs. For example, physical sensor data, virtual sensor data, or a combination thereof may be used to train the DNNs of the software stack(s). Once trained, the DNNs may be tested, verified, and validated within a simulation system that generates a simulated environment for controlling a virtual object using the software stack(s). Simulation data from the simulated environment may be input into the DNNs of the software stack(s), and the DNNs may generate outputs. In some examples, the outputs may be used to control the virtual object within the simulated environment, to determine how the virtual object (and thus a physical object that corresponds to the virtual object) may perform in any number of different situations. In other examples, the outputs may be used to test the accuracy of the DNNs, and the results may be used to generate more data for further training (e.g., data that the DNNs are least consistent processing accurately), fine-tune the DNNs, verify the DNNs, and/or validate the DNNs. In any example, the simulated environment may be generated to create difficult to navigate, dangerous, unsafe, and/or otherwise unpredictable situations for the virtual object to navigate. As a result, previously untested scenarios (e.g., due to safety concerns, difficulty of reproduction, etc.) may be tested, repeated, and improved upon within the simulated environment.” [Col 13 line 51- 61] “As a result, the identical, or substantially identical, hardware components used by the vehicle 102 (e.g., a physical vehicle) to execute the autonomous driving software stack in real-world environments may be used to execute the autonomous driving software stack in the simulated environment 410. The use of the vehicle hardware 104 in the simulation system 400A thus provides for a more accurate simulation of how the vehicle 102 will perform in real-world situations, scenarios, and environments without having to actually find and test the vehicle 102 in the real-world.” [Col 6 line 11-18] “The data store(s) 120 may store sensor data and/or virtual sensor data generated by one or more real-world sensors of one or more vehicle(s) 102 and/or virtual sensors of one or more virtual vehicles, respectively. Data ingestion 122 may include generating and/or recording the data output by the vehicle(s) 102 autonomous vehicle platform (e.g., the vehicle hardware 104 and/or the software stack(s) 116).” [Col 11 line 23-35] “The simulated environment 410 may be generated using virtual data, real-world data, or a combination thereof. For example, the simulated environment may include real-world data augmented or changed using virtual data to generate combined data that may be used to simulate certain scenarios or situations with different and/or added elements (e.g., additional AI objects, environmental features, weather conditions, etc.). For example, pre-recorded video may be augmented or changed to include additional pedestrians, obstacles, and/or the like, such that the virtual objects (e.g., executing the software stack(s) 116 as HIL objects and/or SIL objects) may be tested against variations in the real-world data.”) Overall, one of ordinary skill in the art would have recognized that combining Farabet with Abboush, Cyriac, and Rubaiyat would result in a system that was capable of utilizing a greater diversity of data, enabling even more extensive testing and sensor data coverage.
(5) Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Hardware-in-the-Loop-Based Real-Time Fault Injection Framework for Dynamic Behavior Analysis of Automotive Software Systems (Hereinafter Abboush) in view of Assessment and Assurance of Autonomous Vehicle Safety (Hereinafter Cyriac) in further view of Chen (CN 114022743 A) as well as Jarquin (US 20220114805 A1)
Claim 10. Cyriac teaches wherein the sensor comprises a point cloud acquisition apparatus, and the fault type of the point cloud acquisition apparatus ([Page 65 Par 3] “AVFI injects data faults by modifying sensor measurements such as camera images, LIDAR and GPS data or world measurements like weather type and car speed captured by the AV system.”)
wherein the sensor comprises an integrated navigation device, and the fault type of the integrated navigation device ([Page 65 Par 3] “AVFI injects data faults by modifying sensor measurements such as camera images, LIDAR and GPS data or world measurements like weather type and car speed captured by the AV system.”)
The combination of Abboush and Cyriac does not explicitly teach wherein a point cloud fault comprises at least one of the followings: a point cloud having no discrimination, excessive noise points, a point cloud appearing a ghost, a point cloud appearing a mirror image, point cloud points loss, communication interruption, sensor looseness, or a sensor surface attached by a foreign body; wherein the navigation fault comprises at least one of the followings: orientation angle error, positioning accuracy deviation, communication interruption, or sensor looseness.
Chen makes obvious wherein the navigation fault comprises at least one of the followings: orientation angle error, positioning accuracy deviation, communication interruption, ([Page 5 Par 5] “step 1, receiving the manual selection of fault injection configuration parameter, configuration failure mode comprises failure mode high precision map data fault, GPS data fault, IMU data fault, CAN data fault, ADAS data fault and other sensor signal data fault.” [Page 5 Par 12] “step 1, receiving the manual selection of fault injection configuration parameter, configuration failure mode comprises failure mode high precision map data fault, GPS data fault, IMU data fault, CAN data fault, ADAS data fault and other sensor signal data fault.” [Page 6 Par 4 – Par 14] “As shown in FIG. 2 is a signal interrupt flow chart provided by the embodiment of the present invention, and FIG. 2 can be known, the signal interrupt process comprises: In step 201, the interrupt state of each signal is initialized to be non-interrupt. In step 202, the type of signal interrupt (GPS, IMU, CAN, ADAS and HDMAP) is selected according to the failure mode step 203, obtaining the current IMU time through the callback IMU time. step 204, setting the interrupt time according to the specified time point or time period, recording the current interrupt time of the IMU time, automatically calculating the end of the IMU time. step 205, real-time judging whether the current time is matched with the IMU time of automatically ending interruption, exceeding or just equivalent, if so, executing the step 206, otherwise, executing step 206'. step 206, invoking the fault generator, setting the corresponding signal state is non-interrupt, executing step 207. step 206', invoking the fault generator, setting the corresponding signal state is interrupt updating IMU time, executing step 207. step 207, invoking the fault injector, obtaining the corresponding signal interrupt state, selecting interrupt by the current received data to zero to cause signal interrupt; When the selection is not interrupted, the received data stream is normally executed. In one possible embodiment mode, failure mode is high precision map data fault, injecting fault data mode adopts signal interrupt mode; failure mode GPS data fault, IMU data fault, CAN data fault, ADAS data fault or other sensor signal data fault, injecting fault data by signal interrupt or injection generating error mode”)or sensor looseness.
Chen is analogous art because it is within the field of vehicle sensor fault injection. It would have been obvious to one of ordinary skill in the art to combine Chen with Abboush and Cyriac before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to more efficiently inject positioning, particularly GPS, faults. Chen notes that current fault injection systems, particularly when dealing with positioning sensors, struggle to efficiently simulate certain types of faults, such as full signal interruptions ([Page 2 Par 2-3] “the multi-sensor fusion data comprises GPS, IMU, CAN, ADAS, high precision map (HDMap) and so on, it is a fusion data produced offline by LCMLOG tool. on the basis of the multi-sensor fused vehicle positioning software, it is necessary to analyze the positioning confidence of the vehicle positioning software under different sensor signal loss or abnormal condition through a fault injection method, so as to quantitatively analyze the influence degree of different sensor signals to the vehicle positioning software, Using fault injection method can effectively find the abnormal condition of the vehicle body locating program, help improve the robustness of the vehicle body locating program and processing fault ability. As a result, the location confidence is an important indicator of whether the customer will collect the results of the location. In the actual positioning confidence analysis, the hardware fault of single-near collecting data is difficult to accurately create the condition of signal deletion or various different abnormal data of various different sensors, even if these conditions are created, the manpower needed to be paid, time and capital cost is very large.”) To this end, Chen presents a system for more effectively and efficiently injecting faults in positioning sensors ([Page 2 Par 4-6] “The invention aims at the technical problem in the existing technology, providing a fault injection method and system based on multi-sensor fusion data, performing fault injection in the process of accessing multi-sensor fusion data stream to obtain the sensing fault data, finding the abnormal condition of the vehicle body positioning track fused by the multi-sensor and analyzing the influence of the fault sensor data to the positioning confidence level; according to the data collected by each sensor effectively reproducing the vehicle body activity scene and track, and the data after multi sensor fusion can be used repeatedly; effectively reducing the manpower caused by manufacturing fault from hardware, time and capital cost; it provides powerful data support for the location abnormal analysis of the actual vehicle body; The operator can master the operation method under the guidance of brief specification, and the effect is obvious. According to a first aspect of the present invention, there is provided a fault injection method based on multi-sensor fusion data, comprising: step 1, receiving the manual selection of fault injection configuration parameter, the configuration parameter failure mode, the failure mode high precision map data fault, GPS data fault, IMU data fault, CAN data fault, ADAS data fault and other sensor signal data fault; step 2, in the process of playing back the multi-sensor fusion data, the multi-sensor fusion data is transmitted to the signal receiving processing module of the original data corresponding to each failure mode, generating fault data according to the failure mode, injecting the fault data to the corresponding signal receiving processing module.”) Overall, one of ordinary skill in the art would have recognized that combining Chen with Abboush and Cyriac would allow the system to more effectively and efficiently simulate faults in sensors for which particular fault types are difficult to produce otherwise, ultimately making the system more useful for a wider variety of sensor configurations and potential faults.
The combination of Abboush, Cyriac, and Chen does not explicitly teach wherein a point cloud fault comprises at least one of the followings: a point cloud having no discrimination, excessive noise points, a point cloud appearing a ghost, a point cloud appearing a mirror image, point cloud points loss, communication interruption, sensor looseness, or a sensor surface attached by a foreign body;
Jarquin makes obvious wherein a point cloud fault comprises at least one of the followings: a point cloud having no discrimination, excessive noise points, a point cloud appearing a ghost, a point cloud appearing a mirror image, point cloud points loss, ([Abstract] “These multimodal perception systems may augment raw data within a development framework to improve performance of object detection, classification, tracking, and sensor fusion under varying external conditions, such as adverse weather and light, as well as possible sensor errors or malfunctions like miss-calibration, noise, and dirty or faulty sensors. This augmentation may include injection of noise, occlusions, and misalignments from raw sensor data, and may include ground-truth labeling to match the augmented data. This augmentation provides improved robustness of the trained perception algorithms against calibration, noise, occlusion, and faults that may exist in real-world scenarios.”[Par 35] “FIG. 6 is a pictorial drawing illustrating a perception sensor obfuscation 600, according to an embodiment. In addition to input dataset cropping and resizing transformations matched with ground truth labels, perception sensor obfuscation 600 may be used to obfuscate the input signal by adding noise to an image or LiDAR point cloud (e.g., ranging dataset). FIG. 6 depicts a scene captured by a LiDAR point cloud and an image capture device, where the calibrated captured image and point cloud are extracted from a multimodal dataset and calibrated, overlapped, and output to provide an image of a common scene area. A LiDAR region occlusion is shown in point cloud removal output 605, which shows a removal of points from the source LiDAR point cloud in a LiDAR range scanning region. An image region occlusion is shown in image removal output 615, which shows a masking of pixels from the source image in one or more regions. A LiDAR noise occlusion is shown in point cloud noise output 610, which shows an injection of noise (e.g., Gaussian noise) into the source LiDAR point cloud in a LiDAR range scanning region. An image region occlusion is shown in image noise output 620, which shows an injection of noise into the source image in one or more regions.”) communication interruption, sensor looseness, or a sensor surface attached by a foreign body;
Jarquin is analogous art because it is within the field of vehicle sensor data augmentation. It would have been obvious to one of ordinary skill in the art to combine Jarquin with Abboush, Cyriac, and Chen before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to further expand upon the LIDAR-specific injection capabilities of the system. Jarquin notes the importance of thorough testing of vehicle sensors and associated perception systems, and how the use of such thorough testing improves the safety and reliability of vehicles ([Par 2] “Autonomous vehicles may be used to provide transportation without requiring full driver (e.g., operator) control. Fully autonomous vehicles may be used to navigate to a destination without any driver input while avoiding pedestrians, other vehicles, and other obstacles. Partially autonomous vehicles may receive a control input from a driver and may modify the vehicle control (e.g., steering, braking) to augment the navigation to a destination. These autonomous vehicles may identify and avoid obstacles using one or more input navigation sensors, such as an image capture device (e.g., camera), Light Detection and Ranging System (LiDAR), and RADAR. However, these navigation sensors are often subject to degraded performance under varying external conditions (e.g., adverse weather, varying light conditions) or under sensor errors or malfunctions (e.g., miss-calibration, noise, dirty sensors, faulty sensors).” [Par 15-17] “The automated driving perception systems described herein provide technical solutions for technical problems facing navigation sensors for autonomous vehicle navigation. These systems may be used to improve the performance of individual navigation sensors, and may be used to combine inputs from multiple navigation sensors to provide a multimodal perception system. In an example, image capture devices (e.g., cameras) may be used to generate an image dataset, and ranging devices (e.g., LiDAR, RADAR) may be used to generate a ranging dataset (e.g., point cloud dataset). These multimodal perception systems may make use of overlapping fields of view when placing sensors instrumentation to provide independent inputs in every surrounding region, which may be used to provide omnidirectional detection of vehicles and other objects around the vehicle. These multimodal perception systems may work to improve or guarantee complete environmental sensing and robust perception performance under varying external conditions, such as adverse weather and light, as well as possible sensor errors or malfunctions like miss-calibration, noise, and dirty or faulty sensors. This difference between performance under ideal conditions and performance under adverse weather or sensor conditions may be referred to as a domain gap. To improve perception system tasks of localization, object detection, object classification, and sensor fusion (e.g., combining various navigation sensors), the multimodal perception systems described herein are designed to identify and minimize or eliminate perception degradation due to these external conditions or sensor errors. In an example, these multimodal perception systems may improve performance of object detection, classification, tracking, and sensor fusion by augmenting raw data (e.g., datasets) within a development framework. This augmentation may include injection of noise, occlusions, and misalignments from raw sensor data, and may include ground-truth labeling to match the augmented data. This augmentation provides improved robustness of the trained perception algorithms against calibration, noise, occlusion, and faults that may exist in real-world scenarios. When using these multimodal perception systems within a development framework, designers of automated driving perception systems may programmatically inject sensor errors during a machine learning training phase. Training perception models based on the datasets with injected errors may be used to improve the ability of data-driven machine learning solutions to operate (e.g., correctly identify objects) in the presence of adverse conditions that are likely to occur.”) It should be further noted that while Cyriac mentions the system therein of being capable of LIDAR fault injection, what these faults consist of and how exactly these faults are produced is not discussed (see [Page 65 Par 3] of Cyriac.) To these ends, Jarquin presents a system that enables thorough fault injections in LIDAR systems including multiple types of faults ([Par 27] “Augmentation 400 may include an augmentor transformation. The augmentor transformation may produce an output object (e.g., output image) with augmented characteristics, such as resizing an image, reducing a field of view of a LiDAR point cloud, adding transformations to a transform tree, or adding noise to an image or LiDAR point cloud” [Par 35] “FIG. 6 is a pictorial drawing illustrating a perception sensor obfuscation 600, according to an embodiment. In addition to input dataset cropping and resizing transformations matched with ground truth labels, perception sensor obfuscation 600 may be used to obfuscate the input signal by adding noise to an image or LiDAR point cloud (e.g., ranging dataset). FIG. 6 depicts a scene captured by a LiDAR point cloud and an image capture device, where the calibrated captured image and point cloud are extracted from a multimodal dataset and calibrated, overlapped, and output to provide an image of a common scene area. A LiDAR region occlusion is shown in point cloud removal output 605, which shows a removal of points from the source LiDAR point cloud in a LiDAR range scanning region. An image region occlusion is shown in image removal output 615, which shows a masking of pixels from the source image in one or more regions. A LiDAR noise occlusion is shown in point cloud noise output 610, which shows an injection of noise (e.g., Gaussian noise) into the source LiDAR point cloud in a LiDAR range scanning region. An image region occlusion is shown in image noise output 620, which shows an injection of noise into the source image in one or more regions.”) Overall, one of ordinary skill in the art would have recognized that combining Jarquin with Abboush, Cyriac, and Chen would allow for deeper, more varied LIDAR fault injections, ultimately further expanding the variety of scenarios that can be simulated.
(6) Claims 12 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Hardware-in-the-Loop-Based Real-Time Fault Injection Framework for Dynamic Behavior Analysis of Automotive Software Systems (Hereinafter Abboush) in view of Assessment and Assurance of Autonomous Vehicle Safety (Hereinafter Cyriac) in further view of Deep Shutter Unrolling Network (Hereinafter Liu)
Claim 12. Abboush teaches further comprising: sending a sensor model of the sensor to the simulation device to enable the simulation device to determine, ([Page 12 Par 4] “Utilizing the real sensors connected to the HiL simulator or simulated sensor model, the real controller takes sensor data from various sources and sends the actuation command to the simulated actuator in the HiL simulator so that the action is performed according to the control logic.”) based on a vehicle motion model, ([Page 12 Par 5- Page 13 Par 1] “Real-time FI has been proposed by us to simulate various anomalies or faults in the overall system and to analyze their effects at the vehicle level, taking into account the real time constraints. In this way, not only sensor faults but also control and communication faults can be simulated. Besides, the robustness of the safety mechanism can also be evaluated in real time. The behavior of the system under test SUT in the presence of faults can be accurately analyzed by considering vehicle dynamics and incorporating models of the environment. To be specific, the controller model is deployed and executed in the real ECU, and the dynamic models of the engine, transmission, environment and vehicle dynamics are executed on real-time processor hardware. Moreover, by enabling and selecting different driving and fault scenarios, a comprehensive validation of the SUT with a controlled simulated system under abnormal conditions can be achieved.” [Page 13 Par 3] “To define driving maneuvers and set environmental conditions for the vehicle, the vehicle environment and driver model are used to complement the virtual powertrain. More specifically, both the powertrain and vehicle dynamics are modeled to provide longitudinal driving characteristics, including vehicle resistances, transmission model, vehicle resistances and driver characteristics.”)
([Page 15 Par 2] “To configure the driving scenario, the driving cycle is selected from the list provided by dSPACE in controlDesk. The driving mode is set to automatic with the predefined vehicle speed. Figure 9 shows the selected drive cycles in dSPACE ControlDesk. The red dot represents the starting point and the blue dot the current location. The cycles can be changed according to the tester’s requirements.”) ([Page 16 Par 1 – Page 17 Par 1] “This section discusses the results for employing the real-time FI framework proposed in this paper. The selected faults are injected based on the duration of their occurrence as permanent (a single and simultaneous/concurrent faults) and transient faults in different modes so that the effects of the faults on the system behavior can be analyzed in a real-time simulation. … A single fault denotes that only a single variable is manipulated and forced to its extreme value. For the purpose of observing the reaction of the system in the presence of faults in the sensors and control signals of the actuators, all of the faults defined in Table 5 are injected in the corresponding location at the specified time. To this end, an acceleration pedal position sensor is selected as a fault location. There are various types of faults that are injected on this sensor at different times of the driving scenario. The gain fault is injected on this sensor at 5 s with 𝑑𝑣 a value of 10 in Equation (1). The results are recorded at system level, where the engine RPM (Figure 10a) and vehicle speed (Figure 11a) are observed. Whenever the driver tries to maintain the speed of the vehicle by making a small fluctuation in the position of the acceleration pedal, this small fluctuation results in noise in the vehicle speed and engine RPM. This noise can be seen at a time of 135 s to 145 s, and many more similar situations can be observed in Figure 10a and Figure 11a. The noise fault is also injected at 5 s of the driving scenario, when the vehicle speed is 0 and the gear knob is in a neutral position. As soon as the fault is injected, the engine RPM values on the system level fluctuate between 2200 and 2500 rpm, whereas, in a normal case (fault-free), it fluctuates between 700 and 800 rpm. This results in a higher energy consumption. Moreover, when the driver tries to maintain the speed of the vehicle, some erratic noise is observed in the engine RPM. Figure 10c shows the effect of the noise fault on the engine RPM. On the other hand, there is a very small effect observed on the vehicle speed after the injection of the noise fault. The drift fault is injected at the starting point of the driving scenario. The effects of this fault is observed on the system level values of the engine RPM (Figure 10f) from 30 s, and when the gear knob of the engine is at a neutral position. As shown in Figure 10f, the engine RPM values slowly start to increase. Moreover, there is a slight increase in the vehicle speed also observed at 85 s to 90 s and between 130 s and 150 s, as shown in Figure 11d. The hard-over fault (Figure 10g) is injected at 36 s, and the effects of this fault are immediately observed on the system level values of the engine RPM. The values immediately jump from 700–800 rpm to 2200–2500 rpm. The system values of the hard-over fault are very similar to the noise and drift fault at a time of 90 s to 120 s; however, there is a small difference. The engine RPM values after the drift fault are between 2200 and 2400 rpm, whereas they reach up to 2500 rpm in the hard-over and noise faults. Moreover, there is more noise in the engine rpm values with the hard-over fault when compared to noise fault.” [Figure 10] Shows sensor data at various times)
Cyriac makes obvious determine poses of the virtual vehicle at different moments in an acquisition period and to render environmental data at the different moments based on the sensor model to obtain sensor data ([Figure 10] Shows a diagram of the system including the use of the CARLA vehicle simulation system [Page 63 Par 1 – Page 65 Par 1] “CARLA is an open source open world driving simulator which has an inbuilt library of urban roads, environments, pedestrians, vehicles and weather conditions (e.g., clear, rainy, sunset). … CARLA also provides an autonomous driving sensor suite which includes a variety of sensors like RGB cameras, LIDAR and GPS that can be ingested by ADAs for simulations. Figure 11 shows a screenshot of the CARLA simulator. CARLA operates by running two components. The server is responsible for generating and rendering the virtual environments while the client functions as an ADA, which receives sensor inputs from the server and sends control commands to the server (as shown in Figure 10). The server leverages Unreal Engine (popularly used in making video games) as the rendering and physics engine. The ADA under test uses [65] as the autonomous controller, which in turn employs an IL-CNN for the perception, planning and localization systems. In the test environment used for this experiment, the client is fed with RGB images from the forward facing camera sensor on the hood of the AV. The server sends the camera images together with other measurements of the car like speed and location to the client. The neural network model in [65] takes the sensor data from the server, perceives the image and produces a driving action for the AV. These driving decisions are then sent from the client to server, which services those commands and applies them to the AV’s actuators. This enables the AV to complete missions like navigating from one point to another in the simulated world.” [Figure 13] Shows examples of sensor data over time [Figure 12] Shows various sensor outputs affected by faults from the perspective of the virtual hood-mounted camera. [Page 68 Par 2 – Page 69 Par 1] “To assess the impact on fault injection on the end-to-end AV system, AVFI reports the three metrics. The mission success rate (MSR) is the percentage of times that the ADA was able to complete a navigation mission in a predetermined constant amount of time. Higher MSR values indicate higher resiliency. Traffic violations per kilometer (VPK) is the number of traffic violations per kilometer driven in a fault injection experiment. Traffic violations encompass lane violations, driving on curbs, and collision with other pedestrians, cars or property. Finally, accidents per kilometer (APK) is the number of accidents, which include any type of collision, per kilometer driving in a fault injection experiment. The experiment and fault model setup can be seen in Table 7.”)
PNG
media_image5.png
442
654
media_image5.png
Greyscale
The combination of Abboush and Cyriac does not explicitly teach determine data at different moments in an acquisition period of a frame to obtain sensor data of the frame
Liu makes obvious determine data at different moments in an acquisition period of a frame to obtain sensor data of the frame ([Page 5945 Col 2 Par 4] “Thus, we propose two datasets: the Carla-RS dataset and the Fastec-RS dataset. Our datasets are synthesized via high speed cameras and simulate the real image formation process. The Carla-RS dataset is generated from a virtual 3D environment provided by the Carla simulator [5]. Carla simulator is an open-source platform for autonomous driving research and it provides seven photorealistic 3D virtual towns. We implement a rolling shutter camera model since the original simulator does not support it. We also relax the constraint that the camera is mounted on a ground vehicle, such that we can freely move our rolling shutter camera in six DoF. 250 sequences are randomly sampled and each sequence has 10 consecutive frames. Both a constant translational velocity model and a constant angular rate model are used for the sequence generation, which is typically hold in real scenarios due to the short time interval (i.e., < 50ms) between two consecutive frames. In total, we generate 2500 rolling shutter images at a resolution of 640 × 448 pixels.” [Examiner’s note: rolling shutter distortion is the result of image data from different positions being combined into a single frame due to high-speed movement])
Liu is analogous art because it is within the field of vehicle sensor systems. It would have been obvious to one of ordinary skill in the art to combine Liu with Abboush and Cyriac before the effective filing date. It would have been obvious to one of ordinary skill in the art to make this combination to better simulate issues that can affect physical cameras, such as the distortion caused by rolling shutter cameras. Liu notes how, despite the ubiquity of rolling shutter cameras and the potential of errors caused by such, effective ways of simulating these effects have not been explored publicly and the methods for correcting such distortions have proved cumbersome ([Page 5941 Col 1 Par 1 -5942 Col 1 Par 3] “CMOS imaging sensors are widely used in many consumer products. Most CMOS sensors capture images with a rolling shutter mechanism. In contrast to a global shutter camera, which captures all pixels at the same time, a rolling shutter camera sequentially captures the image pixels row by row. Therefore, different types of distortions, e.g. skew, smear or wobble, will appear if the camera is moving during the image capture. It is well known that many vision tasks (e.g. structure from motion, visual odometry, pose estimation or depth prediction) suffer from rolling shutter distortions [1, 11, 15, 16, 26,27]. The rolling shutter effect correction problem has thus received considerable attention in the past [24,30,32]. Existing works on rolling shutter effect correction can be categorized into classical approaches and single image based deep learning approaches. … Single image based rolling shutter effect correction is an ill-posed problem and relies heavily on prior assumptions (e.g. straight lines must remain straight), either formulated explicitly or learned implicitly by a deep network, which limit their applicability to real scenarios. Classical multi-image based approaches are more general and instead rely on geometric constraints from multiple views to perform the rectification. However, they usually formulate it as a computationally ex pensive optimization problem for 6 DoF camera motions, which prevents the algorithm from being used in time con strained applications. … Since there is no public dataset available, we also pro pose two novel datasets: the Fastec-RS dataset and the Carla-RS dataset, as our second contribution. The Fastec RS dataset has 2584 image pairs. It is generated via a professional high-speed camera (with a framerate of 2400 FPS) and captured in real environments. Since the camera is mounted on a ground vehicle which undergoes limited motion, we also create the Carla-RS dataset with general six degree of freedom (DoF) motions. This dataset is generated from a virtual 3D environment and has 2500 image pairs. To further research in this area we make both our code and the datasets public”) To this end, Liu presents a system for simulating rolling shutter distortion within the CARLA vehicle simulator as a part of the design of its deep learning-based rolling shutter correction system ([Page 5945 Col 2 Par 4] “Thus, we propose two datasets: the Carla-RS dataset and the Fastec-RS dataset. Our datasets are synthesized via high speed cameras and simulate the real image formation process. The Carla-RS dataset is generated from a virtual 3D environment provided by the Carla simulator [5]. Carla simulator is an open-source platform for autonomous driving research and it provides seven photorealistic 3D virtual towns. We implement a rolling shutter camera model since the original simulator does not support it. We also relax the constraint that the camera is mounted on a ground vehicle, such that we can freely move our rolling shutter camera in six DoF. 250 sequences are randomly sampled and each sequence has 10 consecutive frames. Both a constant translational velocity model and a constant angular rate model are used for the sequence generation, which is typically hold in real scenarios due to the short time interval (i.e., < 50ms) between two consecutive frames. In total, we generate 2500 rolling shutter images at a resolution of 640 × 448 pixels.” [Page 5941 Col 2 Par 2 – Page 5942 Col 1 Par 1] “…we propose an efficient end-to-end deep neural network for rolling shutter effect correction. Our method solves a generic rectification problem from two consecutive frames. It is able to take advantage of the parallel computational power of a graphic card and runs in near real time. Furthermore, benefiting from the representational power of a deep network, our network is also able to learn good image priors to further boost the quality of the rectified image. Different from the above image-to-image translation problems, in which the estimations usually rely on its local neighborhood pixels of the input image, the rolling shutter effect correction problem is more challenging. A pixel of the rectified image might lie far away from its corresponding pixel of the input rolling shutter image, depending on the types of motion, 3D scene structure as well as its capturing time. To resolve these challenges, we propose a novel network architecture for rolling shutter image correction.”) Overall, one of ordinary skill in the art would have recognized that combining the rolling-shutter simulation system of Liu with Abboush and Cyriac would enable more realistic simulation of the output of vehicle sensors and therefore an overall more useful and accurate testing system.
Claim 16. Abboush teaches wherein a virtual vehicle is simulated in the simulation device, the method further comprising: ([Figure 2] Shows the system setup and the fault injection framework connected to the HiL vehicle simulation system. Note that while the system is HiL, the vehicle is simulated and can be considered virtual. [Page 12 Par 5] “The behavior of the system under test SUT in the presence of faults can be accurately analyzed by considering vehicle dynamics and incorporating models of the environment. To be specific, the controller model is deployed and executed in the real ECU, and the dynamic models of the engine, transmission, environment and vehicle dynamics are executed on real-time processor hardware.”) receiving a sensor model of the sensor, ([Page 12 Par 4] “Utilizing the real sensors connected to the HiL simulator or simulated sensor model, the real controller takes sensor data from various sources and sends the actuation command to the simulated actuator in the HiL simulator so that the action is performed according to the control logic.”) ([Figure 4] Shows examples of healthy sensor data compared with data generated with an injected fault [Page 12 Par 5] “Real-time FI has been proposed by us to simulate various anomalies or faults in the overall system and to analyze their effects at the vehicle level, taking into account the real time constraints. In this way, not only sensor faults but also control and communication faults can be simulated. Besides, the robustness of the safety mechanism can also be evaluated in real time. The behavior of the system under test SUT in the presence of faults can be accurately analyzed by considering vehicle dynamics and incorporating models of the environment. To be specific, the controller model is deployed and executed in the real ECU, and the dynamic models of the engine, transmission, environment and vehicle dynamics are executed on real-time processor hardware.”) ([Page 15 Par 2] “To configure the driving scenario, the driving cycle is selected from the list provided by dSPACE in controlDesk. The driving mode is set to automatic with the predefined vehicle speed. Figure 9 shows the selected drive cycles in dSPACE ControlDesk. The red dot represents the starting point and the blue dot the current location. The cycles can be changed according to the tester’s requirements.”) based on the sensor model, to obtain the sensor data ([Page 16 Par 1 – Page 17 Par 1] “This section discusses the results for employing the real-time FI framework proposed in this paper. The selected faults are injected based on the duration of their occurrence as permanent (a single and simultaneous/concurrent faults) and transient faults in different modes so that the effects of the faults on the system behavior can be analyzed in a real-time simulation. … A single fault denotes that only a single variable is manipulated and forced to its extreme value. For the purpose of observing the reaction of the system in the presence of faults in the sensors and control signals of the actuators, all of the faults defined in Table 5 are injected in the corresponding location at the specified time. To this end, an acceleration pedal position sensor is selected as a fault location. There are various types of faults that are injected on this sensor at different times of the driving scenario. The gain fault is injected on this sensor at 5 s with 𝑑𝑣 a value of 10 in Equation (1). The results are recorded at system level, where the engine RPM (Figure 10a) and vehicle speed (Figure 11a) are observed. Whenever the driver tries to maintain the speed of the vehicle by making a small fluctuation in the position of the acceleration pedal, this small fluctuation results in noise in the vehicle speed and engine RPM. This noise can be seen at a time of 135 s to 145 s, and many more similar situations can be observed in Figure 10a and Figure 11a. The noise fault is also injected at 5 s of the driving scenario, when the vehicle speed is 0 and the gear knob is in a neutral position. As soon as the fault is injected, the engine RPM values on the system level fluctuate between 2200 and 2500 rpm, whereas, in a normal case (fault-free), it fluctuates between 700 and 800 rpm. This results in a higher energy consumption. Moreover, when the driver tries to maintain the speed of the vehicle, some erratic noise is observed in the engine RPM. Figure 10c shows the effect of the noise fault on the engine RPM. On the other hand, there is a very small effect observed on the vehicle speed after the injection of the noise fault. The drift fault is injected at the starting point of the driving scenario. The effects of this fault is observed on the system level values of the engine RPM (Figure 10f) from 30 s, and when the gear knob of the engine is at a neutral position. As shown in Figure 10f, the engine RPM values slowly start to increase. Moreover, there is a slight increase in the vehicle speed also observed at 85 s to 90 s and between 130 s and 150 s, as shown in Figure 11d. The hard-over fault (Figure 10g) is injected at 36 s, and the effects of this fault are immediately observed on the system level values of the engine RPM. The values immediately jump from 700–800 rpm to 2200–2500 rpm. The system values of the hard-over fault are very similar to the noise and drift fault at a time of 90 s to 120 s; however, there is a small difference. The engine RPM values after the drift fault are between 2200 and 2400 rpm, whereas they reach up to 2500 rpm in the hard-over and noise faults. Moreover, there is more noise in the engine rpm values with the hard-over fault when compared to noise fault.” [Figure 10] Shows sensor data at various times)
Cyriac makes obvious the sensor model comprising a rendering scheme of the sensor data and an initial pose of the sensor relative to the virtual vehicle; wherein rendering the sensor data according to the fault occurrence probability comprises: predicting poses of the virtual vehicle at different moments in an acquisition period ([Figure 10] Shows a diagram of the system including the use of the CARLA vehicle simulation system [Page 63 Par 1 – Page 65 Par 1] “CARLA is an open source open world driving simulator which has an inbuilt library of urban roads, environments, pedestrians, vehicles and weather conditions (e.g., clear, rainy, sunset). … CARLA also provides an autonomous driving sensor suite which includes a variety of sensors like RGB cameras, LIDAR and GPS that can be ingested by ADAs for simulations. Figure 11 shows a screenshot of the CARLA simulator. CARLA operates by running two components. The server is responsible for generating and rendering the virtual environments while the client functions as an ADA, which receives sensor inputs from the server and sends control commands to the server (as shown in Figure 10). The server leverages Unreal Engine (popularly used in making video games) as the rendering and physics engine. The ADA under test uses [65] as the autonomous controller, which in turn employs an IL-CNN for the perception, planning and localization systems. In the test environment used for this experiment, the client is fed with RGB images from the forward facing camera sensor on the hood of the AV. The server sends the camera images together with other measurements of the car like speed and location to the client. The neural network model in [65] takes the sensor data from the server, perceives the image and produces a driving action for the AV. These driving decisions are then sent from the client to server, which services those commands and applies them to the AV’s actuators. This enables the AV to complete missions like navigating from one point to another in the simulated world.” [Figure 13] Shows examples of sensor data over time [Figure 12] Shows various sensor outputs affected by faults from the perspective of the virtual hood-mounted camera. [Page 68 Par 2 – Page 69 Par 1] “To assess the impact on fault injection on the end-to-end AV system, AVFI reports the three metrics. The mission success rate (MSR) is the percentage of times that the ADA was able to complete a navigation mission in a predetermined constant amount of time. Higher MSR values indicate higher resiliency. Traffic violations per kilometer (VPK) is the number of traffic violations per kilometer driven in a fault injection experiment. Traffic violations encompass lane violations, driving on curbs, and collision with other pedestrians, cars or property. Finally, accidents per kilometer (APK) is the number of accidents, which include any type of collision, per kilometer driving in a fault injection experiment. The experiment and fault model setup can be seen in Table 7.”)
The combination of Abboush and Cyriac does not explicitly teach determine data at different moments in an acquisition period of a frame to obtain sensor data of the frame
Liu makes obvious determine data at different moments in an acquisition period of a frame to obtain sensor data of the frame ([Page 5945 Col 2 Par 4] “Thus, we propose two datasets: the Carla-RS dataset and the Fastec-RS dataset. Our datasets are synthesized via high speed cameras and simulate the real image formation process. The Carla-RS dataset is generated from a virtual 3D environment provided by the Carla simulator [5]. Carla simulator is an open-source platform for autonomous driving research and it provides seven photorealistic 3D virtual towns. We implement a rolling shutter camera model since the original simulator does not support it. We also relax the constraint that the camera is mounted on a ground vehicle, such that we can freely move our rolling shutter camera in six DoF. 250 sequences are randomly sampled and each sequence has 10 consecutive frames. Both a constant translational velocity model and a constant angular rate model are used for the sequence generation, which is typically hold in real scenarios due to the short time interval (i.e., < 50ms) between two consecutive frames. In total, we generate 2500 rolling shutter images at a resolution of 640 × 448 pixels.” [Examiner’s note: rolling shutter distortion is the result of image data from different positions being combined into a single frame due to high-speed movement])
Liu is analogous art because it is within the field of vehicle sensor systems. It would have been obvious to one of ordinary skill in the art to combine Liu with Abboush and Cyriac before the effective filing date. It would have been obvious to one of ordinary skill in the art to make this combination to better simulate issues that can affect physical cameras, such as the distortion caused by rolling shutter cameras. Liu notes how, despite the ubiquity of rolling shutter cameras and the potential of errors caused by such, effective ways of simulating these effects have not been explored publicly and the methods for correcting such distortions have proved cumbersome ([Page 5941 Col 1 Par 1 -5942 Col 1 Par 3] “CMOS imaging sensors are widely used in many consumer products. Most CMOS sensors capture images with a rolling shutter mechanism. In contrast to a global shutter camera, which captures all pixels at the same time, a rolling shutter camera sequentially captures the image pixels row by row. Therefore, different types of distortions, e.g. skew, smear or wobble, will appear if the camera is moving during the image capture. It is well known that many vision tasks (e.g. structure from motion, visual odometry, pose estimation or depth prediction) suffer from rolling shutter distortions [1, 11, 15, 16, 26,27]. The rolling shutter effect correction problem has thus received considerable attention in the past [24,30,32]. Existing works on rolling shutter effect correction can be categorized into classical approaches and single image based deep learning approaches. … Single image based rolling shutter effect correction is an ill-posed problem and relies heavily on prior assumptions (e.g. straight lines must remain straight), either formulated explicitly or learned implicitly by a deep network, which limit their applicability to real scenarios. Classical multi-image based approaches are more general and instead rely on geometric constraints from multiple views to perform the rectification. However, they usually formulate it as a computationally ex pensive optimization problem for 6 DoF camera motions, which prevents the algorithm from being used in time con strained applications. … Since there is no public dataset available, we also pro pose two novel datasets: the Fastec-RS dataset and the Carla-RS dataset, as our second contribution. The Fastec RS dataset has 2584 image pairs. It is generated via a professional high-speed camera (with a framerate of 2400 FPS) and captured in real environments. Since the camera is mounted on a ground vehicle which undergoes limited motion, we also create the Carla-RS dataset with general six degree of freedom (DoF) motions. This dataset is generated from a virtual 3D environment and has 2500 image pairs. To further research in this area we make both our code and the datasets public”) To this end, Liu presents a system for simulating rolling shutter distortion within the CARLA vehicle simulator as a part of the design of its deep learning-based rolling shutter correction system ([Page 5945 Col 2 Par 4] “Thus, we propose two datasets: the Carla-RS dataset and the Fastec-RS dataset. Our datasets are synthesized via high speed cameras and simulate the real image formation process. The Carla-RS dataset is generated from a virtual 3D environment provided by the Carla simulator [5]. Carla simulator is an open-source platform for autonomous driving research and it provides seven photorealistic 3D virtual towns. We implement a rolling shutter camera model since the original simulator does not support it. We also relax the constraint that the camera is mounted on a ground vehicle, such that we can freely move our rolling shutter camera in six DoF. 250 sequences are randomly sampled and each sequence has 10 consecutive frames. Both a constant translational velocity model and a constant angular rate model are used for the sequence generation, which is typically hold in real scenarios due to the short time interval (i.e., < 50ms) between two consecutive frames. In total, we generate 2500 rolling shutter images at a resolution of 640 × 448 pixels.” [Page 5941 Col 2 Par 2 – Page 5942 Col 1 Par 1] “…we propose an efficient end-to-end deep neural network for rolling shutter effect correction. Our method solves a generic rectification problem from two consecutive frames. It is able to take advantage of the parallel computational power of a graphic card and runs in near real time. Furthermore, benefiting from the representational power of a deep network, our network is also able to learn good image priors to further boost the quality of the rectified image. Different from the above image-to-image translation problems, in which the estimations usually rely on its local neighborhood pixels of the input image, the rolling shutter effect correction problem is more challenging. A pixel of the rectified image might lie far away from its corresponding pixel of the input rolling shutter image, depending on the types of motion, 3D scene structure as well as its capturing time. To resolve these challenges, we propose a novel network architecture for rolling shutter image correction.”) Overall, one of ordinary skill in the art would have recognized that combining the rolling-shutter simulation system of Liu with Abboush and Cyriac would enable more realistic simulation of the output of vehicle sensors and therefore an overall more useful and accurate testing system.
(7) Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Hardware-in-the-Loop-Based Real-Time Fault Injection Framework for Dynamic Behavior Analysis of Automotive Software Systems (Hereinafter Abboush) in view of Assessment and Assurance of Autonomous Vehicle Safety (Hereinafter Cyriac) in further view of Farabet (US 11436484 B2)
Claim 14. Cyriac teaches wherein transmitting the target sensor data to the autonomous driving software system comprises: ([Page 62 Par 2- Page 65 Par 1] “The safety assessment approach uses three key components, namely 1) the urban driving simulator CARLA [64], 2) the imitation learning based convolutional neural network (IL-CNN) [65] as the autonomous driving agent (ADA) and 3) AVFI as the fault injection-based assessment for the ADA. The overall architecture of the simulation and fault injection framework of our approach is shown in Figure 10. … CARLA is an open source open world driving simulator which has an inbuilt library of urban roads, environments, pedestrians, vehicles and weather conditions (e.g., clear, rainy, sunset). It also provides the flexibility to users to create their own maps and assets so that they can recreate any scenario or environment they want to test. CARLA also provides an autonomous driving sensor suite which includes a variety of sensors like RGB cameras, LIDAR and GPS that can be ingested by ADAs for simulations. Figure 11 shows a screenshot of the CARLA simulator. CARLA operates by running two components. The server is responsible for generating and rendering the virtual environments while the client functions as an ADA, which receives sensor inputs from the server and sends control commands to the server (as shown in Figure 10). The server leverages Unreal Engine (popularly used in making video games) as the rendering and physics engine. The ADA under test uses [65] as the autonomous controller, which in turn employs an IL-CNN for the perception, planning and localization systems. In the test environment used for this experiment, the client is fed with RGB images from the forward facing camera sensor on the hood of the AV. The server sends the camera images together with other measurements of the car like speed and location to the client. The neural network model in [65] takes the sensor data from the server, perceives the image and produces a driving action for the AV. These driving decisions are then sent from the client to server, which services those commands and applies them to the AV’s actuators. This enables the AV to complete missions like navigating from one point to another in the simulated world.”)
([Page 62 Par 2- Page 65 Par 1] “… CARLA also provides an autonomous driving sensor suite which includes a variety of sensors like RGB cameras, LIDAR and GPS that can be ingested by ADAs for simulations. Figure 11 shows a screenshot of the CARLA simulator. CARLA operates by running two components. The server is responsible for generating and rendering the virtual environments while the client functions as an ADA, which receives sensor inputs from the server and sends control commands to the server (as shown in Figure 10). The server leverages Unreal Engine (popularly used in making video games) as the rendering and physics engine. The ADA under test uses [65] as the autonomous controller, which in turn employs an IL-CNN for the perception, planning and localization systems. In the test environment used for this experiment, the client is fed with RGB images from the forward facing camera sensor on the hood of the AV. The server sends the camera images together with other measurements of the car like speed and location to the client. The neural network model in [65] takes the sensor data from the server, perceives the image and produces a driving action for the AV. These driving decisions are then sent from the client to server, which services those commands and applies them to the AV’s actuators. …”)
transmitting the ([Page 62 Par 2- Page 65 Par 1] “The safety assessment approach uses three key components, namely 1) the urban driving simulator CARLA [64], 2) the imitation learning based convolutional neural network (IL-CNN) [65] as the autonomous driving agent (ADA) and 3) AVFI as the fault injection-based assessment for the ADA. The overall architecture of the simulation and fault injection framework of our approach is shown in Figure 10. … CARLA is an open source open world driving simulator which has an inbuilt library of urban roads, environments, pedestrians, vehicles and weather conditions (e.g., clear, rainy, sunset). It also provides the flexibility to users to create their own maps and assets so that they can recreate any scenario or environment they want to test. CARLA also provides an autonomous driving sensor suite which includes a variety of sensors like RGB cameras, LIDAR and GPS that can be ingested by ADAs for simulations. Figure 11 shows a screenshot of the CARLA simulator. CARLA operates by running two components. The server is responsible for generating and rendering the virtual environments while the client functions as an ADA, which receives sensor inputs from the server and sends control commands to the server (as shown in Figure 10). The server leverages Unreal Engine (popularly used in making video games) as the rendering and physics engine. The ADA under test uses [65] as the autonomous controller, which in turn employs an IL-CNN for the perception, planning and localization systems. In the test environment used for this experiment, the client is fed with RGB images from the forward facing camera sensor on the hood of the AV. The server sends the camera images together with other measurements of the car like speed and location to the client. The neural network model in [65] takes the sensor data from the server, perceives the image and produces a driving action for the AV. These driving decisions are then sent from the client to server, which services those commands and applies them to the AV’s actuators. This enables the AV to complete missions like navigating from one point to another in the simulated world.”)
The combination of Abboush and Cyriac does not explicitly teach converting a format of virtual data into a format of data acquired by the sensor during a road test; the format-converted data
Farabet teaches converting a format of virtual data into a format of data acquired by the sensor during a road test; the format-converted data ([Abstract] “In various examples, physical sensor data may be generated by a vehicle in a real-world environment. … virtual sensor data generated by virtual sensors within the simulated environment may be encoded to a format consistent with the format of the physical sensor data generated by the vehicle.”)
Farabet is analogous art because it is within the field of vehicle sensor simulation. It would have been obvious to one of ordinary skill in the art to combine Farabet with Abboush and Cyriac before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to generate more accurate sensor data. Farabet notes how typical systems that generate sensor data for vehicle system testing and training struggle with accuracy, and the tradeoffs between more accurate real-world data and safer, easier to produce synthetic data ([Col 1 line 25-53] “Autonomous vehicles and semi-autonomous vehicles rely on machine learning, and specifically deep neural networks (DNNs), for performing any number of operations for operating, piloting, and navigating the vehicle. For example, DNNs may be used for object detection, lane and road boundary detection, safety analysis, drivable free-space analysis, control generation during vehicle maneuvers, and the like. However, in order to verify and validate the usage of the DNNs in autonomous or semi-autonomous vehicles, the DNNs need to be trained on a large amount of data, which requires vast amounts of time and effort, and yet still does not always guarantee universally accurate or usable results. For example, conventional systems often rely on data generated by physical vehicles navigating real-world environments to train the DNNs prior to deployment in working models. This approach has several limitations, however. For example, vehicles can only navigate to so many places, recreating dangerous or unique situations is difficult in the real-world environment, and testing the DNNs in these real-world environments may be dangerous. For example, especially where a DNN is used for obstacle avoidance or other safety measures, testing the DNNs in real-world environments may not be practical. On the other hand, an automaker likely will not deploy an autonomous vehicle into the real-world until an acceptable level of safety has been achieved. As a result, these competing interests make generating a sound, safe, accurate, and reliable autonomous driving system increasingly difficult.”) To this end, Farabet presents a vehicle testing system capable of utilizing synthetic data, real-word data, and combinations of both ([Col 1 line 63- Col 2 line 24] “In contrast to conventional systems, such as those described above, the systems of the present disclosure leverage a simulated environment to test one or more autonomous driving software stacks that include a multitude of DNNs. For example, physical sensor data, virtual sensor data, or a combination thereof may be used to train the DNNs of the software stack(s). Once trained, the DNNs may be tested, verified, and validated within a simulation system that generates a simulated environment for controlling a virtual object using the software stack(s). Simulation data from the simulated environment may be input into the DNNs of the software stack(s), and the DNNs may generate outputs. In some examples, the outputs may be used to control the virtual object within the simulated environment, to determine how the virtual object (and thus a physical object that corresponds to the virtual object) may perform in any number of different situations. In other examples, the outputs may be used to test the accuracy of the DNNs, and the results may be used to generate more data for further training (e.g., data that the DNNs are least consistent processing accurately), fine-tune the DNNs, verify the DNNs, and/or validate the DNNs. In any example, the simulated environment may be generated to create difficult to navigate, dangerous, unsafe, and/or otherwise unpredictable situations for the virtual object to navigate. As a result, previously untested scenarios (e.g., due to safety concerns, difficulty of reproduction, etc.) may be tested, repeated, and improved upon within the simulated environment.” [Col 13 line 51- 61] “As a result, the identical, or substantially identical, hardware components used by the vehicle 102 (e.g., a physical vehicle) to execute the autonomous driving software stack in real-world environments may be used to execute the autonomous driving software stack in the simulated environment 410. The use of the vehicle hardware 104 in the simulation system 400A thus provides for a more accurate simulation of how the vehicle 102 will perform in real-world situations, scenarios, and environments without having to actually find and test the vehicle 102 in the real-world.” [Col 6 line 11-18] “The data store(s) 120 may store sensor data and/or virtual sensor data generated by one or more real-world sensors of one or more vehicle(s) 102 and/or virtual sensors of one or more virtual vehicles, respectively. Data ingestion 122 may include generating and/or recording the data output by the vehicle(s) 102 autonomous vehicle platform (e.g., the vehicle hardware 104 and/or the software stack(s) 116).” [Col 11 line 23-35] “The simulated environment 410 may be generated using virtual data, real-world data, or a combination thereof. For example, the simulated environment may include real-world data augmented or changed using virtual data to generate combined data that may be used to simulate certain scenarios or situations with different and/or added elements (e.g., additional AI objects, environmental features, weather conditions, etc.). For example, pre-recorded video may be augmented or changed to include additional pedestrians, obstacles, and/or the like, such that the virtual objects (e.g., executing the software stack(s) 116 as HIL objects and/or SIL objects) may be tested against variations in the real-world data.”) Overall, one of ordinary skill in the art would have recognized that combining Farabet with Abboush and Cyriac would result in a system that was capable of utilizing a greater diversity of data, enabling even more extensive testing and sensor data coverage.
Conclusion
The following prior art is made of record but not relied upon:
ML-Based Fault Injection for Autonomous Vehicles: A Case for Bayesian Fault Injection – While this reference teaches a thorough fault injection system, it performs its injection at a sensor abstraction layer that combines data from multiple sensors, and does not perform its injection on any direct sensor output.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael P Mirabito whose telephone number is (703)756-1494. The examiner can normally be reached M-F 10:30 am - 6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emerson Puente can be reached at (571) 272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.P.M./ Examiner, Art Unit 2187
/EMERSON C PUENTE/ Supervisory Patent Examiner, Art Unit 2187
