DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicant
The following is a Non-Final, first Office Action responsive to Applicant’s communication of 8/2/23, in which applicant filed the application. Claims 1-11 are pending in the instant application and have been rejected below.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/9/2023 is being considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Here, claim 2 recites “surrogate model construction means.” Based on [0079, 0081, 0153] as published, the corresponding structure is a computer processor executing stored instructions to perform the function recited.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 2 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recites the limitation "surrogate model construction means". Claim 1 already removed the phrasing, so there is insufficient antecedent basis for this limitation (requiring “means”) in the claim. Examiner suggests amending claim 2 consistent with claim 1 to recite “wherein of the surrogate model is for each domain unit.” Other similar amendments, would also be acceptable.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-11 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e. an abstract idea) without reciting significantly more.
Step One - First, pursuant to step 1 in MPEP 2106.03, the claim 1 is directed to a system which is a statutory category. Step 2A, Prong One - MPEP 2106.04 - The claim 1 recites a series of mathematical relationships – construct a surrogate model that simulates behavior of an analysis target, using operation data of the analysis target as training data;
select the surrogate model that is judged to best reproduce the behavior of the analysis target by the operation data according to verification contents, under specified condition, from the operation data of the analysis target; and
verify the analysis target using the selected surrogate model.
At this time, the claim is viewed as a series of mathematical relationships. Construct a “surrogate model”, in Applicant’s [0003] as published , states the “surrogate model is an alternative model for evaluation by simulation and are used to derive approximate solutions” and [0027] as published states a “model that enables refinement of approximate solutions.” The claim is “simulates behavior of analysis target” which is any mathematical relationship of a prediction using “operation data.” The final step is “verify” the analysis using the selected surrogate model (which is a prediction). Without further details, at this time, the claim is directed to an abstract idea.
Step 2A, Prong Two - MPEP 2106.04 - This judicial exception is not integrated into a practical application. At this time, additional elements include: a memory storing instructions; and
one or more processors configured to execute the instructions to:
…
using operation data of the analysis target as “training data”.
To extent “training data” in the 1st step requires “machine learning”, this is considered an additional element at step 2a, prong two and step 2B as “apply it [abstract idea] on a computer” (MPEP 2106.05f) and “field of use” (MPEP 2106.05h) for generally “apply it” – “use training data” to then build a simulation/prediction.
The memory, processors executing instructions are viewed as apply the exception using a generic computer component (See MPEP 2106.05(f)) and field of use (MPEP 2106.05h). Accordingly, the additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B in MPEP 2106.05 - The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, even if the claim is interpreted/amended to include a computer, it is considered MPEP 2106.05(f) (Mere Instructions to Apply an Exception – “Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible.” Alice Corp., 134 S. Ct. at 235) and MPEP 2106.05h (field of use) as above in Step 2a, prong two. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept.
The claim fails to recite any improvements to another technology or technical field, improvements to the functioning of the computer itself, use of a particular machine, effecting a transformation or reduction of a particular article to a different state or thing, adding unconventional steps that confine the claim to a particular useful application, and/or meaningful limitations beyond generally linking the use of an abstract idea to a particular environment. See 84 Fed. Reg. 55. The claim is not patent eligible. Viewed individually or as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself.
Independent claim 8, at Step One, is pursuant to step 1 in MPEP 2106.03, directed to a method which is a statutory category. It is directed to an abstract idea for the same reasons as claim 1; it is rejected for the same reasons at step 2a, prong two and step 2B. Examiner notes that at this time, claim 8 does not even recite a computer. Examiner suggests as a first step, reciting a computer perform each step. Once the claim is amended, the claim is still rejected for the same reasons as claim 1. These limitations individually or in combination are viewed as “apply it [abstract idea] on a computer” (MPEP 2106.05f) and “field of use” (MPEP 2106.05h).
Claim 10, at Step One, is pursuant to step 1 in MPEP 2106.03, directed to a method which is a statutory category. It is directed to an abstract idea for the same reasons as claim 1; it is rejected for the same reasons at step 2a, prong two and step 2B. These limitations individually or in combination are viewed as “apply it [abstract idea] on a computer” (MPEP 2106.05f) and “field of use” (MPEP 2106.05h).
Claims 2, 9, and 11 narrows the abstract idea by having a surrogate model (for prediction/simulation) for each domain unit (i.e. topic) that represents a smallest amount of verification.
Claims 3, 10, and 12 narrows the abstract idea by constructing the surrogate model for each domain by grouping by function.
Claim 5 narrows the abstract idea by inputting a parameter when defects occurred to calculate a first evaluation value, apply under a normal condition to the surrogate model to calculate a second evaluation value and comparing the 1st and 2nd evaluation values.
Claim 6 narrows the abstract idea for similar reasons as claim 5 with parameters, evaluation values, and defects.
Claim 7 narrows the abstract idea by selecting a surrogate that matches an item to be analyzed.
Therefore, the claim(s) are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
For more information on 101 rejections, see MPEP 2106.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3 and 8-11 are rejected under 35 U.S.C. 103 as being unpatentable over Ben Abdessalem, et al, “Testing Advanced Driver Assistance Systems using Multi-Objective Search and Neural Networks,” 2016, In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, pages 63-74, in view of Ros Sanchez (US 2019/0317510).
Concerning claim 1, Ben Abdessalem disclose:
A verification system (Ben Abdessalem – see page 63, Col. 2 - an advanced driver assistance system (ADAS); Pedestrian Detection Vision based (PeVi) system; see page 70, col. 1, 1st paragraph - We ran all the experiments on a laptop with a 2.5 GHz CPU and 16GB of memory. Based on our experiments, each PeVi simulation (i.e., each call to PreScan), on average, takes 2 min) comprising:
a memory storing instructions (Ben Abdessalem - see page 70, col. 1, 1st paragraph - We ran all the experiments on a laptop with a 2.5 GHz CPU and 16GB of memory.
see also Ros Sanchez see par 34 - Instructions executed at a processor (e.g., CPU 102) may be loaded from a program memory associated with the CPU 102)); and
one or more processors configured to execute the instructions (Ben Abdessalem - see page 70, col. 1, 1st paragraph - We ran all the experiments on a laptop with a 2.5 GHz CPU and 16GB of memory.
see also Ros Sanchez see par 34 - Instructions executed at a processor (e.g., CPU 102) may be loaded from a program memory associated with the CPU 102; see par 81 - The processor may be responsible for managing the bus and processing, including the execution of software stored on the machine-readable media.) to:
construct a surrogate model that simulates behavior of an analysis target, using operation data of the analysis target as training data (Ben Abdessalem – see page 67, col. 1, 1st paragraph - We create a surrogate model for each fitness function to predict the fitness values without running the actual simulations. Such surrogate models are often developed using machine learning techniques such as classification, regression or neural networks [5]. Neural networks can be used with supervised or unsupervised training algorithms [33]. In our work, we are able to obtain output values for training input data by running simulations. Hence, we use neural networks in a supervised training mode; see page 67, col. 1, 3rd paragraph - Given a fitness function F, we build a surrogate model of F by training a neural network. To do so, we use a set of observations containing input values and known output values [56]. We divide the observation set into a training set and a test set
see also Ros Sanchez – see par 55 - FIG. 6 is a block diagram of an autonomous agent system configured for full-stack verification using system simulated sensor information, according to aspects of the present disclosure. In this configuration, an autonomous agent system 600 (or alternatively an autonomous vehicle system) is configured to replace an object detector module (e.g., 460) with a neural network 620 (e.g., a surrogate function 1) trained to generate simulated sensor information data in response to ground truth information 610. see par 62, FIG. 7 - At block 708, a surrogate function Φ is learned based on a set of outputs of the detection module M.sub.i along with an expected response and a vector of measurements. );
select the surrogate model that is judged to best reproduce the behavior of the analysis target by the operation data according to verification contents, under specified condition, from the operation data of the analysis target (Ben Abdessalem – see page 67, Col. 1, 3rd paragraph - Given a fitness function F, we build a surrogate model
of F by training a neural network. To do so, we use a set of observations containing input values and known output values; Finding the best values
for these parameters and selecting the best performing algorithm in our case is addressed in our empirical evaluation (Section 6). see page 69, Col. 2, Section 6 - We evaluate the prediction accuracy of surrogate models using the coefficient of determination (R2) [56] that measures the predictive power of a surrogate model by identifying how well a test set fits the model. The higher the value
of R2, the more accurate the surrogate model is.
Ros Sanchez – see par 62 - the set of outputs of the detection module is: M(o.sub.i)=ŷ.sub.i, (e.g., a noisy output ŷ.sub.i) along with the expected response y.sub.i (e.g., ground truth), and the vector of measurements m.sub.i are used to learn a surrogate function φ.sub.θ (y.sub.i,m.sub.i), where θ represents a set of learned parameters that define the behavior of φ. This process may be carried out using minimization by standard optimization techniques based on gradient decent and risk minimization methods. Here, £ is an appropriate loss function, such as cross-entropy or lp-norm (see Equation (1)). In this aspect of the present disclosure, the purpose of φ is to learn the noise model associated with the output of M.sub.i starting at an ideal prediction (ground truth).); and
verify the analysis target using the selected surrogate model (Ben Abdessalem –See page 63, Abstract – We provide a testing approach for ADAS by combining multiobjective search with surrogate models developed based on neural networks. We use multi-objective search to guide testing towards the most critical behaviors of ADAS. see page 67, col. 1, 3rd paragraph - The test set is, then, used to evaluate the accuracy of the predictions produced by ˆ F when applied to points outside the training set. see page 70, col. 1, 2nd paragraph - First, we identified the training algorithm and the configuration values that lead to the most accurate neural network-based surrogate models for the PeVi case study. To do so, for each of our three PeVi fitness functions, we compared 18 different neural network
configurations. The comparison is based on a k-fold cross validation with k = 5 [4, 24].
It is unclear if the “cross validation” sufficiently discloses the last step of “verify” the analysis target.
Ros Sanchez discloses:
“verify” the analysis target using the selected surrogate model (Ros Sanchez – see par 7 - The object detection module of the autonomous vehicle system may be replaced with the neural network and a sensory input of the autonomous vehicle system is replaced with ground truth information during the full-stack verification. The neural network may be further configured to apply a surrogate function to the ground truth information to simulate sensor information data to at least a planner module of the autonomous vehicle system. see par 23 - Aspects of the present disclosure are directed to a system and method for fast full-stack verification of autonomous agents using system simulated sensor information. In aspects of the present disclosure, the system generated sensor information simulates real-world conditions with sufficient fidelity for certifying correct behavior and safety of an autonomous agent under development. One configuration for full-stack autonomous agent verification is based on a strategy for reparametrizing modules in a pipeline of the autonomous system that consume sensory data. see par 64 - In aspects of the present disclosure, method 700 is repeated for each of the set of the other sensor modules of Csensor to generate a new set Φ={φ.sub.1, . . . , φ.sub.M}. Then the modules M are substituted by their surrogate φ.sub.i to form the autonomous agent system 600 of FIG. 6. In this example, the object detector module of the autonomous agent system 600 is replaced with the surrogate function Φ. see par 68, FIG. 8 - In order to produce a coo estimator with enough fidelity for verification of M.sub.i, it is important to have a well-sampled set of measurements {m.sub.i} that maximizes the coverage of an ontology associated to the module M.sub.i.
Ben Abdessalem and Ros Santos are analogous art as they are directed to using surrogate models with vehicle data (Ben Abdessalem Abstract; and Ros Santos Abstract). Ben Abdessalem discloses using a laptop with a 2.5 GHz CPU and 16GB of memory (see page 70, col. 1, 1st paragraph), testing with surrogate models (Abstract), and identifying the training algorithm and the configuration values that lead to the most accurate neural network-based surrogate models for the PeVi case study using a k-fold cross validation with k = 5 (see page 70, col. 1, 2nd paragraph). Ros Santos improves upon Ben Abdessalem by disclosing executing instructions loaded from program memory (see par 34, 81) and full stack verification for vehicle sensor data and certifying correct behavior and safety of an autonomous agent (See par 7, 23) where surrogate used to maximize coverage (See par 64, 68). One of ordinary skill in the art would be motivated to further include executing instructions from stored memory and verification of sensor data and certifying of data to efficiently improve upon the CPU and memory and the “cross validation” in Ben Abdessalem.
Accordingly, it would have been obvious to one of ordinary skill in the art before
the effective filing date of the claimed invention to modify the system and method of
using surrogate modeling to explore search space for a vehicle as disclosed in Ben Abdessalem, and further using verification and certification of sensor data as disclosed in Ros Santos, since the claimed invention is merely a combination of old elements, and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable and there is a reasonable expectation of success.
Concerning independent claim 8, Ben Abdessalem and Ros Santos disclose:
A verification method (Ben Abdessalem – see page 63, Col. 2 - an advanced
driver assistance system (ADAS); Pedestrian Detection Vision based (PeVi) system; see page 70, col. 1, 1st paragraph - We ran all the experiments on a laptop with a 2.5 GHz CPU and 16GB of memory. Based on our experiments, each PeVi simulation (i.e., each call to PreScan), on average, takes 2 min) comprising
The remaining limitations are similar to claim 1 above, so claim 8 is rejected for the same reasons.
Concerning independent claim 10, Ben Abdessalem and Ros Santos disclose:
A non-transitory computer readable information recording medium storing a verification program, when executed by a processor, the data management program that performs a method for (Ben Abdessalem - see page 70, col. 1, 1st paragraph - We ran all the experiments on a laptop with a 2.5 GHz CPU and 16GB of memory.
see also Ros Sanchez see par 34 - Instructions executed at a processor (e.g., CPU 102) may be loaded from a program memory associated with the CPU 102; see par 81 - The processor may be responsible for managing the bus and processing, including the execution of software stored on the machine-readable media):
The remaining limitations are similar to claim 1 above, so claim 10 is rejected for the same reasons.
Concerning claims 2, 9, and 11, Ben Abdessalem and Ros Santos disclose:
The verification system according to claim 1, wherein the the surrogate model construction means constructs the surrogate model for each domain unit, which is the smallest unit of verification (Ben Abdessalem – see page 65, Col. 1, Section 2 - Based on the Pedestrian Detection Vision based (PeVi) specification, the cone-shaped space in front of a car that is scanned by the PeVi camera is divided into three warning areas illustrated in Figure 2; page 65, Col. 2, “PeVi Input and Output” - In general, PeVi’s function is impacted by several physical phenomena and environment factors. For example, road friction or wind may affect vehicle
speed, which in turn, influences PeVi’s behavior. We developed a domain model to precisely capture these elements. This domain model essentially specifies a restricted simulation environment that is sufficient for testing PeVi. page 65, col. 2, 2nd paragraph same section - The domain model is shown in Figure 3. Based on this model a test scenario for PeVi contains the following input: (1) the value of the scene light intensity; (2) the weather condition that can be normal, foggy, rainy, or snowy; (3) The road type that can be straight, curved, or ramped; (4) the
roadside objects, namely, trees and cars parked next to the road; (5) the camera’s field of view; (6) the initial speed of the vehicle; and (7) the initial position, the orientation (theta) and the speed of the pedestrian; see page 67 – difference between output of surrogate model and predictive function are minimized.
Ros Sanchez – see par 62 - the set of outputs of the detection module is: M(o.sub.i)=ŷ.sub.i, (e.g., a noisy output ŷ.sub.i) along with the expected response y.sub.i (e.g., ground truth), and the vector of measurements m.sub.i are used to learn a surrogate function φ.sub.θ (y.sub.i,m.sub.i), where θ represents a set of learned parameters that define the behavior of φ. This process may be carried out using minimization by standard optimization techniques based on gradient decent and risk minimization methods. Here, £ is an appropriate loss function, such as cross-entropy or lp-norm (see Equation (1)). In this aspect of the present disclosure, the purpose of φ is to learn the noise model associated with the output of M.sub.i starting at an ideal prediction (ground truth).
It would be obvious to combine Ben Abdessalem and Ros Santos for the same reasons as claim 1 above.
Concerning claim 3, Ben Abdessalem and Ros Santos discloses:
The verification system according to claim 1, wherein the processor is configured to execute the instructions to construct the surrogate model for each domain unit that groups units by automobile function (Ben Abdessalem - page 65, Col. 2, “PeVi Input and Output” - In general, PeVi’s function is impacted by several physical phenomena and environment factors. For example, road friction or wind may affect vehicle speed, which in turn, influences PeVi’s behavior. We developed a domain model to precisely capture these elements (e.g. road friction/winfd; environment factors). This domain model essentially specifies a restricted simulation environment that is sufficient for testing PeVi. page 65, col. 2, 2nd paragraph same section - The domain model is shown in Figure 3. Based on this model a test scenario for PeVi contains the following input: (1) the value of the scene light intensity; (2) the weather condition that can be normal, foggy, rainy, or snowy; (3) The road type that can be straight, curved, or ramped; (4) the roadside objects, namely, trees and cars parked next to the road; (5) the camera’s field of view; (6) the initial speed of the vehicle; and (7) the initial position, the orientation (theta) and the speed of the pedestrian.
Ros Sanchez – see par 33 - The estimated noise model is combined with the ground truth 3D sensory image data to represent system simulated sensor information data. This system simulated sensor information data is generated by a trained neural network module of an autonomous agent in response to the ground truth 3D sensor image data. The system simulated sensor information data is used rather than a sensor signal from one or more sensors (e.g., light detection and ranging (LIDAR) sensors, sonar sensors, red-green-blue (RGB) cameras, RGB-depth (RGB-D) cameras, and the like) of the autonomous agent).
It would be obvious to combine Ben Abdessalem and Ros Santos for the same reasons as claim 1 above.
Claims 4-7 are rejected under 35 U.S.C. 103 as being unpatentable over Ben Abdessalem, et al, “Testing Advanced Driver Assistance Systems using Multi-Objective Search and Neural Networks,” 2016, In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, pages 63-74, in view of Ros Sanchez (US 2019/0317510), as applied to claims 1-3 and 8-11 above, and further in view of Beglerovic, et al., “Testing of autonomous vehicles using surrogate models and stochastic optimization,” In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pages 1-6.
Concerning claim 4, Ben Abdessalem and Ros Santos disclose:
The verification system according to claim 1, wherein the processor is configured to execute the instructions to:
input the operation data including a parameter used in operating the analysis target, apply the input parameters to the selected surrogate model to calculate an evaluation value, and identify the optimal parameters as a verification result based on the calculated evaluation value (Ben Abdessalem – see page 67, Section 4 - We cast the problem of test case generation for ADAS as a multi-objective search optimization problem. Specifically, we identified three fitness functions in Section 2 to characterize critical behaviors of the PeVi system and its environment. The solutions to our problem are obtained by minimizing these three fitness functions using a multiobjective Pareto optimal approach; computing fitness functions F1… Fk for each individual in P (line 4) (see FIG. 4, showing Non-dominated Sorting Genetic Algorithm (NSGAII)); FIG. 4, line 6 is “rank”; see page 70, Col. 1, 2nd paragraph - To do so, for each of our three PeVi fitness functions, we compared 18 different neural network configurations. The comparison is based on a k-fold cross validation with k = 5); and
apply the obtained verification result to a simulator for the analysis target… (Ben Abdessalem –See page 63, Abstract – We provide a testing approach for ADAS by combining multiobjective search with surrogate models developed based on neural networks. We use multi-objective search to guide testing towards the most critical behaviors of ADAS.
Ros Santos – see par 46 - Aspects of the present disclosure are directed to a system and method for fast full-stack verification (e.g., full system verification) of autonomous agents. Verification of autonomous agents is an important task for creating and certifying new autonomous systems, such as driverless cars and robots. Conventional systems for verifying autonomous systems and/or agents rely on logging the behavior of the system and/or agents under study while the system/agent interacts with its surrounding environment. This verification process may involve creating stress situations to test the behavior of agents in corner cases or atypical situations. see par 55- neural network (surrogate function 1); see par 56 – planner modules uses surrogate function (FIG. 6); Following improving of the modules within the pipeline of the autonomous agent system 600 to achieve the full-stack verification, the controller may include a collision avoidance module 642. In aspects of the present disclosure, collision avoidance module 642 provides decision making and control over the behavior of autonomous vehicle 602; see par 68-69 – FIG. 8 is a flowchart illustrating a method for training modules in simulation for full-stack verification of an autonomous agent system; produce a coo estimator with enough fidelity for verification of M.sub.i, it is important to have a well-sampled set of measurements {m.sub.i} that maximizes the coverage of an ontology associated to the module M.sub.i. , using the method of 800 in FIG. 8; surrogate function coo is learned; ).
To any extent Ben Abdessalem and Ros Sanchez don’t disclose “Evaluation values,” Etheridge discloses the remainder of the limitations:
input the operation data including a parameter used in operating the analysis target, apply the input parameters to the selected surrogate model to calculate an “evaluation value,” and identify the optimal parameters as a verification result based on the calculated evaluation value (Beglerovic see page 3, col. 1, 1st paragraph and FIG. 2 - By selecting an appropriate cost function cψ, it is possible to guide the testing towards regions where the behavior is not satisfactory and where the evaluation criterion ψ is not satisfied); See page 3, col. 2, 2nd paragraph - If the faulty behavior has not been found and if the maximum number of iterations has not been reached, the numerical evaluations of the cost functions [c1...cn] are passed to the Surrogate modeling block. The surrogate model is iteratively extended using the values of [c1...cn] and provides the approximation function ˆcψ(p) to the Stochastic Optimization where various optimization algorithms can be used.)
apply the obtained verification result “to a simulator for the analysis target to obtain operating result” (Beglerovic see FIG. 2 – after cost function and surrogate modeling, simulation is performed; see page 4, Section V, 1st paragraph – for surrogate modeling , we use the Radial Basis Function Approximation to find an estimation of a real system or process and assigning a radial basis symmetrical kernel function of each sample. 2nd paragraph – 3rd paragraphs – tested kernels; use Kriging model and Gaussian kernels; to validate proposed method, a highway scenario used; testing is done for an emergency brake assist ADAS system; goal is to avoid collisions by breaking. see page 5, Section A-B – FIG> 3 and simulation setup overview in FIG. 4)
It would be obvious to combine Ben Abdessalem and Ros Santos for the same reasons as claim 1 above. Ben Abdessalem, Ros Santos, and Beglerovic are analogous art as they are directed to using surrogate models with vehicle data (Ben Abdessalem Abstract; and Ros Santos Abstract; Beglerovic Abstract). Ben Abdessalem discloses testing with surrogate models (Abstract) and identifying the training algorithm and the configuration values that lead to the most accurate neural network-based surrogate models for the PeVi case study using a k-fold cross validation with k = 5 (see page 70, col. 1, 2nd paragraph). Ros Santos improves upon Ben Abdessalem by disclosing full stack verification for vehicle sensor data and certifying correct behavior and safety of an autonomous agent (See par 7, 23, 46, 55-56) where surrogate used to maximize coverage and simulation for verification used (See par 64, 68-69). Beglerovic improves upon Ben Abdessalem and Ros Sanchez by disclosing using evaluation criterion, testing, and simulation for surrogate models to then have a result. One of ordinary skill in the art would be motivated to further include using evaluation criterion, testing, and simulation for surrogate models to then have a result to efficiently improve upon the surrogate models in Ben Abdessalem and the surrogate models and verification in Ros Sanchez.
Accordingly, it would have been obvious to one of ordinary skill in the art before
the effective filing date of the claimed invention to modify the system and method of
using surrogate modeling to explore search space for a vehicle as disclosed in Ben Abdessalem, to further use verification and certification of sensor data as disclosed in Ros Santos, to further use using evaluation criterion, testing, and simulation for surrogate models as disclosed in Beglerovic, since the claimed invention is merely a combination of old elements, and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable and there is a reasonable expectation of success.
Concerning claim 5, Ben Abdessalem and Ros Santos and Beglerovic disclose:
The verification system according to claim 1, wherein the processor is configured to execute the instructions to:
input data at time of a defect, including a parameter when the defect occurred in the analysis target (Ben Abdessalem – see page 72, Section 8 - developed meta-heuristics capturing critical aspects of the system and its environment to guide the search toward exercising behaviours that are likely to reveal faults; combining our search algorithm with neural networks improves
the quality of the generated test cases under a limited
and realistic time budget), and apply the input parameter to the selected surrogate model to calculate a first evaluation value (Beglerovic [disclosing entire limitation]–
page 2, col. 2, last paragraph - By limiting our search for a specific
set of parameters p ∈ ˆ P, we can vastly improve the speed and avoid exploring regions of the search space ˆ P that are of no interest. see page 3, col. 1, 1st paragraph and FIG. 2 - By selecting an appropriate cost function cψ, it is possible to guide the testing towards regions where the behavior is not satisfactory and where the evaluation criterion ψ is not satisfied; see FIG. 2, page 3; page 3, col. 2, last paragraph – A new evaluation is done with the new parameters and a better
model of the approximated function ˆcψ(p) is built until the algorithm reaches a faulty behavior or the maximum number of iterations.); and
apply data under the normal condition to the surrogate model to calculate a second evaluation value (Ben Abdessalem – see page 65, col. 2 – weather condition can be “normal”), and compare the first evaluation value with the second evaluation value to estimate a part of the defect (Beglerovic [disclosing entire limitation]– see page 2, col. 1, 2nd paragraph - In order to identify the faulty behavior regions, appropriate cost functions, which can be minimized by various optimization methods, must be defined. As we are interested in the region around the worst behavior, and we want to avoid false positives in the form of local minima, global optimization methods are needed. One drawback of global optimization algorithms is that they require many function evaluations (simulation runs) to find the global optimum. To overcome this limitation in the proposed approach, we create a surrogate model with inexpensive evaluations on which the optimization algorithms can run. With each new iteration a better model of the system is built around the faulty region; see page 2, col. 2, 4th paragraph - each model M exhibits certain behavior during the simulation or real world trial. This behavior is denoted as Φ(M,p) of the model M with respect to the set of parameters p ∈ P. Φ(M,P) represents the behavior of M in respect to all possible variations of parameters in the parameter space P. If some behavior Φ(M,p) satisfies the criteria ψ, the system is working correctly, and we can write Φ(M,p) |= ψ.
see page 4, Section V, 1st paragraph - For the surrogate modeling, we decided to use the RBF - Radial Basis Function approximation. The main idea of RBF is to find an estimation ˆ f(x) of a real system or process f(x) by sampling the function f(x) in samples xi ∈ Xs and assigning a radial basis symmetrical kernel function φ for each sample; see page 5, Section A - A detailed overview of the scenario can be seen in Fig.3. The vehicle starts from a still stand and accelerates with constant acceleration until it reaches a maximum velocity of 100km/h , moving along the x axis. Simulation duration is 10 seconds at which the car reaches 128 m. The parameter space ˆ P is represented by the obstacle’s (x, y) coordinates,
leading to a 2D search space. An error θ = 1, 5◦ is introduced ranging from [1◦, 2.5◦], and a search for the worst case crash is going to be conducted; see page 5, Section C – selection of cost function; explore necessary and sufficient conditions for a collision to occur; Some good practices that can be considered when constructing a cost function are: smoothness - as it will
enable a better approximation when using surrogate modeling and convexity - this will ensure quicker convergence to the global minimum. For our case study, we are going to use a cost function based on the time to collision ttc between the
vehicle and obstacle. Crashes that occur at higher speed have lower evaluation
value then crashes at lower speeds).
It would be obvious to combine Ben Abdessalem and Ros Santos and Beglerovic for the same reasons as claim 1 above. In addition, Beglerovic improves upon Ros Santos and Beglerovic has an evaluation function for approaching “real world trial” using evaluation value (See page 2, Col. 2, 4th paragraph), exploring faulty regions of a search space (page 2), and page 5 gives explanation of vehicle parameters from a scenario, along with error in the sensor data, and different evaluation/cost function values.
Concerning claim 6, Ben Abdessalem and Ros Santos and Beglerovic disclose:
The verification system according to claim 1, wherein the processor is configured to execute the instructions to:
input evaluation data including a parameter under normal conditions of the analysis target, and apply the input parameter to the selected surrogate model (Ben Abdessalem – see page 65, col. 2, last section - Based on this model a test scenario for PeVi contains the following input: (1) the value of the scene light intensity; (2) the weather condition that can be normal, foggy, rainy, or snowy…; see page 67, “Search with Surrogate Model”; See page 68, col. 1, 3rd paragraph, FIG. 4 – predicted fitness values obtained from surrogate models to compute the partial order rank and select the best individuals A) to calculate an evaluation value (Beglerovic [disclosing entire limitation]– page 2, col. 2, last paragraph - By limiting our search for a specific set of parameters p ∈ ˆ P, we can vastly improve the speed and avoid exploring regions of the search space ˆ P that are of no interest. see page 3, col. 1, 1st paragraph and FIG. 2 - By selecting an appropriate cost function cψ, it is possible to guide the testing towards regions where the behavior is not satisfactory and where the evaluation criterion ψ is not satisfied; see FIG. 2, page 3; page 3, col. 2, last paragraph – A new evaluation is done with the new parameters and a better model of the approximated function ˆcψ(p) is built until the algorithm reaches a faulty behavior or the maximum number of iterations); and
verify, when a defect is estimated to occur based on the evaluation value, the operation using a simulator for the analysis target Ben Abdessalem –See page 63, Abstract – We provide a testing approach for ADAS by combining multiobjective search with surrogate models developed based on neural networks. We use multi-objective search to guide testing towards the most critical behaviors of ADAS.
Ros Santos – see par 46 - Aspects of the present disclosure are directed to a system and method for fast full-stack verification (e.g., full system verification) of autonomous agents. Verification of autonomous agents is an important task for creating and certifying new autonomous systems, such as driverless cars and robots. Conventional systems for verifying autonomous systems and/or agents rely on logging the behavior of the system and/or agents under study while the system/agent interacts with its surrounding environment. This verification process may involve creating stress situations to test the behavior of agents in corner cases or atypical situations. see par 55- neural network (surrogate function 1); see par 56 – planner modules uses surrogate function (FIG. 6); Following improving of the modules within the pipeline of the autonomous agent system 600 to achieve the full-stack verification, the controller may include a collision avoidance module 642. In aspects of the present disclosure, collision avoidance module 642 provides decision making and control over the behavior of autonomous vehicle 602; see par 68-69 – FIG. 8 is a flowchart illustrating a method for training modules in simulation for full-stack verification of an autonomous agent system; produce a coo estimator with enough fidelity for verification of M.sub.i, it is important to have a well-sampled set of measurements {m.sub.i} that maximizes the coverage of an ontology associated to the module M.sub.i. , using the method of 800 in FIG. 8; surrogate function coo is learned) for an estimated part of the defect (Beglerovic – see FIG. 2 – after cost function and surrogate modeling, simulation is performed; see page 4, Section V, 1st paragraph – for surrogate modeling , we use the Radial Basis Function Approximation to find an estimation of a real system or process and assigning a radial basis symmetrical kernel function of each sample. 2nd paragraph – 3rd paragraphs – tested kernels; use Kriging model and Gaussian kernels; to validate proposed method, a highway scenario used; testing is done for an emergency brake assist ADAS system; goal is to avoid collisions by breaking; see page 5, Section B – Simulation setup - The vehicle dynamics and ADAS model represent the model M from the problem statement. The simulation setup overview can be seen in Fig.4. In order to use the proposed approach, only a helper function needs to be available to run the simulation model with parameters p ∈ ˆ P and receive back the evaluated cost function; see page 6, col. 1, 2nd paragraph - We can conclude that the Kriging model managed to find test cases with crashes with higher probability while using less
simulation evaluations).
It would be obvious to combine Ben Abdessalem and Ros Santos and Beglerovic for the same reasons as claim 1 and 4-5 above.
Concerning claim 7, Ben Abdessalem and Ros Santos disclose:
The verification system according to claim 1, wherein the processor is configured to execute the instructions to select the surrogate model that matches a data item to be analyzed and an input/output data item (Ben Abdessalem – see page 65, col. 2 - In general, PeVi’s function is impacted by several physical phenomena and environment factors. For example, road friction or wind may affect vehicle
speed, which in turn, influences PeVi’s behavior. However, given that the testing budget both in terms of manual and computational effort is limited, we identified, through our discussions with the domain expert, the most essential elements impacting the PeVi system. We developed a domain model to precisely capture these elements. This domain model essentially specifies a restricted simulation environment that is sufficient for testing PeVi. Further, this domain model characterizes the PeVi inputs and the outputs generated after simulating PeVi;
See also Beglerovic – see page 2, col. 2 - each model M exhibits certain behavior during the simulation or real world trial; This behavior is denoted as Φ(M,p) of the model M with respect to the set of parameters p ∈ P. Φ(M,P) represents the behavior of M in respect to all possible variations of parameters in the parameter space P; page 2, col. 2, last paragraph - By limiting our search for a specific set of parameters p ∈ ˆ P, we can vastly improve the speed and avoid exploring regions of the search space ˆ P that are of no interest; see FIG. 2, page 3; page 3, col. 2, last paragraph – A new evaluation is done with the new parameters and a better model of the approximated function ˆcψ(p) is built until the algorithm reaches a faulty behavior or the maximum number of iterations see page 4, col. 2, 3rd paragraph - Beside the most likely global minimum search, the Kriging models require another optimization step for finding adequate parameters γ and p. The value of p was fixed to p = 2, as proposed in [15], leading to a simpler optimization task for finding γ. It is important to state that the limitation of all surrogate models is that the cost function needs to be smooth in order to achieve the best modeling results; see page 6, Summary - We proposed an approach
where a computationally inexpensive surrogate model of the system behavior is built, and optimization algorithms are then applied on the surrogate and not the real system. For the optimization tasks the Differential Evolution and Particle Swarm Optimization were implemented. The testing evaluation was conducted on a highway scenario and an Emergency Breaking Assist ADAS. The scenario consisted of a passenger car driving in a straight line and an obstacle
position was varied. An error in the sensor’s field of vision was introduced and the task of the algorithms was to find the test case with the worst crash evaluation).
It would be obvious to combine Ben Abdessalem and Ros Santos and Beglerovic for the same reasons as claim 1 and 4-6 above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Hong (US 2023/0051737) – directed to considering errors in vehicle acceleration data (See Abstract, par 69), creating a plurality of surrogate models corresponding to a plurality of scenarios that includes temperature, speed (See par 72-73).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IVAN R GOLDBERG whose telephone number is (571)270-7949. The examiner can normally be reached 830AM - 430PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anita Coupe can be reached at 571-270-3614. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IVAN R GOLDBERG/ Primary Examiner, Art Unit 3619
Prosecution Insights