CTNF 18/465,935 CTNF 100919 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. 12-151 AIA 26-51 12-51 Status of Claims Claims 1 – 20 are pending and examined herein. Claims 8 – 10, 13 – 19 are rejected under 35 U.S.C. 112(b). Claims 1 – 20 are rejected under 35 U.S.C. 101. Claims 1 – 20 are rejected under 35 U.S.C. 103. Drawings 06-22-06 AIA The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: [0050,54] vehicle 500 is not in drawings. [0058,60,61,64] battery 521 is not in drawings. [0069,75,76,78,84,85] vehicle 600 is not in drawings. [0075-77] image sensors 660, front facing image sensors 664, side facing image sensors 666, rear facing image sensors are not in drawings [0069] database 615 and database 617 are not in drawings. [0062, 65] electronic control unit 650 is not in drawings. [0075] vehicle acceleration sensors 621, vehicle speed sensors 622, wheelspin sensors 623 are not in drawings. [0083] Network 690 is not in drawings. [0072] controller/CPU 703 is not in drawings [0075] other sensors 732 is not in drawings . Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance. Specification 07-29 AIA The disclosure is objected to because of the following informalities: [0027,28] driving environment, operational environment as 101 which is supposed to be intersection. Correct reference number is 100 [0026] updated AI/ML model(s) as 142, which is used for corner case correction system. Correct reference number is 141 [0035] data stream as 211, which is used for context data. Correct reference number is 213 [0037] Encoder as 211, which is used for context data. Correct reference number is 212 Fig. 4 430 External resources not described in spec Fig. 4 refers to 414 as “Spatio temporal regular expression matching” in spec [0049,56] refers to 414 as case corner correction component. [0056] refers to 514 as case corner correction component. In Fig. 5, 514 is referred as ICE Fig. 5 422, 514A, 515, 524, 526, 532, 536, 540, 548 are not described in spec. Fig. 6 604, 605, 616, 624, 626, 632 are not described in specs. Fig. 6 refers to reference number 612, 614, 620 as V. Accel. ,V. Speed, TPMs but also refers to antenna, power supply, vehicle in same figure. [0075,76] describes reference 630 as proximity sensors instead of corner case correction as referred in drawing and other paragraphs of specification. Fig. 6 refers to 622 as ROLL/PITCH/YAW but [0075] refers to it as vehicle speed sensors. [0077,85] refers to 674 as vehicle audio system. Fig. 6 refers to 674 as Torque Splitter. [0085] refers to 676 as vehicle dashboard system. Fig. 6 refers to 676 as ICE control circuit. [0077] refers to 678 as object detection system. Fig. 6 refers to 678 as Cooling system. [0077] refers to 680 as suspension system [0085] refers to 680 as vehicle display system . Appropriate correction is required. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claims 8 – 10, 13 – 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 8 recites “the AL/ML trained using the virtual dataset” In line 2. Claim 8 is dependent to claim 1 and “AL/ML” lacks antecedent basis as it is not defined in any of these claims. It is unclear whether “AL/ML” is intended to refer to the previously recited “AI/ML model” of claim 1 or to another element. There is insufficient antecedent basis for this limitation in the claim. For examination purposes, “the AL/ML model” will refer to “AI/ML model” previously defined in claim 1. Claims 9 – 10 are dependent on claim 8. They do not resolve the issue of indefiniteness and are rejected with the same rationale. Claim 13 recites “a vehicle” in line 1 and “the vehicle” in line 6. Then it recites “the one or more deployed autonomous vehicles” in line 7. It is unclear whether “the one or more deployed autonomous vehicles” refers to the claimed vehicle, a plurality of vehicles including the claimed vehicles, or other deployed autonomous vehicles separate from the claimed vehicle. Also, claim 13 recites “updated AI/ML model” in line 7. It is unclear whether “updated AI/ML model” is supposed to refer to previously recited “the updated AI/ML model” or referring to another updated AI/ML model. There are insufficient antecedent basis for these limitations in the claim. For examination purposes, “the one or more deployed autonomous vehicles” refer to the same claimed vehicle and “updated AI/ML model” will be recited as “the updated AI/ML model” to refer the same updated AI/ML model. Claim 13 further recites “executing an autonomous control of the vehicle…” in line 6. While claim 13 is an apparatus claim for vehicle, the recited limitation reads as a method step rather than a structural limitation of the vehicle. It is unclear whether the claim is directed to a vehicle having a controller configured to perform certain functions, or to the act of executing autonomous control using the vehicle. Therefore, the scope of the apparatus claim is unclear and recited limitation will be referred as that same controller is executing recited function for the examination purpose. Claims 13 – 19 are dependent on claim 13. They do not resolve the issue of indefiniteness and are rejected with the same rationale. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1 - 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. MPEP § 2109(III) sets out steps for evaluating whether a claim is drawn to patent-eligible subject matter. The analysis of claims 1 – 20, in accordance with these steps, follows. Step 1 Analysis: Step 1 is to determine whether the claim is directed to a statutory category (process, machine, manufacture, or composition of matter. Claims 1 – 12 are directed to a method, meaning that it is directed to the statutory category of process. Claims 13 – 19 are directed to a vehicle, which is the statutory category of machine. Claim 20 is directed to a computer system, which is also the statutory category of machine. Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis: Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101. Regarding claim 1 , the following claim elements are abstract ideas: generating a virtual dataset from an initial dataset, (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper. ) performing formal verification of the AI/ML model; (Performing verification of the AI/ML model is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.) … to execute predictive analysis … (Executing predictive analysis is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.) The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception: wherein the virtual dataset comprises synthesized data samples for corner cases associated with driving environments of autonomous vehicles; (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.) training an artificial Intelligence (AI)/Machine Learning (ML) model using the virtual dataset; (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.) and updating the AI/ML model … for the corner cases associated with driving environments of autonomous vehicles. (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.) Regarding claim 2 , the rejection of claim 1 is incorporated herein. Further, claim 2 recites the following abstract ideas: wherein generating the virtual dataset comprises separating the initial dataset into content and context. (Separating the initial dataset into content and context is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.) Claim 2 does not recite additional elements. Regarding claim 3 , the rejection of claim 2 is incorporated herein. Further, claim 3 recites the following additional element: wherein the initial dataset comprises initial data samples for the corner cases associated with driving environments of autonomous vehicles. (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.) Regarding claim 4 , the rejection of claim 3 is incorporated herein. Further, claim 4 recites the following additional element: wherein the initial dataset comprises initial data samples for the corner cases associated with driving environments of autonomous vehicles. (This is generally linking to a particular technological environment or field of use. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(h). Therefore, this does not amount to significantly more than the judicial exception.) Regarding claim 5 , the rejection of claim 4 is incorporated herein. Further, claim 5 recites the following abstract ideas: wherein the energetic neural network process comprises synthesizing data samples for the corner cases by recontextualizing the initial data samples in the initial dataset (Synthesizing data by recontextualizing recites mathematical calculation, which is mathematical concept.) Claim 5 further recites the following additional element. … using generative models. (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.) Regarding claim 6 , the rejection of claim 5 is incorporated herein. Further, claim 6 recites the following abstract ideas: wherein the virtual dataset for the corner cases is larger than the initial dataset for the corner cases. (Comparing the size of dataset is merely reciting mathematical relationship, which is mathematical concept.) Claim 6 does not recite additional elements. Regarding claim 7 , the rejection of claim 6 is incorporated herein. Further, claim 7 recites the following abstract ideas: wherein a number of synthesized data samples for the corner cases in the virtual dataset is larger than a number of initial samples for the corner cases in the initial dataset. (Comparing the size of dataset is merely reciting mathematical relationship, which is mathematical concept.) Claim 7 does not recite additional elements. Regarding claim 8 , the rejection of claim 1 is incorporated herein. Further, claim 8 recites the following abstract ideas: wherein the formal verification comprises maintaining functional equivalent behavior between the AL/ML trained using the virtual dataset and previous AI/ML models. (Recites an evaluative comparison of model behavior, which is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper. Defining functional equivalent behavior could also recite mathematical relationship, which is mathematical concept.) Claim 8 does not recite additional elements. Regarding claim 9 , the rejection of claim 8 is incorporated herein. Further, claim 9 recites the following abstract ideas: wherein the formal verification comprises applying energy models to the trained AI/ML model. (Applying energy models as described in specification recites mathematical relationship, which is mathematical concept.) Claim 9 does not recite additional elements. Regarding claim 10 , the rejection of claim 8 is incorporated herein. Further, claim 10 recites the following abstract ideas: wherein the formal verification compensates for noise associated with the virtual dataset and synthesized data samples for the corner cases. (Compensating for noise in AI/ML models recite mathematical calculation, which is mathematical concept.) Claim 10 does not recite additional elements. Regarding claim 11 , the rejection of claim 1 is incorporated herein. Further, claim 11 recites the following additional elements: further comprising communicating the updated AI/ML model to one or more deployed autonomous vehicles. (This is mere transmitting data, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.) Regarding claim 12 , the rejection of claim 11 is incorporated herein. Further, claim 12 recites the following additional elements: wherein the updated AI/ML model modifies an autonomous control of the one or more deployed autonomous vehicles for the corner cases. (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.) Regarding claim 13 , the following claim elements are abstract idea: a controller receiving an updated artificial intelligence (AI)/Machine Learning (ML) model (This is mere data gathering, an insignificant extra solution activity, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.) Claim 13 further recites following additional elements executing an autonomous control of the vehicle based on the updated AI/ML model (This is generally linking to a particular technological environment or field of use. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(h). Therefore, this does not amount to significantly more than the judicial exception.) The rest of claim 13 recites substantially similar subject matter to claim 1 and 12 respectively and is rejected with the same rationale, mutatis mutandis . Regarding claim 14 , the rejection of claim 13 is incorporated herein. Further, claim 14 recites the following abstract ideas: wherein the controller generates the updated AI/ML model (Generating the updated AI/ML model merely recites mathematical calculation, which is mathematical concept.) Claim 14 does not recite additional elements. Claims 15 – 17 recite substantially similar subject matter to claim 1 respectively and are rejected with the same rationale, mutatis mutandis . Regarding claim 18 , the rejection of claim 13 is incorporated herein. Further, claim 18 recites the following additional element: wherein the vehicle comprises an autonomous vehicle. (This is generally linking to a particular technological environment or field of use. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(h). Therefore, this does not amount to significantly more than the judicial exception.) Regarding claim 19 , the rejection of claim 13 is incorporated herein. Further, claim 19 recites the following additional element: wherein the updated AI/ML model is received from a computer system communicatively connected to the vehicle. (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.) Regarding claim 20, the following are additional elements: one or more processors; and a memory having instructions stored thereon, which when executed by the one or more processors cause the processors to perform (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.) The rest of claim 20 recites substantially similar subject matter to claim 1 respectively and is rejected with the same rationale, mutatis mutandis . Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. 07-21-aia AIA Claim s 1 , 8, 10 – 20 are rejected under 35 U.S.C. 103 as being unpatentable over Iandola et al. (U.S. Pub. 2018/0275658 A1) in view of Sanchez (U.S. Pub. 2019/0317510 A1), further in view of Levandowski et al. (U.S. Pub. 2020/0192373 A1) . Regarding Claim 1 , Iandola teaches generating a virtual dataset from an initial dataset, wherein the virtual dataset comprises synthesized data samples ([0050] of Iandola states “The data synthesizing module 325 generates synthetic data that represent sensor data of simulated environments from the perspective of sensors of the autonomous control system 110. The data synthesizing module 325 may generate synthetic sensor data in response to one or more requests from modules of the model training system 140. Specifically, the data synthesizing module 325 generates synthetic data by applying modifications to sensor data, or by simulating the sensor data itself to capture how sensor signals will interact with the environment given conditions of the simulated environments depending on the request.”) training an artificial Intelligence (AI)/Machine Learning (ML) model using the virtual dataset; ([0007] of Iandola states “The autonomous control systems use the synthetic data to train computer models for various detection and control algorithms. In general, this allows autonomous control systems to augment training data to improve performance of computer models, simulate scenarios that are not included in existing training data, and/or train computer models that remove unwanted effects or occlusions from sensor data of the environment”) However, Iandola does not teach for corner cases associated with driving environments of autonomous vehicles; performing formal verification of the AI/ML model; and updating the AI/ML model to execute predictive analysis for the corner cases associated with driving environments of autonomous vehicles. Sanchez teaches that for corner cases associated with driving environments of autonomous vehicles; ([0027] of Sanchez ”Simulation environments enable testing of corner cases and extreme situations in a safe way (e.g., without putting human lives in danger). Furthermore, a verification process for the autonomous agents may be executed in parallel by running copies of a simulator in multiple distributed nodes.” [0072] of Sanchez states “The method 900 may also include optimizing the controller module of the autonomous vehicle prior to controlling the behavior of the autonomous vehicle. For example, the controller module 640 may be reconfigured to improve, for example, operation of the autonomous vehicle during real-world corner cases or atypical situations similar to those presented by the simulated sensor information data.”) performing formal verification of the AI/ML model; ([0031] of Sanchez states “A system and method for full-stack verification of autonomous agents, according to aspects of the present disclosure, trains a convolutional neural network to learn a noise model associated with ground truth 3D sensor image data. That is, an object detection stack of an autonomous agent system is generally a consumer of sensor image data. “ [0071] of Sanchez states ”Referring again to FIG. 9, in block 906, full-stack verification of the autonomous agent system including the trained neural network to apply the surrogate function in response to the ground truth information is performed.”) and updating the AI/ML model … for the corner cases associated with driving environments of autonomous vehicles. ([0072] of Sanchez states ”The method 900 may further include optimizing the planner module of the autonomous vehicle prior to controlling the behavior of the autonomous vehicle. For example, the planner module 630 may be reconfigured to improve, for example, simulated operation of the autonomous vehicle during corner cases or atypical situations presented by the simulated sensor information data. The method 900 may also include optimizing the controller module of the autonomous vehicle prior to controlling the behavior of the autonomous vehicle. For example, the controller module 640 may be reconfigured to improve, for example, operation of the autonomous vehicle during real-world corner cases or atypical situations similar to those presented by the simulated sensor information data.”) Levandowski teaches that … execute predictive analysis ([0123] of Levandowski states “FIG. 5 depicts additional details of the one or more autonomous vehicle models that may reside at the fleet vehicle, database 444, and/or be generated by the autonomous vehicle model generation system 416. The model may include a machine learning algorithm 530 which takes data, trains a model on the data, and uses the trained model to make predictions on new data. At each step, the model 530 makes predictions and gets feedback about how accurate its generated predictions were.” [0209] of Levandowski states “Alternatively, or in addition, statistically accuracy may include a measure of an output, such as a steering adjustment, throttle adjustment, and/or braking adjustment compared to a correct output, where a correct output may be from another vehicle, another system, a post processing technique (e.g., image data is processed at a location other than the autonomous vehicle and such processing may provide different steering angle adjustments, throttle adjustments, braking adjustments, than a model executing at the autonomous vehicle in real-time. In some instances, a statistical accuracy may be based on based on at least one of a quantity of course corrections or a quantity of course deviations, where a course correction includes determining that an input associated with a manual override was received. In some instances, a course deviation includes determining that a path traveled by an autonomous vehicle is different from a projected path traveled by the autonomous vehicle.” ML may be updated and used to make predictions, with feedback used to correct prediction errors.) It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Iandola, Sanchez, and Levandowski. Iandola teaches generating synthetic data for autonomous control systems by modifying captured sensor data or simulating sensor data for a virtual environment, and using the synthetic data to train computer models for detection and control algorithms. Sanchez teaches verification of an autonomous agent system including a trained neural network and verification during corner cases or atypical situations. Levandowski teaches updating an autonomous vehicle model and providing the updated model to an autonomous vehicle or fleet vehicle for use in autonomous driving control. One with ordinary skill in the art would have been motivated to incorporate the teachings of Sanchez, Levandowski into that of Iandola as all references address complementary parts of the same autonomous vehicle model framework. The combination would have been predictable to allow the trained autonomous vehicle AI/ML model to be verified for corner case and updated to vehicles for accurate autonomous control. Regarding Claim 8 , the rejection of claim 1 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, and Levandowski teaches wherein the formal verification comprises maintaining functional equivalent behavior between the AL/ML trained using the virtual dataset and previous AI/ML models. ([0063] of Sanchez states ”In this example, a given module Mi (e.g., a sensor module Mi) is used to learn a surrogate function φ θ that does not depend on the sensory raw data oi. That is, the function φ is trained to mimic the signature (behavior) of M starting from an ideal estimation (ground truth). In other words, the function φ learns the noise model associated to Mi. This is of special interest because such a function mimics the behavior of Mi without generating sensory data.” [0102] of Levandowski states “In at least some configurations, the application server 122 can provide models to the vehicles 104 and/or receive image data to update the models.” [0214] of Levandowski states ”In addition, a number of course deviations and/or corrections may be utilized to determine if a new autonomous vehicle model 452 is needed and/or should be updated. In accordance with embodiments of the present disclosure, a first course of action, such as providing a steering adjustment, may be determined at step 2024. If an actual steering adjustment is different from the suggested steering adjustment, for example at step 2028, a new autonomous vehicle model may be requested and/or received at step 2020. The method may end at step 2040.”) Regarding Claim 10 , the rejection of claim 8 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, and Levandowski teaches wherein the formal verification compensates for noise associated with the virtual dataset and synthesized data samples for the corner cases. ([0033] of Sanchez states “Aspects of the present disclosure are related to a machine learning system and method for estimating a noise model associated with ground truth 3D sensory image data. The estimated noise model is combined with the ground truth 3D sensory image data to represent system simulated sensor information data.” [0062] of Sanchez states “At block 708, a surrogate function Φ is learned based on a set of outputs of the detection module Mi along with an expected response and a vector of measurements. For example, the set of outputs of the detection module is: M(oi)=ŷi, (e.g., a noisy output ŷi) along with the expected response yi (e.g., ground truth), and the vector of measurements mi are used to learn a surrogate function φθ (yi,mi), where θ represents a set of learned parameters that define the behavior of φ. This process may be carried out using minimization by standard optimization techniques based on gradient decent and risk minimization methods. Here, £ is an appropriate loss function, such as cross-entropy or lp-norm (see Equation (1)). In this aspect of the present disclosure, the purpose of φ is to learn the noise model associated with the output of Mi starting at an ideal prediction (ground truth).” Sanchez teaches learning and applying a noise model within formal verification of the AV system.) Regarding Claim 11 , the rejection of claim 1 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, and Levandowski teaches further comprising communicating the updated AI/ML model to one or more deployed autonomous vehicles. ([0064] of Levandowski states “The computing environment 200 may include one or more vehicle sensors and systems 208, computing system (processor) 204, interface 228, vehicle control systems (e.g., steering system 236, braking system 240, acceleration system 244, environmental control 248, infotainment system 252, etc.), a communication system to the vehicle control systems (e.g., a control area network (CAN) bus 232, a navigation system 212, one or more data stores (e.g., user information 224, model(s) 220, image data 216, etc.). These associated components may be electrically and/or communicatively coupled to one another via at least one bus or other interconnection. In some configurations, the one or more associated components may send and/or receive signals across a communication network to a separate entity, for example, server 122.” [0102] of Levandowski states “In at least some configurations, the application server 122 can provide models to the vehicles 104 and/or receive image data to update the models.” [0118] of Levandowski states “Accordingly, user of a fleet vehicle 404 and/or 408, an autonomous vehicle model manager, and/or an autonomous vehicle model custodian may interact with one or more of the interfaces 424A-424D, to provide and receive images, updated models, new models and the like.”) Regarding Claim 12 , the rejection of claim 11 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, and Levandowski teaches wherein the updated AI/ML model modifies an autonomous control of the one or more deployed autonomous vehicles for the corner cases. ([0121] of Levandowski states “As an example of an autonomous vehicle model that may be generated by the model manager 428 and/or reside at a fleet vehicle 404 and/or 408, the autonomous vehicle model 452 may include one or more image processing portions 456, convolutional neural network portions 460, and/or the vehicle adjustment portions 464. Accordingly, and as one example in accordance with embodiments of the present disclosure, the autonomous vehicle model 452 may be implemented at the model verification module 440 and/or the fleet vehicle. Thus, the autonomous vehicle model 452 may receive one or more images 412, process the images at the image processing portion 456, perform feature extraction and classification at the convolutional neural network 460, and adjust one or more of a steering angle and/or velocity utilizing the steering angle adjust 468 and velocity adjust 472 of the vehicle adjust portion 464.” [0208] of Levandowski states “FIG. 19A depicts a method utilized to adjust one or more parameters of an autonomous vehicle based on an autonomous vehicle model in accordance with embodiments of the present disclosure. As depicted in FIG. 19A, the method 1900 may begin at step 1904 where the method proceeds to receive an image from an image acquisition device at step 1908. At step 1912, the image may be preprocessed, as previously described, and one or more features may be extracted and classified at step 1916. At step 1920, features may be classified from the first image using the autonomous vehicle model and any adjustment may be recommended using the same autonomous vehicle model. At step 1928, the autonomous vehicle model may provide the recommendation to the steering angle adjust 468 and/or the velocity adjust 472 where a vehicle adjustment may be made. The method 1900 may then end at step 1932.” [0213] of Levandowski states “In accordance with some examples, an autonomous vehicle model 452 or a portion of the autonomous vehicle model 452 may be updated. For example, a specific node (for example 304C) associated with identifying one or more lane markers may be updated at a first time, while a specific node (for example, 304I/304J) associated with controlling a steering angle of a vehicle may be updated at a second time. Alternatively, or in addition, both of the previously mentioned nodes may be updated at a same time. In some instances, the update may include a parameter, such as a single threshold; in other instances, an entire node may be replaced with a node included in the update.”) Regarding Claim 13 , the combination of Iandola, Sanchez, and Levandowski teaches A controller receiving an updated artificial intelligence (AI)/Machine learning (ML) model ([0089] of Levandowski states ”The communications interface 256 can also include a controller/microprocessor and a memory/storage/cache. The communications interface 256 can interact with the memory/storage/cache which may store information and operations necessary for configuring and transmitting or receiving the information described herein.” Fig. 4 of Levandowski also describes model generation/model verification/ model exchange through a communication network to fleet vehicles.) PNG media_image1.png 1582 1024 media_image1.png Greyscale The rest of claim 13 recites substantially similar subject matter to claim 1 and 12 respectively and is rejected with the same rationale, mutatis mutandis . Regarding Claim 14 , the rejection of claim 13 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, and Levandowski teaches wherein the controller generates the updated AI/ML model (Fig. 4 of Levandowski comprises model generation as part of the AV model architecture. As communication interfaces contain controller, it is used to generate the updated AI/ML model.) Claims 15 – 17 recite substantially similar subject matter to claim 1 respectively and are rejected with the same rationale, mutatis mutandis . Regarding Claim 18 , the rejection of claim 13 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, and Levandowski teaches wherein the vehicle comprises an autonomous vehicle. ([0016] of Iandola states “Though described herein as an autonomous vehicle, the control decisions of the autonomous controls system may provide semi-autonomous control rather than complete control of the vehicle, for example to supplement or override user control, or as primary means of control that can be overridden by a user.” [0007] of Sanchez states “A controller of an autonomous vehicle may be configured to improve the behavior of the autonomous vehicle during real-world extreme situations.” [0002] of Levandowski states “The present disclosure is generally directed to vehicle systems, in particular, toward autonomous vehicles.”) Claims 19 – 20 recite substantially similar subject matter to claims 11, 1 respectively and are rejected with the same rationale, mutatis mutandis . 07-21-aia AIA Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Iandola et al. (U.S. Pub. 2018/0275658 A1) in view of Sanchez (U.S. Pub. 2019/0317510 A1), Levandowski et al. (U.S. Pub. 2020/0192373 A1), further in view of Keser et al. (NPL: ”Content Disentanglement for Semantically Consistent Synthetic-to-Real Domain Adaptation”) . Regarding Claim 2 , the rejection of claim 1 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, and Levandowski does not explicitly teach wherein generating the virtual dataset comprises separating the initial dataset into content and context. However, Keser teaches that wherein generating the virtual dataset comprises separating the initial dataset into content and context. (Pg. 1 abstract of Kaser states “Synthetic data generation is an appealing approach to generate novel traffic scenarios in autonomous driving. However, deep learning perception algorithms trained solely on synthetic data encounter serious performance drops when they are tested on real data. Such performance drops are commonly attributed to the domain gap between real and synthetic data... Our method performs content disentanglement by employing shared content encoder and fixed style code.” Applying Kaser’s content disentanglement within synthetic data generation framework would have been obvious choice while dealing with autonomous vehicle.) It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Keser into the combination of Iandola, Sanchez, and Levandowski. Iandola teaches generating synthetic data for autonomous control systems by modifying captured sensor data or simulating sensor data for a virtual environment, and using the synthetic data to train computer models for detection and control algorithms. Sanchez teaches verification of an autonomous agent system including a trained neural network and verification during corner cases or atypical situations. Levandowski teaches updating an autonomous vehicle model and providing the updated model to an autonomous vehicle or fleet vehicle for use in autonomous driving control. Keser teaches autonomous driving synthetic to real domain adaptation, where the method performs content disentanglement using a shared content encoder and fixed style code. One with ordinary skill in the art would have been motivated to incorporate the teachings of Keser into the combination of Sanchez, Levandowski, Iandola to improve generating useful AV training data for scenarios that are difficult to collect in the real world. The combination would have been predictable to improve robustness and usefulness of the synthetic dataset by preserving driving scene content while varying contextual appearance and produce recontextualized training samples . 07-21-aia AIA Claim s 3 – 7 are rejected under 35 U.S.C. 103 as being unpatentable over Iandola et al. (U.S. Pub. 2018/0275658 A1) in view of Sanchez (U.S. Pub. 2019/0317510 A1), Levandowski et al. (U.S. Pub. 2020/0192373 A1), Keser et al. (NPL: ”Content Disentanglement for Semantically Consistent Synthetic-to-Real Domain Adaptation”), further in view of Nie et al. (NPL: ”Controllable and Compositional Generation with Latent-Space Energy-Based Models”) . Regarding Claim 3 , the rejection of claim 2 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, Levandowski, and Keser teaches wherein separating the initial dataset into content and context comprises (Pg. 1 abstract of Kaser states “Synthetic data generation is an appealing approach to generate novel traffic scenarios in autonomous driving. However, deep learning perception algorithms trained solely on synthetic data encounter serious performance drops when they are tested on real data. Such performance drops are commonly attributed to the domain gap between real and synthetic data... Our method performs content disentanglement by employing shared content encoder and fixed style code.” Applying Kaser’s content disentanglement within synthetic data generation framework would have been obvious choice while dealing with autonomous vehicle.) However, the combination does not teach applying an energetic neural network process. Nie teaches that applying an energetic neural network process. (Pg. 1 Abstract section of Nie states “Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications, but it still remains as a great challenge. In particular, the compositional ability to generate novel concept combinations is out of reach for most current models. In this work, we use energy based models (EBMs) to handle compositional generation over a set of attributes. To make them scalable to high-resolution image generation, we introduce an EBM in the latent space of a pre-trained generative model such as StyleGAN.” Pg. 2 Introduction section of Nie states “An appealing solution to the compositionality problem is to use energy-based models (EBMs) for controllable generation [12, 18, 13]. This is due to the fact that energy functions representing different semantics can be combined together to form compositional image generators… In this paper, we leverage the compositionality of EBMs and the generative power of state-of-the-art pre-trained models such as StyleGANs [28, 29] to achieve high-quality controllable and compositional generation. Particularly, we propose an EBM that addresses the problem of controlling an existing generative model instead of generating images directly with EBMs or improving the sampling quality.”) It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Nie into the combination of Iandola, Sanchez, Levandowski, and Keser. Iandola teaches generating synthetic data for autonomous control systems by modifying captured sensor data or simulating sensor data for a virtual environment, and using the synthetic data to train computer models for detection and control algorithms. Sanchez teaches verification of an autonomous agent system including a trained neural network and verification during corner cases or atypical situations. Levandowski teaches updating an autonomous vehicle model and providing the updated model to an autonomous vehicle or fleet vehicle for use in autonomous driving control. Keser teaches autonomous driving synthetic to real domain adaptation, where the method performs content disentanglement using a shared content encoder and fixed style code. Nie teaches using energy based models in the latent space of a pre-trained generative model for compositional generation over attributes and teaches that energy functions representing different semantics can be combined to form compositional image generators. One with ordinary skill in the art would have been motivated to incorporate the teachings of Nie into the combination of Sanchez, Levandowski, Iandola, Keser to improve controlling semantic attributes of generated images in a generative model latent space. The combination would have been predictable to apply energy based neural network process to content/context separation to provide energy based control over content and context/style attributes used to synthesize improved AV data samples quality. Regarding Claim 4 , the rejection of claim 3 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, Levandowski, Keser, and Nie teaches wherein the initial dataset comprises initial data samples for the corner cases associated with driving environments of autonomous vehicles. ([0052] of Sanchez states “This verification process may involve creating stress situations to test the behavior of agents in corner cases or atypical situations. Given the risk of these tests for human lives, such tests are usually performed in simulation environments. That is, simulation enables testing of autonomous agent systems for corner cases and extreme situations in a safe way, without putting human lives in danger, for example, as shown in FIGS. 4A and 4B.” [0068] of Sanchez states “For these cases, the following procedure is illustrative. A method 800 begins at block 802, in which a ground truth set {yi}M used to train Mi (in the real world) is considered. At block 804, a scene is set in a simulator S* according to the values of {yi}M. At block 806, an associated set {mi} is generated from the simulator S*. At block 808, sensory data ôi is rendered in the simulator S* from the generated scene” [0006] of Iandola states ”The system generates synthetic data by introducing one or more simulated modifications to sensor data captured by the sensors or by simulating the sensor data for a virtual environment. The sensors can be passive sensors that include a receiver that detects and measures various forms of energy that are naturally emitted from the physical environment, or active sensors that emit energy and then measure the energy that is reflected back to one or more receivers in the sensor.” It would have been obvious to use captured samples associated with atypical driving situations as initial corner case samples.) Regarding Claim 5 , the rejection of claim 4 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, Levandowski, Keser, and Nie teaches wherein the energetic neural network process comprises synthesizing data samples for the corner cases by recontextualizing the initial data samples in the initial dataset using generative models. (Pg. 3 III Approach section of Keser states “We also introduce several constraints to accurately reconstruct or translate the image to the target domain with the decoder. The first constraint is enabled by accurate reconstruction of the input image. The encoder embeds the input image into content code; then, the decoder combines one domain’s embedding with its style codes. Here, the network minimizes the reconstruction loss, which is defined as follows: …. Lastly, adversarial learning is applied to match the distribution of translated images to the target domain distribution. The generated images in the cross-domain should be indistinguishable from real images in the target domain. In our work, the discriminator only receives the random patches p of source and translated images [17]. Computation of reconstruction, cycle reconstruction and adversarial losses are depicted Figure 2.” Pg. 4 A. Network Architecture section of Keser states “Our network consists of one generator (encoder and decoder) and two discriminators. Encoder and decoder architectures are based on MUNIT [8] implementation and discriminator architecture is based on PatchGAN [9] implementation.” The domain adaptation network of Keser takes the content representation of an initial synthetic sample and generates a new image in the real domain by applying a different style/context code, which is the recontextualization of initial data samples using a generative model.) Regarding Claim 6 , the rejection of claim 5 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, Levandowski, Keser, and Nie teaches wherein the virtual dataset for the corner cases is larger than the initial dataset for the corner cases. ([0007] of Iandola states “The autonomous control systems use the synthetic data to train computer models for various detection and control algorithms. In general, this allows autonomous control systems to augment training data to improve performance of computer models, simulate scenarios that are not included in existing training data, and/or train computer models that remove unwanted effects or occlusions from sensor data of the environment” [0029] of Iandola states ”The model training system 140 may use the synthetic sensor data to augment existing training data and generally improve the accuracy of computer models. By training the computer models on augmented training data, the computer models can perform with improved accuracy when they are applied to sensor data from a physical sensor operating in an environment having the same type of artifacts that were included in the simulated data.” It would have been obvious that generating additional synthesized samples for scenarios absent from existing data results in a virtual dataset with a greater number of samples than the initial dataset, as augmentation is the stated purpose of the system.) Regarding Claim 7 , the rejection of claim 6 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, Levandowski, Keser, and Nie teaches wherein a number of synthesized data samples for the corner cases in the virtual dataset is larger than a number of initial samples for the corner cases in the initial dataset. ([0007] of Iandola states “The autonomous control systems use the synthetic data to train computer models for various detection and control algorithms. In general, this allows autonomous control systems to augment training data to improve performance of computer models, simulate scenarios that are not included in existing training data, and/or train computer models that remove unwanted effects or occlusions from sensor data of the environment” [0029] of Iandola states ”The model training system 140 may use the synthetic sensor data to augment existing training data and generally improve the accuracy of computer models. By training the computer models on augmented training data, the computer models can perform with improved accuracy when they are applied to sensor data from a physical sensor operating in an environment having the same type of artifacts that were included in the simulated data.” It would have been obvious that generating additional synthesized samples for scenarios absent from existing data results in a virtual dataset with a greater number of samples than the initial dataset, as augmentation is the stated purpose of the system. [0072] of Sanchez states “For example, the controller module 640 may be reconfigured to improve, for example, operation of the autonomous vehicle during real-world corner cases or atypical situations similar to those presented by the simulated sensor information data.” Iandola’s synthesized data regarding situation could be corner cases described in Sanchez.) 07-21-aia AIA Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Iandola et al. (U.S. Pub. 2018/0275658 A1) in view of Sanchez (U.S. Pub. 2019/0317510 A1), Levandowski et al. (U.S. Pub. 2020/0192373 A1), further in view of Vernaza et al. (U.S. Pub. 2019/0095731 A1) . Regarding Claim 9 , the rejection of claim 8 is incorporated herein. Furthermore, the combination of Iandola, Sanchez, and Levandowski teaches wherein the formal verification… ([0005] of Sanchez states “A method for full-stack verification of autonomous agents may include training a neural network to learn a noise model associated with an object detection module of an autonomous agent system of an autonomous vehicle.”) However, the combination does not explicitly teach comprises applying energy models to the trained AI/ML model Vernaza teaches that comprises applying energy models to the trained AI/ML model ([0004] of Vernaza states “The system includes an imaging device that captures images of a vehicle in traffic. A processing device including an energy-based model is stored in a memory of the processing device to stochastically model future behavior of the vehicle. The energy-based model includes a generator that produces a distribution of possible future trajectories of the vehicle, an energy model that samples the distribution of possible future trajectories according to an energy value of each trajectory in the distribution of possible future trajectories to determine probable future trajectories, and an optimizer that optimizes parameters of each of the generator and the energy model.” [0016] of Vernaza states “To more accurately predict all possible future trajectories, the model predicts uncertain goal-directed behavior according to uncertainty using a conditional probability model. As the probability model is represented in an unnormalized form, exact inference is intractable. Hence, a generator is trained to produce samples from the learned probability model. The generator is represented as a sequence of invertible policy evaluations, which allows efficient evaluation of the generator and its probability density. To prevent overfitting of data, the conditional probability model is represented as a structured Gibbs distribution (or a structured energy function). Thus, the log-probability of a putative behavior is proportional to a sum of factors, including, e.g., an obstacle collision cost score and an acceleration penalty, among other factors.”) It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Vernaza into the combination of Iandola, Sanchez, and Levandowski. Iandola teaches generating synthetic data for autonomous control systems by modifying captured sensor data or simulating sensor data for a virtual environment, and using the synthetic data to train computer models for detection and control algorithms. Sanchez teaches verification of an autonomous agent system including a trained neural network and verification during corner cases or atypical situations. Levandowski teaches updating an autonomous vehicle model and providing the updated model to an autonomous vehicle or fleet vehicle for use in autonomous driving control. Vernaza teaches an AV domain energy-based model for predicting future vehicle behavior, including an energy model that samples possible future vehicle trajectories according to energy values to determine probable future trajectories. One with ordinary skill in the art would have been motivated to incorporate the teachings of Vernaza into the combination of Sanchez, Levandowski, Iandola to provide known AV domain technique using energy values on AV behavior verification system of Sanchez. The combination would have been predictable to allow the trained AV model’s predicted vehicle behavior to be evaluated using energy values during verification before the updated model is provided to the vehicle. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to BYUNGKWON HAN whose telephone number is (571)272-5294. The examiner can normally be reached M-F: 9:00AM-6PM PST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /BYUNGKWON HAN/ Examiner, Art Unit 2121 /Li B. Zhen/ Supervisory Patent Examiner, Art Unit 2121 Application/Control Number: 18/465,935 Page 2 Art Unit: 2121 Application/Control Number: 18/465,935 Page 3 Art Unit: 2121 Application/Control Number: 18/465,935 Page 4 Art Unit: 2121 Application/Control Number: 18/465,935 Page 5 Art Unit: 2121 Application/Control Number: 18/465,935 Page 6 Art Unit: 2121 Application/Control Number: 18/465,935 Page 7 Art Unit: 2121 Application/Control Number: 18/465,935 Page 8 Art Unit: 2121 Application/Control Number: 18/465,935 Page 9 Art Unit: 2121 Application/Control Number: 18/465,935 Page 10 Art Unit: 2121 Application/Control Number: 18/465,935 Page 11 Art Unit: 2121 Application/Control Number: 18/465,935 Page 12 Art Unit: 2121 Application/Control Number: 18/465,935 Page 13 Art Unit: 2121 Application/Control Number: 18/465,935 Page 14 Art Unit: 2121 Application/Control Number: 18/465,935 Page 15 Art Unit: 2121 Application/Control Number: 18/465,935 Page 16 Art Unit: 2121 Application/Control Number: 18/465,935 Page 17 Art Unit: 2121 Application/Control Number: 18/465,935 Page 18 Art Unit: 2121 Application/Control Number: 18/465,935 Page 19 Art Unit: 2121 Application/Control Number: 18/465,935 Page 20 Art Unit: 2121 Application/Control Number: 18/465,935 Page 21 Art Unit: 2121 Application/Control Number: 18/465,935 Page 22 Art Unit: 2121 Application/Control Number: 18/465,935 Page 23 Art Unit: 2121 Application/Control Number: 18/465,935 Page 24 Art Unit: 2121 Application/Control Number: 18/465,935 Page 25 Art Unit: 2121 Application/Control Number: 18/465,935 Page 26 Art Unit: 2121 Application/Control Number: 18/465,935 Page 27 Art Unit: 2121 Application/Control Number: 18/465,935 Page 28 Art Unit: 2121 Application/Control Number: 18/465,935 Page 29 Art Unit: 2121
Prosecution Insights