DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Correction Letter
The non-final rejection mailed on 10/23/2025 failed to include the rejection of claim 12. Due to this, a corrected non-final rejection is made here. The period for reply of 3 MONTHS set in said Office action is restarted to begin with the mailing date of this letter. A corrected copy of the last Office action is enclosed.
Status of Claims
Claims 1-32 filed on 12/18/2023 are presently examined.
Claim Objections
Claim 32 is objected to because of the following informalities: “fourth program instructions for execute the trained…” should instead recite “fourth program instructions for executing the trained…” Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-32 are rejected under 35 USC § 101 because the claimed invention is directed to an abstract idea without significantly more.
101 Analysis – Step 1
Claims 1-28 are directed to a method. Claims 29-31 are directed toward a system comprising a processor (i.e. a machine). Claim 32 is directed toward a computer program product comprising a non-transitory computer-readable medium (i.e. a machine). Therefore, claims 1-32 are within at least one of the four statutory categories.
101 Analysis – Step 2A, Prong I
Regarding Prong I of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity, and/or c) mental processes.
Independent claim 32 recites similar limitations as independent claims 1 and 29 and will be used as a representative claim.
Claim 32 is recited below and limitations that recite an abstract idea are emphasized in bolding below:
A software program product for generating a driving assistant model, comprising:
a non-transitory computer readable storage medium;
first program instructions for computing at least one semantic driving scenario by computing at least one permutation of at least one initial semantic driving scenario;
second program instructions for providing the at least one semantic driving scenario to a simulation generator to produce simulated driving data describing at least one simulated driving environment;
third program instructions for training a driving assistant model using the simulated driving data to produce a trained driving assistant model; and
fourth program instructions for execute the trained driving assistant model and at least one autonomous driving model, where the trained driving assistant model provides at least one driving instruction to the at least one autonomous driving model while the at least one autonomous driving model is operating;
wherein the first, second, third and fourth program instructions are executed by at least one computerized processor from the non-transitory computer readable storage medium.
The examiner submits that the above bolded limitation(s) constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitation in the human mind. The bolded limitations in the context of this claim encompasses a person mentally taking note of an initial sematic driving scenario and modifying a characteristic about said scenario, producing simulated driving data describing a simulated environment based on the modified initial semantic driving scenario, training a driving assistance model using the simulated data, and providing one driving instruction to the model while the model is operating. A driving assistant model can be the mental process a driver uses while driving a vehicle, and a person can mentally imagine new scenarios based on their previous driving experience, and anticipate how they would control the vehicle. Accordingly, the claim recites at least one abstract idea.
101 Analysis – Step 2A, Prong II
Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A software program product for generating a driving assistant model, comprising:
a non-transitory computer readable storage medium;
first program instructions for computing at least one semantic driving scenario by computing at least one permutation of at least one initial semantic driving scenario;
second program instructions for providing the at least one semantic driving scenario to a simulation generator to produce simulated driving data describing at least one simulated driving environment;
third program instructions for training a driving assistant model using the simulated driving data to produce a trained driving assistant model; and
fourth program instructions for execute the trained driving assistant model and at least one autonomous driving model, where the trained driving assistant model provides at least one driving instruction to the at least one autonomous driving model while the at least one autonomous driving model is operating;
wherein the first, second, third and fourth program instructions are executed by at least one computerized processor from the non-transitory computer readable storage medium.
For the following reason(s), the examiner submits that the above underlined additional limitations do not integrate the above-noted abstract idea into a practical application.
The examiner submits that these additional limitations merely use a sensors to perform an insignificant extra-solution activity of a computer (processor, generic computer components, non-transitory computer-readable medium storing instructions) to perform otherwise mental judgements is not sufficient to integrate the abstract idea into a practical application.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
101 Analysis – Step 2B
Regarding Step 2B of the 2019 PEG, representative independent claim 32 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of using a processor or generic computer components to gather data and perform the otherwise mental judgements amounts to nothing more than applying the exception using generic computer components. Generally applying an exception using a generic computer component cannot provide an inventive concept. Further the additional limitations are directed toward additional aspects of the judicial exception and/or well-understood, routine and conventional additional elements that do not integrate the judicial exception into a practical application, merely use generic computer components in their ordinary capacity to perform an otherwise mental process or judgement, and do not amount to significantly more. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network).
Dependent claims 2-28 and 30-31 do not recite any further limitations that cause the claims to be patent eligible. Rather, the limitations of dependent claims are directed toward additional aspects of the judicial exception and/or well-understood, routine and conventional additional elements that do not integrate the judicial exception into a practical application, merely use generic computer components in their ordinary capacity to perform an otherwise mental process or judgement or data gathering, and do not amount to significantly more.
Examiner suggests including positive recitation of vehicle control such as supported by Applicant’s specification in [page 25, lines 17-21] “a control circuit operates an actuator of the vehicle. Some examples of an actuator of a vehicle include, but are not limited to, an accelerator, a break, a vehicle light and a horn.” While there may be other ways to recite a positive control limitation, a non-limiting example is reciting explicitly that the model sends a driving instruction to control an actuator of the vehicle while the model and vehicle are operating.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-9, 14-16, 19, 29-32 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Choe et al. (US 20210309248 A1).
Regarding claims 1, 29, and 32, Choe discloses A method for generating a driving assistant model ([0007] systems and methods [0030] “various functions may be carried out by a processor executing instructions stored in memory.” [claims 1 and 16] system and processor performing the invention.), comprising:
computing at least one semantic driving scenario by computing at least one permutation of at least one initial semantic driving scenario ([0031] “a 3D model generator 102 generating multiple representations (e.g., virtual instances) of an object (e.g., road debris) adhering to a variety of criteria in a virtual, simulated environment generated by simulator 104 … The semantic augmentor 110 may augment the images with (e.g., by inserting) one or more generated graphical representations of the object into the loaded ground truth data.”);
providing the at least one semantic driving scenario to a simulation generator to produce simulated driving data describing at least one simulated driving environment ([0036] “simulator 104 may provide a virtual environment to capture data of a virtual vehicle driving in a virtual environment from the perspective of a virtual sensor mounted to the virtual vehicle … The virtual instances of objects generated by the 3D model generator 102 may be inserted into the virtual environment of the simulator 104.”);
training a driving assistant model using the simulated driving data to produce a trained driving assistant model ([0031] “The augmented real-world image may be finalized by dataset generator 114 and used by trainer 116 to train a machine learning model.”); and
providing by the trained driving assistant model at least one driving instruction to at least one autonomous driving model while the at least one autonomous driving model is operating ([0031] “Once trained, the machine learning model may be deployed via deployment module 118 to be used by the vehicle 1100 to detect debris or objects on the driving surface using images and to determine next steps in response to a detected road object. The next steps or operations may include world model management, path planning, control decisions, obstacle avoidance, and/or other operations of an autonomous or semi-autonomous driving software stack.”).
Regarding claim 2, Choe discloses The method of claim 1, wherein at least one of the at least one permutation is computed by providing at least one of the at least one initial semantic driving scenario to a generative machine learning model trained to compute, in response to input comprising a semantic driving scenario, a permutation of the semantic driving scenario ([0045] “semantic augmentor 110 may determine which representations of an object to use to augment a real-world image … may sample (randomly, according to one or more embodiments) criteria or conditions (e.g., location, lighting, orientation, pose, environmental conditions, and/or appearance attributes) and select one or more representations that match, or closely match, the sampled criteria”).
Regarding claim 3, Choe discloses The method of claim 1, further comprising:
in response to the at least one autonomous driving model receiving the at least one driving instruction providing input, by the at least one autonomous driving model, to at least one control circuit of a vehicle ([0031] “the machine learning model may be deployed via deployment module 118 to be used by the vehicle 1100 to detect debris or objects on the driving surface using images and to determine next steps in response to a detected road object. The next steps or operations may include world model management, path planning, control decisions, obstacle avoidance, and/or other operations of an autonomous or semi-autonomous driving software stack.”).
Regarding claim 4, Choe discloses The method of claim 3, wherein the trained driving assistant model is installed in the vehicle ([0186] “Once the machine learning models are trained, the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles over the network(s) 1090.”).
Regarding claims 5 and 30, Choe discloses The method of claim 1, wherein the trained driving assistant model provides the at least one driving instruction to the at least one autonomous driving model via at least one digital communication network ([0090] “The vehicle 1000 further includes a network interface 1024” [0150] “The network interface 1024 may be used to enable wireless connectivity … This functionality may be part of a cooperative adaptive cruise control functionality of the vehicle 1000.” [0186] “Once the machine learning models are trained … the machine learning models may be used by the server(s) 1078 to remotely monitor the vehicles.”).
Regarding claim 6, Choe discloses The method of claim 1, further comprising receiving, by the trained driving assistant from the at least one autonomous driving model, driving data collected while the autonomous driving model is operating ([0129] “The neural network may take as its input at least some subset of parameters, such as bounding box dimensions, ground plane estimate obtained (e.g. from another subsystem), inertial measurement unit (IMU) sensor 1066 output that correlates with the vehicle 1000 orientation, distance, 3D location estimates of the object obtained from the neural network and/or other sensors (e.g., LIDAR sensor(s) 1064 or RADAR sensor(s) 1060), among others.”);
wherein providing the at least one driving instruction to the at least one autonomous driving model is in response to receiving the driving data from the at least one autonomous driving model ([0089] “The controllers(s) 1036 may also receive outputs from machine learning model(s) regarding an action (e.g., drive over or maneuver around the object or come to a complete stop) to implement in response to a detected object on a driving surface.”).
Regarding claim 7, Choe discloses The method of claim 1, further comprising:
accessing driving event data describing at least one driving event detected in other driving data collected during operation of at least one other autonomous driving model (shadow driver) in another vehicle driven by a human driver ([0185] “The server(s) 1078 may receive, over the network(s) 1090 and from the vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work.” [0186] “Once the machine learning models are trained, the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles over the network(s) 1090”); and
computing the at least one initial semantic driving scenario using the driving event data ([0186] “The server(s) 1078 may be used to train machine learning models (e.g., neural networks) based on training data. The training data may be generated by the vehicles, and/or may be generated in a simulation (e.g., using a game engine).”).
Regarding claim 8, Choe discloses The method of claim 7, wherein the driving event data comprises one or more of:
at least one signal, captured while the other vehicle is driven by the human driver, by at least one sensor installed in the other vehicle ([0187] “the server(s) 1078 may receive data from the vehicles” [0185] “The server(s) 1078 may receive, over the network(s) 1090 and from the vehicles, image data ");
captured driving data collected while the other vehicle is driven by the human driver and while the at least one signal is captured ([0129] “The neural network may take as its input … inertial measurement unit (IMU) sensor 1066 output that correlates with the vehicle 1000 orientation [0024] “the systems and methods described herein may be used by non-autonomous vehicles, semi-autonomous vehicles (e.g., in adaptive driver assistance systems (ADAS)”); and
computed driving data computed by the shadow driver, using the at least one signal, while the other vehicle is driven by the human driver ([0185] “The server(s) 1078 may receive, over the network(s) 1090 and from the vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work.” The computed driving data is the semantic understanding of the captured sensor/image data.).
Regarding claim 9, Choe discloses The method of claim 8, wherein the driving event data further comprises annotation data describing one or more relations between the at least one signal, the captured driving data and the computed driving data ([0043] “The ground truth data may be a real-world image with annotation data, such as bounding shapes and labels (e.g., types of objects in the road, hazard rating).” [0129] “The neural network may take as its input … bounding box dimensions … inertial measurement unit (IMU) sensor 1066 output that correlates with the vehicle 1000 orientation … 3D location estimates of the object obtained from the neural network”).
Regarding claim 14, Choe discloses The method of claim 1, wherein the simulated driving data comprises a plurality of synthetic signals, each simulating one of a plurality of signals captured from at least one physical driving environment equivalent to the at least one simulated driving environment by a plurality of sensors mounted on yet another vehicle while traversing the at least one physical driving environment ([0031] “The simulator 104 may include a virtual vehicle with a virtual sensor that records and captures simulation data from the perspective of the virtual sensor as the virtual vehicle drives on a driving surface. In some embodiments, the representations (e.g., virtual instances), as generated by the 3D model generator 102, may then be inserted into a virtual environment of simulator 104 and recorded in images from the perspective of the virtual sensor.” [0034] “create virtual instances that have a corresponding orientation or pose with respect to the angle of the virtual sensor mounted on the virtual car in the simulated environment. In any example, a sensor model corresponding to a real-world sensor of the vehicle 1100 may be used in a virtual sensor in the simulated environment.”).
Regarding claim 15, Choe discloses The method of claim 1, wherein the simulated driving data comprises a ground truth of the at least one simulated driving environment ([0008] “training machine learning models using real-world images augmented with simulated objects and ground truth data”).
Regarding claim 16, Choe discloses The method of claim 8, wherein the at least one sensor comprises at least one of: a camera, an electromagnetic radiation sensor, a microphone, a thermometer, an acceleration sensor, a rolling shutter camera, a velocity sensor, an audio sensor, a radio detection and ranging sensor (radar), a laser imaging, detection, a ranging sensor (LIDAR), an ultrasonic sensor, a thermal sensor, and a far infra-red (FIR) sensor and a video camera ([0088] Choe’s sensors include at least: RADAR, LIDAR, ultrasonic, speed, microphone, infrared camera, regular cameras.).
Regarding claim 19, Choe discloses The method of claim 1, further comprising testing the driving assistant model using the simulated driving data to produce the trained driving assistant model, additionally or alternatively to training the driving assistant model using the simulated driving data ([0025] “a real-world image (and its corresponding ground truth data) may be augmented with the instances of the simulated objects to generate image data to train the machine learning model as well as to generate ground truth data to test the machine learning model.”).
Regarding claim 31, Choe discloses The system of claim 29, wherein the at least one initial semantic driving scenario is computed using driving event data describing at least one driving event detected in other driving data collected during operation of at least one other autonomous driving model (shadow driver) in a vehicle driven by a human driver ([0184] “The system 1076 may include server(s) 1078, network(s) 1090, and vehicles, including the vehicle 1000.” [0185] “The server(s) 1078 may receive, over the network(s) 1090 and from the vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work.”);
wherein at least one sensor is installed in the vehicle in an identified configuration ([0019] “FIG. 10B is an example of camera locations and fields of view for the example autonomous vehicle”);
wherein the at least one autonomous driving model is installed in another vehicle ([0186] “Once the machine learning models are trained, the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles over the network(s) 1090”); and
wherein at least one other sensor is installed in the other vehicle in the identified configuration (vehicle 1000 is the example vehicle for all the vehicles the server receives data from to train the model.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 10-13 are rejected under 35 U.S.C. 103 as being unpatentable over Choe as applied to claim 8 above, and further in view of Choe (2) (US 20220122001 A1), hereinafter referred to as Choe (2).
Regarding claim 10, Choe fails to explicitly disclose The method of claim 8, wherein computing the at least one initial semantic driving scenario is further using at least one object identified in the at least one signal and not identified in the computed driving data wherein the at least one initial semantic driving scenario comprises the at least one object ([0096] “long-range camera(s) 1098 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, especially for objects for which a neural network has not yet been trained.” [0027] “the machine learning model may learn to detect previously unknown objects” Choe suggests the idea that the neural network may not recognize and label unknown objects that cameras alone can detect the presence of.). and
However, Choe (2) teaches computing the at least one initial semantic driving scenario is further using at least one object identified in the at least one signal and not identified in the computed driving data; and wherein the at least one initial semantic driving scenario comprises the at least one object ([0051] “there was a false negative detection on a partially visible vehicle. As a result, multiple similar synthetic images were generated using a game-engine based simulator. Such synthetic data was added to the training dataset, and the new network was trained with other real and synthetic data. The newly trained model was evaluated on an evaluation dataset and the cases of false positive and false negative detections were reported and the cycle was repeated. In another example, an existing DNN trained on real data had an issue in detecting partially-visible trucks. The imitator was therefore caused to create multiple images with partially visible trucks at various places, times of day, and weather conditions for domain randomization. Along with other synthetic data, this data was used to retrain the network.”).
It would have been obvious to one of ordinary skill in the art to modify Choe with Choe (2)’s teaching of detecting false negative object identification by the neural network in driving scenarios where an object was seen in the camera signal, and subsequently training the neural network using a semantic scenario based on the unrecognized object in further simulations. One would be motivated, with reasonable expectation of success, to train based on the unrecognized objects, in order to improve the neural network to an acceptable level at identifying the previously false negative scenarios (Choe (2) [0051] “After adding the synthetic data, the new network could detect partially visible trucks correctly, at least within acceptable levels of performance.”).
Regarding claim 11, Choe discloses The method of claim 10, further comprising identifying the at least one object in the at least one signal ([FIG. 3] real image objects detected in the image signal are 308, 306, 314, 312, 303, and 310. The virtual object for use in simulations is 206.).
Regarding claim 12, Choe fails to explicitly disclose The method of claim 9, wherein the annotation data comprises an indication of at least one object identified in the at least one signal and not identified in the computed driving data. ([0096] “long-range camera(s) 1098 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, especially for objects for which a neural network has not yet been trained.” [0027] “the machine learning model may learn to detect previously unknown objects” Choe suggests the idea that the neural network may not recognize and label unknown objects that cameras alone can detect the presence of.). and
However, Choe (2) teaches the annotation data comprises an indication of at least one object identified in the at least one signal and not identified in the computed driving data. ([0051] “there was a false negative detection on a partially visible vehicle. As a result, multiple similar synthetic images were generated using a game-engine based simulator. Such synthetic data was added to the training dataset, and the new network was trained with other real and synthetic data. The newly trained model was evaluated on an evaluation dataset and the cases of false positive and false negative detections were reported and the cycle was repeated. In another example, an existing DNN trained on real data had an issue in detecting partially-visible trucks. The imitator was therefore caused to create multiple images with partially visible trucks at various places, times of day, and weather conditions for domain randomization. Along with other synthetic data, this data was used to retrain the network.”).
It would have been obvious to one of ordinary skill in the art to modify Choe with Choe (2)’s teaching of detecting false negative object identification by the neural network in driving scenarios where an object was seen in the camera signal, and subsequently training the neural network using a semantic scenario based on the unrecognized object in further simulations. One would be motivated, with reasonable expectation of success, to train based on the unrecognized objects, in order to improve the neural network to an acceptable level at identifying the previously false negative scenarios (Choe (2) [0051] “After adding the synthetic data, the new network could detect partially visible trucks correctly, at least within acceptable levels of performance.”).
Regarding claim 13, Choe discloses The method of claim 10, wherein computing the at least one permutation comprises changing at least one property of the at least one object ([0032] “3D model generator 102 may generate twenty instances of a simulated construction cone, where each representation of the construction cone uses a different material or color tone, providing a large data set of construction cones with which to train machine learning model”).
Claims 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Choe as applied to claim 1 above, and further in view of Bagschik et al. (US 20240202577 A1), hereinafter referred to as Bagschik.
Regarding claims 17, Choe fails to explicitly disclose The method of claim 1, further comprising validating the driving assistant model using the simulated driving data to produce the trained driving assistant model, additionally or alternatively to training the driving assistant model using the simulated driving data.
However, Bagschik teaches validating the driving assistant model using the simulated driving data to produce the trained driving assistant model, additionally or alternatively to training the driving assistant model using the simulated driving data ([0013] “The modified log data may be substantially similar to the original real-world log data, except for the modified parameter(s) … augmenting the real-world training data with additional driving scenarios.” [0018] “The modified log data may be used to train, test, or validate a machine learning model for the autonomous vehicle controller.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Choe with Bagschik’s teaching of explicitly validating and verifying the machine learning model using simulated driving data to produce the trained driving assistant model. One would be motivated, with reasonable expectation of success, to verify and validate in order to ensure the model is capable of successful navigation without safety errors ([0041] “successful navigation may occur if a simulated autonomous vehicle controlled, at least in part, by the machine learning model traverses a simulated environment without occurrence of a safety critical event”).
Regarding claim 18, Choe fails to explicitly state The method of claim 1, further comprising verifying the driving assistant model using the simulated driving data to produce the trained driving assistant model, additionally or alternatively to training the driving assistant model using the simulated driving data .
However, Bagschik teaches verifying the driving assistant model using the simulated driving data to produce the trained driving assistant model, additionally or alternatively to training the driving assistant model using the simulated driving data ([0013] “verify that the resulting machine learned model can safely navigate such scenarios.” [0125] “a metric may provide a measurable verification that the model 812 is able to navigate situations similar to that represented in the first log data … verify that the model has been improved, and can successfully handle a range of similar driving scenarios, rather than being overfit to the specifics of the real-world driving scenario.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Choe with Bagschik’s teaching of explicitly validating and verifying the machine learning model using simulated driving data to produce the trained driving assistant model. One would be motivated, with reasonable expectation of success, to verify and validate in order to ensure the model is capable of successful navigation without safety errors ([0041] “successful navigation may occur if a simulated autonomous vehicle controlled, at least in part, by the machine learning model traverses a simulated environment without occurrence of a safety critical event”).
Claims 20-22, 25, and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Choe as applied to claim 1 above, and further in view of Atsmon et al. (US 20210312244 A1), hereinafter referred to as Atsmon.
Regarding claim 20, Choe discloses The method of claim 1, further comprising: using a generative rendition model to generate at least one digital image according to at least one physical constraint ([0013] “FIG. 5 is an example of a visualization of a real-world image with an augmented simulated object that satisfies threshold distance constraints and an overlap constraint”);
providing the simulated driving data to the driving assistant model for the purpose of one or more of: training the driving assistant model ([0014] “FIG. 6 is an example of a training image including a real-world image with an augmented simulated object that is used to train machine learning models”), verifying the driving assistant model (Choe does not explicitly verify the model, but suggests the idea of verification: [0188] “deep-learning infrastructure of the server(s) 1078 may … verify the health of the … software … in the vehicle 1000 … The deep-learning infrastructure may run its own neural network to identify the objects and compare them with the objects identified by the vehicle 1000 and, if the results do not match and the infrastructure concludes that the AI in the vehicle 1000 is malfunctioning”), testing the driving assistant model ([0025] “the image data to train and test the machine learning model may be a real-world image of a road that includes a synthetic, or simulated, image of an object on the road.”) and validating the driving assistant model (Choe does not explicitly validate the driving assistant model but suggests the idea of validation: [0188] “deep-learning infrastructure of the server(s) 1078 may … evaluate … the health of the … software … in the vehicle 1000 … The deep-learning infrastructure may run its own neural network to identify the objects and compare them with the objects identified by the vehicle 1000 and, if the results do not match and the infrastructure concludes that the AI in the vehicle 1000 is malfunctioning”).
Choe discloses the training of the driving assistant model comprises generating simulated driving data based on real ground-truth driving data, applying constraints to virtual instances of objects in order to train the model on additional scenarios, and also updating the ground truth with simulated data once said simulated data satisfies real-world constraints ([0055] “ground truth data updater 112 may update ground truth data to reflect that existing ground truth data has been augmented with virtual instances of an object. For instance, once a virtual instance inserted into a real-world image satisfies necessary constraints (e.g., distance thresholds and overlap constraints), the ground truth data updater 112 may update the existing ground truth data, as stored for evaluating a machine learning models' effectiveness, to include the inserted virtual instance” rephrased, Choe uses previous simulated data and environments for subsequent simulated driving data and environments. Choe does not train the image generation model, however. Choe trains the driving assistant model. The image generation model is previously trained and merely used in Choe’s application.).
Choe fails to explicitly disclose training a generative rendition model to generate at least one digital image according to at least one physical constraint by providing the generative rendition model with a plurality of training examples, each comprising a plurality of physical constraints of a simulated driving environment and a real digital image corresponding to the plurality of physical constraints, to produce a trained generative rendition model;
wherein producing the simulated driving data comprises computing at least one synthetic digital image using the trained generative rendition model by providing the trained generative rendition model with another plurality of physical constraints of another simulated driving environment.
However, Atsmon teaches training a generative rendition model to generate at least one digital image according to at least one physical constraint by providing the generative rendition model with a plurality of training examples, each comprising a plurality of physical constraints of a simulated driving environment and a real digital image corresponding to the plurality of physical constraints, to produce a trained generative rendition model ([0044] “learning realistic moving object characteristics and realistic movement patterns from real traffic data collected from real traffic environments.” [0042] see description of characteristics. [0095] “the image reconstruction is trained using real images (photos) to learn a distribution of characteristics of a real environment.” [0054] “train an image refining model to produce an output image that is similar to an input synthetic image in its content and similar to a target real environment in its appearance, or style.”); wherein producing the simulated driving data comprises computing at least one synthetic digital image using the trained generative rendition model by providing the trained generative rendition model with another plurality of physical constraints of another simulated driving environment ([0021] “collecting other simulated driving data from the other simulated driving environment; and modifying a plurality of other model parameters of the other simulation generation model to minimize another difference between another simulation statistical fingerprint, computed using the other simulated driving data, and the real statistical fingerprint, computed using the real input data.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Choe with Atsmon’s teaching of training a generative rendition model that generated images using training examples comprising physical constraints of a simulated environment, a real image with physical constraints, as well as producing the simulated driving data based on one synthetic image with physical constraints of another simulated environment. One would be motivated, with reasonable expectation of success, to use this technique in order to reduce the cost of model development (Atsmon [0021] “Using the plurality of environment classes, the plurality of moving agent classes and the plurality of movement pattern classes identified in the real input data to generate more than one training environment reduces cost of development of the model.”).
Regarding claim 21, Choe discloses The method of claim 20, wherein the plurality of physical constraints comprise a plurality of three-dimensional (3D) placements of a plurality of objects in the simulated driving environment ([0033] “the 3D model generator 102 may generate virtual instances of an object at set intervals or increments … the 3D model generator may generate a virtual instance every two feet away from the virtual sensor.” [0035] “Each virtual instance 204, 206, and 208, of the deceased deer has a different position and location”).
Regarding claim 22, Choe discloses The method of claim 21, wherein the other plurality of physical constraints comprises another plurality of 3D placements of another plurality of objects in the other simulated driving environment (Ground truth is updated with augmented virtual instances of objects once they satisfy constraints, see [0055]. [0032] “The virtual objects span … cardboard boxes, rocks, wheels (and wheel parts), wooden pallets, deceased animals, ladders, logs, traffic cones, mattresses, road signs, and other objects.” [0033] “the 3D model generator 102 may generate a set of 3D virtual instances of a mattress, where each instance of the mattress is in a different position and subjected to a different orientation and sun position.” Choe utilizes a plurality of simulated scenarios with different virtual objects, where each object may have its own variation of at least orientation and pose.).
Regarding claim 25, Choe discloses The method of claim 20, wherein the generative rendition model is a previously- trained generative rendition model, trained to generate at least one synthetic digital image in response to data describing an image, the previously-trained generative rendition model trained using a plurality of real digital images ([0027] “a real-world image (and its corresponding ground truth data) may be augmented with the instances of the simulated objects to generate image data to train the machine learning model” the image generator is trained previously, and is trained on real images.).
Regarding claim 28, Choe discloses The method of claim 20, wherein generating the simulated driving data comprises providing the trained generative rendition model with at least one environment-characteristic adjustment value ([0033] “environmental conditions imposed on virtual instances of a virtual object may include varying times of day, positions relative to the sun, weather conditions (e.g., rain, fog, cloudy), visibility distances, and/or different distances from the virtual sensor in the virtual environment for domain randomization.”).
Claims 23, 24, and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Choe in view of Atsmon as applied to claim 20 above, and further in view of Deng et al. ("A Declarative Metamorphic Testing Framework for Autonomous Driving," in IEEE Transactions on Software Engineering), hereinafter referred to as Deng.
Regarding claims 23, 24, and 26, the use of natural language for physical constraints also results in the natural language describing the image itself, since the constraints describe at least parameters of the objects that are modified and visible in the generated synthetic images. Claim 23 will be used as a representative claim.
Choe fails to disclose The method of claim 20, wherein the plurality of physical constraints comprises text in at least one natural language.
However, Deng teaches the plurality of physical constraints comprises text in at least one natural language ([table 3] [page 2, left column, second paragraph] “RMT allows testers to define and create testing rules in natural language. Testers can create their testing rules by referring traffic rules or other domain knowledge. Then RMT leverages a NLP-based semantic parser to extract grammar dependency predicates, identify elements to change in a driving scene by matching extracted predicates with a predefined ontology list, identify transformation to be applied using predicate translation, and create corresponding MR of the testing rule. RMT then generates new road images based on the ontology elements and transformations and then validates the correctness of model predictions based on the MR. Since driving scenarios based on testing rules require sophisticated image transformations such as adding or removing objects in an image, RMT makes use of several advanced computer vision techniques such as image semantic manipulation [24] and image-to-image translation [25] to support these transformations”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Choe with Deng’s teaching of using natural language input to control the physical constraints of objects in scenes and generating new images based on the natural language describing the object parameters. One would be motivated, with reasonable expectation of success, to use natural language in order to support diverse and complicated testing scenarios (Deng [page 4, left column, first paragraph] “The declarative approach of MR generation based on testing rules in natural language can support diverse and complicated testing scenarios in autonomous driving.”).
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Choe in view of Atsmon as applied to claim 20 above, and further in view of Zhong et al. (Guided Conditional Diffusion for Traffic Simulation; arXiv:2210.17366), hereinafter referred to as Zhong.
Regarding claim 27, Choe fails to explicitly disclose The method of claim 20, wherein the generative rendition model is a latent diffusion deep neural network.
However, Zhong teaches the generative rendition model is a latent diffusion deep neural network ([abstract] “a conditional diffusion model for controllable traffic generation (CTG)” [page 2, right column, under B. Diffusion Modeling] “Controllable diffusion models have been explored with classifier [9], classifier-free [28], and reconstruction [29] guidance for image and video generation.” [page 4, right column, after section “Simulating traffic.”] “To perform closed-loop traffic simulation of a scene with many agents, the same model is used for each agent in a standard control loop”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Choe’s image generation model with Zhong’s teaching of an explicit diffusion image generation model used for traffic simulation. One would be motivated with reasonable expectation of success to use a diffusion image generation model for traffic simulation in order to allow users to control the properties of agents in a scenario (Zhong [abstract] “allows users to control desired properties of trajectories at test time (e.g., reach a goal or follow a speed limit) while maintaining realism and physical feasibility through enforced dynamics. The key technical idea is to leverage recent advances from diffusion modeling and differentiable logic to guide generated trajectories to meet rules”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARK R HEIM whose telephone number is (571)270-0120. The examiner can normally be reached M-F 9-6 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fadey Jabr can be reached on 571-272-1516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.R.H./Examiner, Art Unit 3668
/Fadey S. Jabr/Supervisory Patent Examiner, Art Unit 3668
Prosecution Insights