DETAILED ACTION
Claims 1-20 are presented for examination.
Claims 1, 9, and 17 have been amended.
This office action is in response to the amendment submitted on 15-JUN-2026.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments – 35 USC 103
Applicant’s arguments with respect to the 103 rejections have been considered, but are moot in view of the new ground(s) of rejection provided below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-12, and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Gaidon et al. (US20200134379A1) in view of Hillemacher et al. (Model-Based Development of Self-Adaptive Autonomous Vehicles using the SMARDT Methodology) and further in view of Hong (US20220194429A1) and further in view of Ulbrich (Towards Tactical Lane Change Behavior Planning for Automated Vehicles)
Regarding Claim 1, Gaidon teaches a method for agent and scenario modeling to operate an ego vehicle, comprising: analyzing, events extracted from driving log data to identify a dataset of interest from the driving log data ([0003] “The method can include automatically labeling the one or more unlabeled real-world driving logs to generate one or more labeled real-world driving logs. The automatic labeling can include analysis-by-synthesis on the one or more unlabeled real-world driving logs to generate one or more simulated driving logs.” The labeled driving log dataset is the dataset of interest).
extracting an episode of interest from the dataset of interest ([0034] “the auto-labeling module(s) 150 can query the source of unlabeled driving logs for unlabeled driving logs 120 that satisfy one or more desired conditions or parameters, such as a specific file format. The auto-labeling module(s) 150 can be configured to initiate the query automatically and/or responsive to receiving an input from a user (e.g., a person).” Querying according to certain conditions extracts data based on the conditions of interest).
generating a driving scenario of interest based on the episode of interest ([0037] “The simulated driving logs can contain photo-realistic renderings of the observed real-world scenario along with ground truth labels computed programmatically.” The rendering is a generation of a driving scenario).
However, Gaidon doesn’t appear to explicitly teach
through a model-based systems engineering (MBSE) model, high-level
according to an MBSE state transition diagram;
wherein the driving scenario of interest comprises a lane change scenario defined by the MBSE state transition diagram, the lane change scenario comprising a plurality of state transitions including at least a start monitoring target lane state, a waiting for entry space state, an intent pre-indication state, a pre-cross maneuver state, and a crossing state
utilizing the driving scenario of interest to parameterize agent and/or scenario models used for autonomous operation of an ego vehicle.
autonomously operating the ego vehicle utilizing a parameterized agent and scenarios model from the utilizing
identifying a merge gap between a first vehicle and a second vehicle in a target lane computing positions Sil(t) and S2(t) for the first vehicle and the second vehicle respectively along an S-axis indicating position along the target lane
incorporating a predetermined amount of padding distance to account for a safe driving distance and a length of the ego vehicle into the computing of positions S1(t) and S2(t).
Hillemacher teaches through a model-based systems engineering (MBSE) model, high-level (Page 6, Deriving Test Cases from Activity Diagrams, “Besides providing high level specifications, introducing formal OCL/P expressions to activity diagrams (ADs) also enables a systematic derivation (Mingsong et al., 2006) of test cases. The output of the derivation process are test cases, which can be used to test the functional specifications modeled by the ADs.” and Page 3, SMARDT Methodology, “the SMARDT approach uses only a strict and formalized subset of SysML diagrams.” Hillemacher uses SysML an MBSE language to model high level specifications and derive through those specifications test cases for driving scenarios).
according to an MBSE state transition diagram (Page 5, Example Activity Diagram, “Fig. 4 illustrates a simplified activity diagram (AD) describing the car logic. In general, the structure of the ADs used on the second layer of SMARDT is similar to the SysML standard. Inputs and outputs of the function are modeled using ports. Besides the control flow, the object flow of a diagram explicitly indicates when and where the information is passed. Action nodes are used to model single steps of the function. Control nodes, e.g., decision nodes, model any decision logic and parallelism of a function.” Fig 4 shows the usage of state transition diagrams and performing desired logic according to the diagram).
PNG
media_image1.png
710
693
media_image1.png
Greyscale
utilizing the driving scenario of interest to parameterize agent and/or scenario models used for autonomous operation of the ego vehicle (Fig 6 showcases the parameterization of the agent/scenario models. There is an explicit Parameter Tuner module).
autonomously operating the ego vehicle utilizing a parameterized agent and scenarios model from the utilizing (Pg. 13, “We choose a track with a width of 15 meters and a length of 2057.56 meters. The physics and the engine simulation is called with a frequency of 500 Hertz whereas the autonomous driving components are called with a frequency of 50 Hertz. The car used is a Chevrolet Corvette T-Top”).
PNG
media_image2.png
280
538
media_image2.png
Greyscale
Gaidon and Hillemacher are analogous art because they are from the same field of endeavor in autonomous driving modeling and simulation. Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art, to combine Gaidon and Hillemacher to arrive at more effective and less error prone modeling methodologies as well as benefiting from testing the log parameters on ego vehicles. “The SMARDT approach (Specification Methodology Applicable to Requirements, Design, and Testing) uses only a strict and formalized subset of SysML diagrams so that for each layer test cases can be derived automatically to test whether the developed system satisfies the specification of each layer. This enables higher consistency between different abstraction layers of the V-Model when using an agile development process.” (Hillemacher, Page 1)
Hong teaches identifying a merge gap between a first vehicle and a second vehicle in a target lane computing positions S1(t) and S2(t) for the first vehicle and the second vehicle respectively ([0040], [0046] “s(t, n): Movement distance of surrounding vehicle #n (m)” s(t, 1) corresponds to S1(t) and s(t, 2) corresponds to S2(t). See Fig. 2. [0085] “Additionally, the controller 102 of the simulation apparatus may be configured to calculate a reference event preparation distance of the surrounding vehicles 262, 264, and 266 M(safety, ref) (406). The reference event preparation distance of the surrounding vehicles 262, 264, and 266 M(safety, ref) is, as shown in FIG. 2 described above, may be a distance secured for safety between the autonomous vehicle 200 and the closest surrounding vehicle #1 (262). The calculation of the reference event preparation distance of the surrounding vehicles 262, 264, and 266 M(safety, ref) may be performed with reference to precise map data. The reference event preparation distance of the surrounding vehicles 262, 264, and 266 M(safety, ref) may be calculated through Equation 1 below. In Equation 1, V(ego, test) is the test speed of the autonomous vehicle, a(ego, test) is the test acceleration of the autonomous vehicle, and K(safety) is an event preparation distance coefficient of the surrounding vehicle.” [0086 and 0089] EN: further show the various calculations for the safe merge gap distance between the surrounding vehicles. [0094] based on the safe distance calculation shows the possible lane change decision making process.
along an S-axis indicating position along the target lane ([0040] “The terms and parameters used in the simulation environment of the autonomous vehicle 200 according to the embodiment of the present disclosure as shown in FIG. 2 may be defined as follows. S-d coordinate system: In the simulation as shown in FIG. 2, a position and a movement distance are indicated through the s-d coordinate system. The S-d coordinate system is a coordinate system of a movement distance (s) in a traveling direction (a longitudinal direction)—a lateral distance (d) based on the leftmost lane of the vehicle's traveling direction on a precise map used for simulation. The origin of the s-axis of the s-d coordinate system is a starting point of traveling for the simulation.”)
incorporating a predetermined amount of padding distance to account for a safe driving distance and a length of the ego vehicle into the computing of positions S1(t) and S2(t) ([0009] “The setting of the condition for the event to be performed by the surrounding vehicle may include at least one of a safe distance between the autonomous vehicle and the surrounding vehicle; a relative speed of the surrounding vehicle relative to the autonomous vehicle; a type of the event; a degree of risk of the event; an acceleration of the surrounding vehicle when the event is performed; and a lane change time of the surrounding vehicle when the event is performed” [0078] “L(ego): Length of autonomous vehicle (m)”) [0109] “The degree of risk E(risk, ego) may be calculated using Equation 3 below. In Equation 3, s(ego) is the distance traveled by the autonomous vehicle 200 on the road, L(ego) is the length of the autonomous vehicle 200, s(t, n) is the movement distance of the surrounding vehicle #n, V(ego) is the speed of the autonomous vehicle 200, V(t, n) is the speed of the surrounding vehicle #n, and E(risk, end) is the event risk setting range of the surrounding vehicle #n (end).” Also see [0105] and Fig. 2)
Gaidon, Hillemacher, and Hong are analogous art because they are from the same field of endeavor in autonomous driving modeling and simulation. Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art, to combine Gaidon, Hillemacher and Hong to incorporate Hong’s explicit calculation paradigm for ego vehicles including length and safe following distance.
Ulbrich teaches wherein the driving scenario of interest comprises a lane change scenario defined by the (Fig. 5 shows the lane change state transition diagram. Hillemacher is relied upon for the formalized MBSE state transition diagram as shown in Hillemacher Fig. 4 above)
the lane change scenario comprising a plurality of state transitions including at least a start monitoring target lane state (Pg. 991, Fig. 4, and “For planning lane changes, the four high-level hidden state variables are, whether a lane change is possible and beneficial to the left and right respectively. To calculate state estimates for those abstract hidden state random variables, several other underlying random variables need to be calculated: 1) Lane Change Possible Estimation: To estimate if a lane
change is possible, we have to consider if it is possible due to the dynamic traffic situation, due to the infrastructure, due to ability induced skill restrictions and due to the system’s current skill-level induced skill restrictions. 2) Lane Change Beneficial Estimation: To estimate if a lane change is beneficial, it we need to evaluate the dynamic traffic situation for relative velocity gains on neighbor lanes and if a lane change is beneficial due to infrastructure related information.” Pg. 992, “The set of actions U contains the 13 discrete action alternatives of DoLc, FinishLc, PrepareLc, IndicateLc and AbortLc to the left and right and action alternatives for NormalDriving, AbortLcIndication and AbortLcPreparation. Furthermore it entails a targetGapIndex and continuous dimensions for gradual deviations like a longitudinal delta target pose and sampling variation as well as a commanded velocity deviation from the current ego velocity.”
a waiting for entry space state (Pg. 991, Fig. 4, and “3) Gap Quality Assessment: High traffic densities necessitate to adjust the automated vehicle towards a cost-optimal gap. Therefore, the most appropriate gap for a lane change is determined. 4) Calculating and Propagating Uncertainties: Among the key challenges for tactical lane change behavior planning is the inherent uncertainty from any kind of environment perception modules. State estimates from perception modules come along with an uncertainty already. Based on this, hidden state variable estimates are calculated by the dynamic Bayesian network.”)
an intent pre-indication state (Pg. 992 and Fig. 5, IndicateLC state)
a pre-cross maneuver state (Pg. 992 and Fig. 5, PrepareLC state)
a crossing state (Pg. 992 and Fig. 5, DoLc state)
PNG
media_image3.png
500
725
media_image3.png
Greyscale
Gaidon, Hillemacher, Hong and Ulbrich are analogous art because they are from the same field of endeavor in autonomous driving modeling and simulation. Ulbrich teaches the known technique of lane change states in planning and executing lane changes. Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art, to combine Gaidon, Hillemacher, Hong and Ulbrich to incorporate Ulbrich formalized lane change states in the environment and systems provided by Gaidon, Hillemacher and Hong to provide better formalized vocabulary for describing the various states involved in executing lane changes.
Regarding Claim 2, Gaidon in view of Hillemacher, further in view of Hong, and further in view of Ulbrich teaches the method of claim 1. Gaidon further teaches extracting comprises parsing field logs, simulation logs, and/or closed course logs to identify the dataset of interest, including corresponding sequences of events ([0038] “The unsupervised domain adaptation algorithm can use the simulated labeled data (“(xs, ys, zs)” extracted by a “TLog ETL” program that can Extract, Transform, and Load this information from the simulated logs) and unlabeled data from the real-world logs (“(x)”).” ETL is an extraction and parsing process).
Regarding Claim 3, Gaidon in view of Hillemacher, further in view of Hong, and further in view of Ulbrich teaches the method of claim 1. Hillemacher further teaches generating an MBSE flowchart defining a behavior of a predetermined driving situation using the MBSE state transition diagram (Fig 4 above showcases the car logic modeled by an activity diagram including transitions using SysML, an MBSE language).
detecting the episode of interest from the dataset of interest according to the MBSE generated flowchart (Fig 4 above showcases MBSE-based detection logic for example lap start, and lap finish. The dataset of interest was covered in claim 1 above).
modeling the scenario of interest according to the MBSE generated flowchart (Fig 4 showcases a scenario of interest modeled through MBSE flowcharts. Page 2, Col 2, “an algorithm which is capable of creating a realization model from the concrete technical concept by binding all configuration parameters. An evolutionary algorithm is used.” The technical concept is represented by the activity diagrams that map to the MBSE generated flowchart).
Regarding Claim 4, Gaidon in view of Hillemacher, further in view of Hong, and further in view of Ulbrich teaches the method of claim 1. Gaidon further teaches generating a plurality of different driving scenarios of interest based on the episode of interest ([0037] “the analysis-by-synthesis module(s) 152 can generate a set of diverse simulated logs. The simulated driving logs can contain photo-realistic renderings of the observed real-world scenario along with ground truth labels computed programmatically. The analysis-by-synthesis module(s) 152 can use the simulator and computer vision and computer graphics techniques together to reconstruct the recorded driving scenes, or relevant parts thereof.” The analysis-by-synthesis module generates scenarios from records of interest).
training a machine learning model according to the plurality of different driving scenarios of interest ([0039] “The resulting labels from each of the aforementioned techniques can be fused and subsequently used for training.” This is also showcased in Fig 7).
PNG
media_image4.png
509
322
media_image4.png
Greyscale
Regarding Claim 7, Gaidon in view of Hillemacher, further in view of Hong, and further in view of Ulbrich teaches the method of claim 1. Gaidon further teaches generating a plurality of different driving scenarios of interest based on the episode of interest ( This limitation was addressed in Claim 4 above).
verifying coverage of a machine learning model according to the plurality of different driving scenarios of interest (Fig 7 showcases the validation module, and [0049] “a portion of the real-world driving logs can be held out for validating and testing the performance of the unsupervised trained machine learning model. One example of a manual quality assurance process is shown in FIG. 9. The labels for these validation and test sets can be acquired in any suitable manner”).
Regarding Claim 8, Gaidon in view of Hillemacher, further in view of Hong, and further in view of Ulbrich teaches the method of claim 1. Gaidon further teaches generating a plurality of different driving scenarios of interest based on the episode of interest ( This limitation was addressed in Claim 4 above).
generating a simulation for an autonomous vehicle according to the plurality of different driving scenarios of interest ([0037] “the analysis-by-synthesis module(s) 152 can generate a set of diverse simulated logs. The simulated driving logs can contain photo-realistic renderings of the observed real-world scenario along with ground truth labels computed programmatically. The analysis-by-synthesis module(s) 152 can use the simulator and computer vision and computer graphics techniques together to reconstruct the recorded driving scenes, or relevant parts thereof. The ground truth labels computed by the simulator”).
Regarding Claim 9, Gaidon teaches a non-transitory computer-readable medium having program code recorded thereon for agent and scenario modeling, the program code being executed by a processor ([0053] “arrangements described herein may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied or embedded, e.g., stored, thereon. Any combination of one or more computer-readable media may be utilized” and [0004] “The system can include one or more processors. The one or more processors can be programmed to initiate executable operations”).
The remaining limitations are similar to claim 1 taught by Gaidon in view of Hillemacher, and are rejected under the same rationale.
Claims 10-12, 15, and 16 are medium claims reciting limitations similar to claims 2-4, 7 and 8 respectively and are rejected under the same rationale.
Claims 17 is a system claim reciting limitations similar to claims 1 and is rejected under the same rationale.
Regarding Claim 18, Gaidon in view of Hillemacher, further in view of Hong, and further in view of Ulbrich teaches the system of claim 17. Gaidon further teaches a transform module to transform sensor data and the driving log data into high-level events (Page 2, Running Example, “Ideally, the software developer would not need to know the technical manufacturer specific details of the sensors and actuators installed in the vehicle but would rather get a homogeneous access to all available sensor and actuator data via a common interface.” And page 3, SMARDT Methodology, “the SMARDT approach uses only a strict and formalized subset of SysML diagrams (OMG, 2015) with a meaningful and clear semantics (Harel and Rumpe, 2004) to specify the functionality of a system. As a consequence, a higher consistency between different abstraction layers of the V-Model is achieved.” The SMARDT methodology is built around abstracting the data leading to a ‘higher-level’ view of events).
Claims 19, and 20 are system claims reciting limitations similar to claims 2 and 7 and 8 respectively and are rejected under the same rationale.
Claims 5, 6, 13, and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Gaidon et al. (US20200134379A1) in view of Hillemacher et al. (Model-Based Development of Self-Adaptive Autonomous Vehicles using the SMARDT Methodology), further in view of Hong (US20220194429A1) further in view of Ulbrich (Towards Tactical Lane Change Behavior Planning for Automated Vehicles) and further in view of Fremont et al. (Scenic: A Language for Scenario Specification and Scene Generation).
Regarding Claim 5, Gaidon in view of Hillemacher, further in view of Hong, and further in view of Ulbrich teaches the method of claim 1. Fremont teaches training the machine learning agent model according to the plurality of different driving scenarios of interest (Page 73, Col 2, Par 3, “Our experiments used squeezeDet [48], a convolutional neural network real-time object detector for autonomous driving. We used a batch size of 20 and trained all models for 10,000 iterations unless otherwise noted”).
Gaidon, Hillemacher, Hong, Ulbrich, and Fremont are analogous art because they are from the same field of endeavor in autonomous driving modeling and simulation. Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art, to combine Gaidon, Hillemacher, Hong, Ulbrich and Fremont to arrive at more effective training of the machine learning models. “This demonstrates how Scenic can be used to improve performance by generalizing individual failures into scenarios that capture the essence of the problem but are broad enough to prevent overfitting during retraining.” (Fremont, Page 75)
Regarding Claim 6, Gaidon in view of Hillemacher, further in view of Hong, and further in view of Ulbrich teaches the method of claim 1. Fremont further teaches training rule-based agents according to the plurality of different driving scenarios of interest (Page 75, Debugging Failures, “The original failure can then be generalized to a broader scenario describing a class of inputs on which the model misbehaves, which can in turn be used for retraining. We selected one scene from our first experiment…We wrote several scenarios which left most of the features of the scene fixed but allowed others to vary. Specifically, scenario (1) varied the model and color of the car, (2) left the position and orientation of the car relative to the camera fixed but varied the absolute position, effectively changing the background of the scene, and (3) used the mutation feature of Scenic to add a small amount of noise to the car’s position, heading, and color.” This demonstrates rule based training, where certain rules were implemented to produce better training sets).
Claims 13 and 14 recite limitations similar to claim 5 and 6 respectively and are rejected under the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
D’Ambrosio et al (An MBSE Approach for Development of Resilient Automated Automotive Systems): Discloses MBSE in the context of automated vehicles.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMIR DARWISH whose telephone number is (571)272-4779. The examiner can normally be reached 7:30-5:30 M-Thurs.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lewis Bullock can be reached on 571-272-3759. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.E.D./Examiner, Art Unit 2199
/LEWIS A BULLOCK JR/Supervisory Patent Examiner, Art Unit 2199
Prosecution Insights