DETAILED ACTION
This action is in response to the submission filed on 12/21/2022. Claims 1-20 are presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to because Figures 6-16, 18-21, and 23 are blurry and contains text which is difficult or impossible to read. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Applicant is directed towards 37 CFR § 1.84 - Standards for drawings:
(3) Numbers, letters, and reference characters must measure at least .32 cm. ( 1/8 inch) in height. They should not be placed in the drawing so as to interfere with its comprehension. Therefore, they should not cross or mingle with the lines. They should not be placed upon hatched or shaded surfaces. When necessary, such as indicating a surface or cross section, a reference character may be underlined and a blank space may be left in the hatching or shading where the character occurs so that it appears distinct.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites “providing the three dimensional multi-sensor data of the scenario to an autonomous driving system to simulate behavior of the autonomous driving system when faced with the scenario”. However, this phrase is unclear. It is unknown how an autonomous driving system could possibly simulate itself. It is unclear if some other system is simulating the autonomous driving system. Additionally, paragraph [0037] of Applicant’s Publication “may provide identified scenarios 112 to an autonomous driving system (e.g., a neural network system)” which implies that the autonomous driving system is in actuality a neural network system. Any application of prior art is the Examiner’s best interpretation of the claimed subject matter. Claims 2-17 are rejected by virtue of their dependency.
Claims 1 and 18 recite “classifying the time series of two dimensional representations into a sequence of states, wherein the sequence of states associated with the particular drive and the three dimensional multi-sensor data are stored in a computer-readable medium as a scenario” and then “providing the three dimensional multi-sensor data of the scenario”. However, the three-dimensional multi-sensor data was already reduced to a two dimensional sequence of states in a previous limitation. It is unclear how the three dimensional data can be used at all, including being provided to a scenario when it has already been reduced to two dimensions. Claims 2-17 and 19-20 are rejected by virtue of their dependency.
Claim 18 recites “providing the three dimensional multi-sensor data of the scenario to an autonomous driving system to train an artificial intelligence model of the autonomous driving system, wherein three dimensional multi-sensor data of other scenarios that match the state criteria of the query are also provided to the autonomous driving system for training”. Similar to claim 1 above, this phrase is unclear. It is unclear how the data being sent to the autonomous driving system in turn trains a model of the autonomous driving system. It is unclear how the autonomous driving system can perform training by itself. It is unclear if the phrase should be merely ‘providing data of the scenario to train an artificial intelligence model to”. Claims 19-20 are rejected by virtue of their dependency.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. To determine if a claim is directed to patent ineligible subject matter, the Court has guided the Office to apply the Alice/Mayo test, which requires:
1. Determining if the claim falls within a statutory category;
2A. Determining if the claim is directed to a patent ineligible judicial exception consisting of a law of
nature, a natural phenomenon, or abstract idea; and
2B. If the claim is directed to a judicial exception, determining if the claim recites limitations or elements
that amount to significantly more than the judicial exception.(See MPEP 2106).
Step 1: With respect to claims 1-20, applying step 1, the preamble of independent claims 1 and 18 claim a method and a method. As such these claims fall within the statutory categories of process and process.
Step 2A, prong one: In order to apply step 2A, a recitation of claim 1 is copied below. The limitations of the claim that describe an abstract idea are bolded.
A method of simulating operation of an autonomous driving system, comprising:
accessing three dimensional multi-sensor data associated with a plurality of real-world drives in a sensor equipped vehicle;
for a particular drive, reducing the three dimensional multi-sensor data to a time series of two dimensional representations (mental process – observation, evaluation, judgement, opinion);
classifying the time series of two dimensional representations into a sequence of states (mental process – observation, evaluation, judgement, opinion), wherein the sequence of states associated with the particular drive and the three dimensional multi-sensor data are stored in a computer-readable medium as a scenario;
receiving a query that identifies a state criteria;
accessing the scenario based on the sequence of states matching the state criteria of the query;
providing the three dimensional multi-sensor data of the scenario to an autonomous driving system to simulate behavior of the autonomous driving system when faced with the scenario.
The limitations as analyzed include concepts directed to the "mental process" groupings of abstract ideas performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III). The claim involves reducing data and classifying. The steps are simple enough/broadly claimed that they could be performed mentally or with pen and paper. The ‘to simulating behavior’ step is not positively recited (it is saying ‘in order to simulate’ which is intended use), however, if it were positively recited it would constitute a mental process as well. Thus, limitations noted above also fall into the "mental process" groupings of abstract ideas.
Step 2A, prong two: Under step 2A prong two, this judicial exception is not integrated into a practical application because the additional claim limitations outside the abstract idea only present insignificant extra-solution activity and generic computing components. In particular, the claim recites the additional limitations: “accessing three dimensional multi-sensor data associated with…” (insignificant extra-solution activity - mere data gathering/output MPEP 2106.05(g)), “the particular drive and the three dimensional multi-sensor data are stored in a computer-readable medium as a scenario” (insignificant extra-solution activity - mere data gathering/output MPEP 2106.05(g); generic computing components merely carrying out the abstract idea - see MPEP § 2106.05(f) and (b)), “receiving a query that identifies a state criteria” (insignificant extra-solution activity - mere data gathering/output MPEP 2106.05(g)), “accessing the scenario based on the sequence of states matching the state criteria of the query” (insignificant extra-solution activity - mere data gathering/output MPEP 2106.05(g)), “providing the three dimensional multi-sensor data of the scenario to an autonomous driving system…” (insignificant extra-solution activity - mere data gathering/output MPEP 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
Step 2B: Moving on to step 2B of the analysis, the Examiner must consider whether each claim limitation individually or as an ordered combination amounts to significantly more than the abstract idea. This analysis includes determining whether an inventive concept is furnished by an element or a combination of elements that are beyond the judicial exception. For limitations that were categorized as "apply it" or generally linking the use of the abstract idea to a particular technological environment or field of use, the analysis is the same. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional limitations is considered directed towards generic computer components carrying out the abstract idea and data gathering. See MPEP 2106.04(d) referencing MPEP 2106.05(h). Furthermore, as Berkheimer evidence that the claim elements “accessing three dimensional multi-sensor data associated with…”, “the particular drive and the three dimensional multi-sensor data are stored in a computer-readable medium as a scenario”, “receiving a query that identifies a state criteria”, “accessing the scenario based on the sequence of states matching the state criteria of the query”, “providing the three dimensional multi-sensor data of the scenario to an autonomous driving system…” are Well-Understood, Routine, and Conventional, MPEP § 2106.05(d) (II) provides support that mere data collecting is well understood, routine, and conventional: "The courts have recognized the following computer functions as well- understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra- solution activity:
• Receiving or transmitting data over a network, e.g., using the Internet to gather
data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary
computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d
607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image
transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d
1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google,
Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives
and sends information over a network); but see DDR Holdings, LLC v. Hotels.com, L.P.,
773 F.3d 1245, 1258, 113 USPQ2d 1097, 1106 (Fed. Cir. 2014)
• Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP
Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788
F.3d at 1363, 115 USPQ2d at 1092-93
• Presenting offers and gathering statistics, OIP Techs., 788 F.3d at 1362-63, 115
USPQ2d at 1092-93
For the foregoing reasons, claim 1 is directed to an abstract idea without significantly more, and is rejected as not patent eligible under 35 U.S.C. 101. Independent claim 1 is directed to substantially the same subject matter as independent claim 18 and is rejected under similar rationale and further failure to add significantly more. The same conclusion is reached for the dependent claims 2-17 and 19-20.
Claims 2-17 and 19-20 are further directed towards concepts directed to the "mental process" groupings of abstract ideas performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III). The steps are simple enough/broadly claimed that they could be performed mentally or with pen and paper. Thus, limitations noted above also fall into the "mental process" groupings of abstract ideas. This judicial exception is not integrated into a practical application because the additional claim limitations outside the abstract idea only present generic computing components. In particular, the claim recites the additional limitations: “graphical user interface” (generic computing components merely carrying out the abstract idea - see MPEP § 2106.05(f) and (b)). Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional limitations is considered directed towards generic computer components carrying out the abstract idea.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7-8, 10, 15-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over 20210347372 (“Bagschik”) in view of US 20200326717 A1 (“Chen”).
Regarding claim 1, Bagschik teaches:
A method of simulating operation of an autonomous driving system (Bagschik: para [0013]), comprising:
accessing three dimensional multi-sensor data associated with a plurality of real-world drives in a sensor equipped vehicle (Bagschik: para [0017] “utilize sensor data captured by one or more vehicles during operations, to generate a hierarchy of possible arrangements of objects, object states, and parameters associated with the objects with respect to the operating
vehicle”; para [0095], “objects may include obstacles in an environment…Such objects in the environment have a “geometric pose” (which may also be referred to herein as merely “pose”) …Geometric pose may be described in two-dimensions (e.g., using an x-y coordinate system) or three-dimensions (e.g., using an x-y-z or polar coordinate system)…the frame of reference is described with reference to a two- or three-dimensional coordinate system or map that describes the location of objects relative to a vehicle”);
classifying the time series of two dimensional representations into a sequence of states, wherein the sequence of states associated with the particular drive and the three dimensional multi-sensor data are stored in a computer-readable medium as a scenario (Bagschik: para [0009], “FIG. 7 is an example block-diagram illustrating an example architecture associated with generating scenarios”; para [0017], “the scenario generation system, discussed herein, may utilize sensor data captured by one or more vehicles during operations, to generate a hierarchy of possible arrangements of objects, object states, and parameters associated with the objects with respect to the operating vehicle that may be sampled at simulation time to generate a set of scenarios for a given event being tested”);
receiving a query that identifies a state criteria (Bagschik: para [0023], “the scenario generation system determine a criterion set for the simulation. The system may then sample the Gaussian Mixture model (or other statistical model) generated from the parameter sets of occupations matching the criterion set for the simulation based at least in part on a frequency of occurrence of the parameter sets associated with the occupations based on a number of desired scenarios. Once the scenarios are generated, the scenario generation system may fit each scenario to actual map data. For example, if the scenarios include a vehicle stopped at an intersection having two lanes in each direction, a turn lane, and a cross walk, the scenario generation system may identify map locations corresponding to the environmental conditions presented by the scenarios”);
accessing the scenario based on the sequence of states matching the state criteria of the query (Bagschik: para [0030], “For example, the simulation criterion may indicate that the desired scenarios include a region to the front of the vehicle is occupied by a motorized vehicle. In this example, a first occupation type in which the region to the front of the vehicle occupied by the motorized vehicle and the region to the left of the vehicle is occupied by a bicycle and a second occupation type in which the region to the front of the vehicle occupied by the motorized vehicle and the region to the right of the vehicle is occupied by a bicycle may meet the simulation criterion”);
providing the three dimensional multi-sensor data of the scenario to an autonomous driving system to simulate behavior of the autonomous driving system when faced with the scenario (Bagschik: para [0009], “FIG. 7 is an example block-diagram illustrating an example architecture associated with generating scenarios for simulated testing”; para [0013], “ the decisions and reactions of the autonomous vehicles to events and situations that the vehicle may encounter are modeled and simulated using a plurality of scenarios defined and output by a scenario generation system, discussed herein”).
Bagschik does not teach but Chen does teach:
for a particular drive, reducing the three dimensional multi-sensor data to a time series of two dimensional representations (Chen: para [0001], “The LiDAR system analyzes characteristics of the reflected light and develops three-dimensional (3D) point clouds of data that represent 3D shapes of objects in and features of the environment”; para [0007], “the three-dimensional point cloud or to the two-dimensional images generated by transferring the synthetic LiDAR data to a two-dimensional representation”; para [0024], “The system may also receive—such as from data set captured by one or more cameras mounted on a real-world vehicle that includes a real-world LIDAR system—images from a real-world environment (step (132)”, para [0035], “The system converts the synthetic LiDAR point cloud to 2D front view images using equations such as”; para [0020], “a vehicle in a first frame when the vehicle is located a first distance from the observation point”; the frames are the time series of 2d representations);
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Bagschik (directed to simulating an autonomous vehicle) and Chen (directed to converting 3D data to 2D representations) and arrived at simulating an autonomous vehicle with converting 3D data to 2D representations. One of ordinary skill in the art would have been motivated to make such a combination for “correlating synthetic LiDAR data to a real-world domain for use in training an autonomous
vehicle in how to operate in an environment” (Chen: para [0004]).
Regarding claim 2, Bagschik and Chen teach:
The method of claim 1, wherein a two dimensional representation identifies a position of one or more objects relative to the sensor equipped vehicle, wherein the objects include one or more other vehicles, an obstacle, a road sign, a road marking, a stop light, a person, or an animal (Bagschik: para [0057], “At 808, the scenario generation system may determine a set of representative objects for each region based at least in part on a distance between each object in the region and the vehicle…For instance, if a region includes a first pedestrian fifteen meters from the vehicle and a second pedestrian twenty meters from the vehicle, the scenario generation system may assign the first pedestrian as the representative object of the pedestrian type to the region. In another illustrative instance, if a region includes a first pedestrian fifteen meters from the vehicle and a first bicyclist twenty meters from the vehicle, the scenario generation system may assign the first pedestrian as the representative object of the pedestrian type and the first bicyclist as the representative object of the bicyclist type to the region”; para [0058], “a longitudinal position of the representative object, a lateral position of the representative object, a vertical position of the representative object…”).
Regarding claim 3, Bagschik and Chen teach:
The method of claim 1, wherein a particular state of the sequence of states is determined based on a prior state in the sequence of states (Bagschik: para [0084], “The planning system 1026 may be configured to determine a route for the vehicle 1002 to follow to traverse through an environment. For example, the planning system 1026 may determine various routes and paths and various levels of detail based at least in part on the objects detected, the predicted characteristics of the object at future times, and a set of safety requirements corresponding to the current scenario (e.g., combination of objects detected and/or environmental conditions). In some instances, the planning system 1026 may determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location) in order to avoid an object obstructing or blocking a planned path of the vehicle 1002. In some case, a route can be a sequence of waypoints for traveling between the two locations (e.g., the first location and the second location)”).
Regarding claim 4, Bagschik and Chen teach:
The method of claim 3, wherein the particular state is selected from a subset of all possible states, wherein the subset is determined based on the prior state (Bagschik: para [0047], “FIG. 7 is an example block diagram illustrating an example architecture 700 associated with generating scenarios for simulated testing, in accordance with implementations of the disclosure. As discussed above, a translation component 702 may generate occupations 704 using vectorized model data 706 extracted from log data received from one or more vehicles operating on the roadways. The vectorized model data 706 may include parameters such as, but not limited to, representative object type, region, longitudinal velocity of the representative object, lateral velocity of the representative object, longitudinal acceleration of the representative object, lateral acceleration of the representative object, distance from vehicle (e.g., region 2), distance to an intersection, a longitudinal extent of the representative object, a lateral extent of the representative object, a vertical extent of the representative object, a longitudinal position of the representative object, a lateral position of the representative object, a vertical position of the representative object, a yaw of the representative object, a yaw rate of the representative object, a heading or direction of travel of the representative object, a speed of the representative object, and the like. In some cases, the vectorized model data 706 may also include parameters associated with the vehicle being tested. These parameters may include, but are not limited to, current drive mode, current drive state, planned maneuver, total velocity, total acceleration, longitudinal acceleration, lateral acceleration, distance to an intersection, longitudinal acceleration, lateral acceleration, yaw, yaw rate, lane identifier, road identifier, Euclidian position, and the like”).
Regarding claim 5, Bagschik and Chen teach:
The method of claim 4, wherein the subset excludes impossible or unlikely states based on the prior state (Bagschik: para [0028], “. In some cases, the data modeling component 110 may perform filtering to remove or prevent non-relevant or impossible/improbable data (such as log data representing physically impossible parameters) from being incorporated into instances of the occupations 116. For example, the data modeling component 110 may filter data that represents measurements or distances outside of defined threshold or limitations (e.g., removing data representing a vehicle that is 25 meters long)”).
Regarding claim 7, Bagschik and Chen teach:
The method of claim 1, wherein the query further specifies an additional non-state criteria for scenario selection (Bagschik: para [0015], “determine constraints or limitations of autonomous vehicles that may be used … simulations may be used to understand the operational space of an autonomous vehicle in view of surface and/or environmental conditions, faulty components, etc. As a non-limiting example for illustration, use of the simulations may inform a planner system of a vehicle not to exceed a given acceleration or velocity based on a number of objects in the environment and/or presence of precipitation, etc.”).
Regarding claim 8, Bagschik and Chen teach:
The method of claim 7, wherein the non-state criteria comprises one or more of a location, a vehicle velocity, a vehicle acceleration (Bagschik: para [0015], “determine constraints or limitations of autonomous vehicles that may be used … simulations may be used to understand the operational space of an autonomous vehicle in view of surface and/or environmental conditions, faulty components, etc. As a non-limiting example for illustration, use of the simulations may inform a planner system of a vehicle not to exceed a given acceleration or velocity based on a number of objects in the environment and/or presence of precipitation, etc.”), a vehicle velocity pattern, a vehicle velocity trend, a traffic level, a time of day, a length of time, a weather condition, an event indication, a feature active indicator, a lane steering assistance active indicator, an adaptive cruise control indicator, an autonomous driving system active indicator, and a road curvature specification.
Regarding claim 10, Bagschik does not teach but Chen does teach:
The method of claim 1, wherein reducing the three dimensional data comprises consolidating a plurality of points into a location of an object on a two dimensional plane (Chen: para [0001], “The LiDAR system analyzes characteristics of the reflected light and develops three-dimensional (3D) point clouds of data that represent 3D shapes of objects in and features of the environment”; para [0007], “the three-dimensional point cloud or to the two-dimensional images generated by transferring the synthetic LiDAR data to a two-dimensional representation”; para [0024], “The system may also receive—such as from data set captured by one or more cameras mounted on a real-world vehicle that includes a real-world LIDAR system—images from a real-world environment (step (132)”, para [0035], “The system converts the synthetic LiDAR point cloud to 2D front view images using equations such as”).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Bagschik (directed to simulating an autonomous vehicle) and Chen (directed to converting 3D data to 2D representations) and arrived at simulating an autonomous vehicle with converting 3D data to 2D representations. One of ordinary skill in the art would have been motivated to make such a combination for “correlating synthetic LiDAR data to a real-world domain for use in training an autonomous
vehicle in how to operate in an environment” (Chen: para [0004]).
Regarding claim 15, Bagschik and Chen teach:
The method of claim 1, wherein behavior of the autonomous driving system is approved or rejected based on the simulation (Bagschik: para [0046], “As illustrated in FIG. 6, the simulation data 604 may indicate a number of simulations (e.g., simulation 1, simulation 2, etc.) and the result of the simulations (e.g., result 1, result 2). For example, as described above, the result may indicate a pass or a fail based on rules/assertions that were broken/triggered”).
Regarding claim 16, Bagschik and Chen teach:
The method of claim 1, wherein the scenario is accessed based on a contiguous subset of all states associated with the scenario matching the state criteria (Bagschik: para [0030], “For example, the simulation criterion may indicate that the desired scenarios include a region to the front of the vehicle is occupied by a motorized vehicle. In this example, a first occupation type in which the region to the front of the vehicle occupied by the motorized vehicle and the region to the left of the vehicle is occupied by a bicycle and a second occupation type in which the region to the front of the vehicle occupied by the motorized vehicle and the region to the right of the vehicle is occupied by a bicycle may meet the simulation criterion”).
Regarding claim 18, Bagschik teaches:
accessing three dimensional multi-sensor data associated with a plurality of real-world drives in a sensor equipped vehicle (Bagschik: para [0017] “utilize sensor data captured by one or more vehicles during operations, to generate a hierarchy of possible arrangements of objects, object states, and parameters associated with the objects with respect to the operating
vehicle”; para [0095], “objects may include obstacles in an environment…Such objects in the environment have a “geometric pose” (which may also be referred to herein as merely “pose”) …Geometric pose may be described in two-dimensions (e.g., using an x-y coordinate system) or three-dimensions (e.g., using an x-y-z or polar coordinate system)…the frame of reference is described with reference to a two- or three-dimensional coordinate system or map that describes the location of objects relative to a vehicle”);
classifying the time series of two dimensional representations into a sequence of states, wherein the sequence of states associated with the particular drive and the three dimensional multi-sensor data are stored in a computer-readable medium as a scenario (Bagschik: para [0009], “FIG. 7 is an example block-diagram illustrating an example architecture associated with generating scenarios”; para [0017], “the scenario generation system, discussed herein, may utilize sensor data captured by one or more vehicles during operations, to generate a hierarchy of possible arrangements of objects, object states, and parameters associated with the objects with respect to the operating vehicle that may be sampled at simulation time to generate a set of scenarios for a given event being tested”);
receiving a query that identifies a state criteria (Bagschik: para [0023], “the scenario generation system determine a criterion set for the simulation. The system may then sample the Gaussian Mixture model (or other statistical model) generated from the parameter sets of occupations matching the criterion set for the simulation based at least in part on a frequency of occurrence of the parameter sets associated with the occupations based on a number of desired scenarios. Once the scenarios are generated, the scenario generation system may fit each scenario to actual map data. For example, if the scenarios include a vehicle stopped at an intersection having two lanes in each direction, a turn lane, and a cross walk, the scenario generation system may identify map locations corresponding to the environmental conditions presented by the scenarios”);
accessing the scenario based on the sequence of states matching the state criteria of the query (Bagschik: para [0030], “For example, the simulation criterion may indicate that the desired scenarios include a region to the front of the vehicle is occupied by a motorized vehicle. In this example, a first occupation type in which the region to the front of the vehicle occupied by the motorized vehicle and the region to the left of the vehicle is occupied by a bicycle and a second occupation type in which the region to the front of the vehicle occupied by the motorized vehicle and the region to the right of the vehicle is occupied by a bicycle may meet the simulation criterion”);
wherein three dimensional multi-sensor data of other scenarios that match the state criteria of the query are also provided to the autonomous driving system (Bagschik: para [0009], “FIG. 7 is an example block-diagram illustrating an example architecture associated with generating scenarios for simulated testing”; para [0013], “ the decisions and reactions of the autonomous
vehicles to events and situations that the vehicle may encounter are modeled and simulated
using a plurality of scenarios defined and output by a scenario generation system, discussed herein”)
Bagschik does not teach but Chen does teach:
training an autonomous driving system model (Chen: Abstract, “use the trained model of the real-world environment to train an autonomous vehicle.”);
for a particular drive, reducing the three dimensional multi-sensor data to a time series of two dimensional representations (Chen: para [0001], “The LiDAR system analyzes characteristics of the reflected light and develops three-dimensional (3D) point clouds of data that represent 3D shapes of objects in and features of the environment”; para [0007], “the three-dimensional point cloud or to the two-dimensional images generated by transferring the synthetic LiDAR data to a two-dimensional representation”; para [0024], “The system may also receive—such as from data set captured by one or more cameras mounted on a real-world vehicle that includes a real-world LIDAR system—images from a real-world environment (step (132)”, para [0035], “The system converts the synthetic LiDAR point cloud to 2D front view images using equations such as”; para [0020], “a vehicle in a first frame when the vehicle is located a first distance from the observation point”; the frames are the time series of 2d representations);
providing the three dimensional multi-sensor data of the scenario to an autonomous driving system to train an artificial intelligence model of the autonomous driving system (see rejections under 35 USC 112; Chen: [0004], “train a model of a real-world environment, and use the trained model of the real-world environment to train an autonomous vehicle”).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Bagschik (directed to simulating an autonomous vehicle) and Chen (directed to converting 3D data to 2D representations) and arrived at simulating an autonomous vehicle with converting 3D data to 2D representations. One of ordinary skill in the art would have been motivated to make such a combination for “correlating synthetic LiDAR data to a real-world domain for use in training an autonomous
vehicle in how to operate in an environment” (Chen: para [0004]).
Regarding claim 19, Bagschik and Chen teach:
The method of claim 18, wherein a particular state of the sequence of states is determined based on a prior state in the sequence of states (Bagschik: para [0084], “The planning system 1026 may be configured to determine a route for the vehicle 1002 to follow to traverse through an environment. For example, the planning system 1026 may determine various routes and paths and various levels of detail based at least in part on the objects detected, the predicted characteristics of the object at future times, and a set of safety requirements corresponding to the current scenario (e.g., combination of objects detected and/or environmental conditions). In some instances, the planning system 1026 may determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location) in order to avoid an object obstructing or blocking a planned path of the vehicle 1002. In some case, a route can be a sequence of waypoints for traveling between the two locations (e.g., the first location and the second location)”).
Regarding claim 20, Bagschik and Chen teach:
The method of claim 19, wherein the particular state is selected from a subset of all possible states, wherein the subset is determined based on the prior state, wherein the subset excludes impossible or unlikely states based on the prior state (Bagschik: para [0028], “. In some cases, the data modeling component 110 may perform filtering to remove or prevent non-relevant or impossible/improbable data (such as log data representing physically impossible parameters) from being incorporated into instances of the occupations 116. For example, the data modeling component 110 may filter data that represents measurements or distances outside of defined threshold or limitations (e.g., removing data representing a vehicle that is 25 meters long)”).
Allowable Subject Matter
Claims 6, 9, 11-14, and 17 contain allowable subject matter.
The claims will be allowable if the rejections under 35 USC 112 and 101 are overcome.
The independent claims will be in condition for allowance when the allowable dependent claims are incorporated into the independent claims, in addition to overcoming the 112 and 101 rejections.
Bagschik and Chen teach simulating an autonomous vehicle with converting 3D data to 2D representations. However, these references and the remaining prior art of record, alone or in combination, fails to disclose or suggest
(claim 6 )
“wherein the particular state is determined based on detection that a vehicle has intersected with a line painted on a road.
(claim 9)
“wherein the sequence of states is stored in the computer-readable medium as a text-based series of state indicators that is searchable via regular expression search criteria”,
(claim 11)
“wherein reducing the three dimensional data comprises:
identifying missing data associated with one sensor in the three dimensional multi-sensor data; and
using interpolation to determine replacement data for the missing data using data from another sensor in the multi-sensor data or another data source;
wherein the location of the object is determined using the replacement data”,
(claim 13)
“wherein the real-world drives are each associated with a location;
wherein said reducing and classifying are performed using a server that is assigned real- world drives based on the location associated with those real-world drives;
wherein a location associated with a particular real-world drive is temporarily adjusted to avoid overloading over the server”
(claim 17)
“further comprising providing a graphical user interface for identifying the state criteria of the query and one or more additional criteria; and
receiving the state criteria and the one or more additional criteria via the user interface;
wherein the scenario is accessed based on the sequence of states matching the state criteria and the scenario matching the one or more additional criteria received via the user interface”
in combination with the remaining elements and features of the claimed invention. It is for these reasons that the applicant’s invention defines over the prior art of record.
Additional References Cited
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure and are cited in the attached PTOL-892.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NITHYA J. MOLL whose telephone number is (571)270-1003. The examiner can normally be reached Monday-Friday 10am-6pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rehana Perveen can be reached at 571-272-3676. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NITHYA J. MOLL/Primary Examiner, Art Unit 2189
Prosecution Insights