DETAILED ACTION
Claims 1-23 are presented for examination. Claims 1 and 19 stand currently amended.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 17 March 2026 has been entered.
Response to Arguments
Applicant's remarks filed 17 March 2026 have been fully considered and Examiner’s response is as follows:
Applicant remarks page 10 argues:
Examiner reiterates allegation that of Iandola on column 12 lines 49-56 and column 15 lines 4-12 discloses the feature recited in representative Claim 1 of:
"converting said sensory ranging data simulation to an enhanced semantic-data dataset emulating said geographical area by enhancing said plurality of scene objects comprising: adjusting object location coordinates based on said created sensory ranging data; and adapting values of said semantically described parameters, based on said created sensory ranging data."
However, Iandola in those passages merely describes generation of simulated data with certain properties and is completely silent of semantic-data datasets manipulation of any sort.
To vit, in one of these cited passages, Iandola contemplates the modifying of simulated sensory data to include artifacts
This argument is unpersuasive.
Applicant’s argument is based on an implied assertion that “semantic-data datasets manipulation” is required by this claim language. Considering the cited claim language, the language states “converting said sensory ranging data." This is clearly a manipulation of the sensory data, not the semantic-data. The cited claim language continues “adjusting object location … based on said created sensory ranging data.” This again does not recite any explicit manipulation of semantic-data.
Lastly, the claim recites “adapting values of said semantically described parameters, based on said created sensory ranging data.” According to grammar, the thing being adapted is the “parameters” which were semantically described. This is a manipulation of semantic data. Similarly, the artifacts generated by Iandola are a part of the enhanced semantic-data dataset. Thus, adapting the respective sensory data of the artifacts is adapting respective parameters for the larger enhanced semantic data dataset which is created by the converting step.
Applicant remarks pages 11-12 further argues:
In contrast, the claimed subject matter can be figuratively summarized as:
semantic-data dataset + simulated sensory data ➔ enhanced semantic-data dataset
….
This assertion ignores however the crucial distinction that, whereas Iandola is focused on the simulated sensory data and its modification, that is, Iandola's endeavors remain at the sensory space alone, the aforesaid claimed feature in contrast expressly relates to enhancement of the semantic-data dataset itself.
As discussed immediately above, adapting the enhanced semantic-data dataset parameters is an adapting of said semantically described parameters. The enhanced semantic data dataset includes the simulated sensory data. Thus, adapting of the sensory data is an adapting of at least this subset of the semantic-data dataset parameters.
Furthermore, claim 1 clause 1 recites “structured semantic-data dataset representing a plurality of scene objects.” Thus, adapting the scene objects is also an adapting of the semantic-data dataset parameters because the represented scene objects are within the broadest reasonable interpretation of “one or more of the values of said semantically described parameters” as claimed in the adapting clause. That is, the objects represented by the semantic data are parameters of said data.
Applicant remarks page 12-13 further argues:
color is not a semantically described parameter necessarily.
Claim 1 recites “said semantically described parameters are indicative of at least one of color, size, shape, text on signboards and states of traffic lights.” Therefore, the semantically described parameters encompass at least the expressly enumerated parameter of “color.”
Furthermore, in particular with “color” Examiner’s rejection is based upon a combination of references and further cites Pahud column 1 lines 59-60 and column 2 lines 13-20. Merely pointing out that Iandola’s pixel values do not teach a semantically described color is insufficient to traverse Examiner’s rejection which cites Pahud in regard to color.
Applicant remarks page 14 further argues:
Additionally, this reasoning too repeats the same fallacy noted herein of failing to acknowledge the essential difference between modifying sensory data and modifying semantic data, which is in the core of the claimed subject matter at hand, as discussed in detail herein.
This argument is unpersuasive. Applicant’s claim expressly combines sensory data and semantic-data to create the “enhanced structured semantic-data dataset” and now critiques Examiner’s rejection because the adapting clause claim mapping discusses the sensory data aspect of the enhanced semantic-data dataset rather than loosely defined other semantic-data. However, there is no clearly distinct other semantic-data as these are data which are combined to form the larger dataset. Applicant’s argument is arguing semantics rather than any clearly identifiable feature or characteristic of the respective datasets which separates them from the claim mapping.
Claim 1 fifth clause literally recites “said plurality of scene objects in said structured semantic-data dataset.” Thus, the objects are explicitly encompassed by the structured semantic-data dataset. Yet, Applicant’s argument requires the sensory data of those objects to somehow not be considered part of said semantic-data dataset such that the object’s sensory data is not “semantically described parameters stored in said discrete object records.” Examiner is not persuaded.
Applicant remarks page 15-16 further argues:
Applicant respectfully submits that the Examiner's rebuttal is legally deficient and factually incorrect under both prongs of the analogous art test established by the Federal Circuit. The argument below addresses each element of the Examiner's position in tum and demonstrates that Pahud fails both the field-of-endeavor test and the reasonable-pertinence test, and therefore cannot properly serve as a basis for an obviousness rejection under 35 U.S.C. § 103.
III. PAHUD FAILS THE FIELD-OF-ENDEAVOR PRONG
A. The Field of Endeavor of the Claimed Subject Matter
The field of endeavor of the claimed subject matter at hand is autonomous vehicle simulation and sensor data modeling for training and validation of autonomous driving systems. ….
The claimed method creates sensor-noise-aware semantic datasets for use by a host vehicle simulation engine that executes an autonomous driving system.
B. Field of Endeavor of Pahud
…. Pahud's own abstract states: ''A system and method for generating a scene or animation based at least in part upon text entered in a natural language form."
This argument is unpersuasive.
Examiner’s rejection has clearly established Pahud is “analogous art of simulating using text-based input.” This is a reasonable synopsis of at least the cited statement from Pahud’s abstract. Pahud’s field of endeavor is within the same field of endeavor as the claimed subject matter.
Examiner asserts the field of endeavor of the instant invention can reasonably be described as simulating realistic 3D virtual world for autonomous vehicle testing using semantic parameters. This interpretation is supported by the entire Specification, but also notably, claims 1 and 19 preamble recites “creating data for a host vehicle simulation.” Claim 1 third clause recites “creating virtual three dimensional (3D) visual realistic scene.” Furthermore, Examiner’s asserted field endeavor for the instant invention is consistent with Applicant’s own characterization cited above.
Examiner’s rejection has clearly established Pahud is “analogous art of simulating using text-based input.” This is within the same field of endeavor as the claimed subject matter of “simulating … using semantic parameters”.
Applicant remarks page 18 further argues:
First, the Examiner's characterization conflates the claim language ("semantically described parameters") with the field of endeavor of the reference. … The field of endeavor is determined by the technology being practiced, not by the vocabulary used to describe a single claim element. In re Bigio, 381 F.3d at 1325-26 (field of endeavor is assessed based on the overall context of the invention, not individual claim terms).
Examiner’s assessment of both the instant invention and Pahud is not based upon solely individual words being present but an overall assessment of both. As discussed above, Examiner is unpersuaded that the technology of either is wholly separate from each other.
Applicant remarks page 18 further argues:
Second, the Examiner's characterization of Pahud as being in the field of "simulating using text-based input" is an artificially narrow and misleading description of Pahud's actual field. Pahud does not simulate anything in the technical sense relevant to autonomous vehicle development.
This argument is unpersuasive. The statement “simulating using text-based input” does not assert that Pahud is relevant for autonomous vehicle development. Nor does establishing that Pahud and the instant invention are within the same field of endeavor require such a finding. Merely identifying some differences between Pahud and the instant invention does not conclusively establish that they are not within the same field of endeavor. Both the similarities and differences in structure and function should be considered. See MPEP §2141.01(a)(II).
Applicant remarks page 19 further argues:
IV. PAHUD FAILS THE REASONABLE-PERTINENCE PRONG
….
MPEP 2141.01(a)(I) states “Note that ‘same field of endeavor’ and ‘reasonably pertinent’ are two separate tests for establishing analogous art.” Examiner does not assert or rely upon Pahud being “reasonably pertinent” although such a conclusion is not disavowed either. Examiner has established Pahud is within the same field of endeavor (see above) and that is sufficient for making the combination of references.
Applicant remarks page 22 further argues:
"The broadest reasonable interpretation (see MPEP §2111) of 'semantic-data dataset' is textual data. In this context semantic data is data in text form, also known as a natural language description. A further example is that an adjective is semantic data because adjectives are words (semantic) which provide a description (data)." (Office Action, p. 7)
This interpretation is technically incorrect and unsupported by any authority. The Examiner provides no citation - not a dictionary definition, not a technical standard, not a treatise - for the proposition that "semantic data" means "natural language text" or "textual data." This is a conclusory assertion made without factual or authoritative basis.
Applicant does not provide any dictionary definition either. Applicant Remarks page 22 cites Specification paragraphs 48, 57, and figure 6 OSI. The originally filed Specification has page numbers and line numbers without any paragraph numbering. It is unclear which paragraphs are 48 and 57 respectively. However, nowhere does the Specification provide any semblance of a definition for “semantic-data.” Regarding figure 6, nothing within the Specification identifies that the OSI of figure 6 is semantic-data, let alone establishes any definition for “semantic-data.” Accordingly, claim interpretation according to MPEP §2111 is proper as no lexicographic definition is found within the Specification.
Oxford English Dictionary includes the definition: Semantic, adjective “2.a. “Of or relating to (the study of) meaning in language.” “Semantic, Adj., Sense 2.a.” Oxford English Dictionary, Oxford UP, (March 2026) available from <https://doi.org/10.1093/OED/1039377210>. Accordingly, “semantic data” would be data of or relating to meaning in language. Examiner notes natural language text data is data which relates a meaning in language. Therefore, the broadest reasonable interpretation of “semantic data” encompasses at least text data such as character strings and other computerized text data.
Examiner further notes that text data is broader than merely natural language text.
Applicant remarks page 23 further argues:
The Examiner's BRI of "semantic-data dataset" as "natural language text" or "textual data" is therefore not only unsupported but is directly contradicted by the specification and by the well-established technical meaning of the term in the relevant field. Such interpretation is thus not inline with MPEP § 2111.
Examiner’s broad interpretation as text data is consistent with the Specification. For example, the Specification, and claims 1 and 19, recite “said semantically described parameters are indicative of at least one of color, size, shape, text on signboards and states of traffic lights.” Color, size, and shape are capable of being described by text. Similarly, text on a signboard is capable of being described by text. The states of traffic lights are capable of being described by text, e.g. {red, yellow, green, flashing, malfunctioning, out} are textual descriptors of the state of a traffic light. Accordingly, Examiner’s interpretation of “semantic-data” is consistent with its usage within the Specification.
Applicant remarks page 23-24 further argues:
3. The Examiner's "Both Are Virtual Creations" Rationale Is Legally Insufficient
….
This argument is unpersuasive. Applicant’s argument fails to establish any legal basis upon which Examiner’s statement is either deficient or required to establish a rejection under §103. Without any legal basis, it is irrelevant whether the analysis within the section is true or not. Regarding field of endeavor, Examiner has already discussed this issue above.
Merely establishing that Applicant would analyze or interpret Pahud differently in some fashion is not a traversal of Examiner’s rejection.
Applicant remarks page 24-25 further argues:
4. The Examiner's Stated Motivation to Combine Is Irrational
The Examiner states that the motivation to combine Pahud with Iandola and Ratrout is:
"One having ordinary skill in the art would have found motivation to use [sic] using natural language text input to create a simulated scene into the system of simulating a vehicle environment for the advantageous purpose of easily create and view an animated scene." (Office Action, p. 6)
This stated motivation is irrational and lacks the "articulated reasoning with some rational underpinning" required by KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398, 418 (2007). The claimed invention is a technical data pipeline used by autonomous vehicle engineers and simulation systems - it is not a consumer-facing tool for "easily creating and viewing animated scenes."
This argument is unpersuasive.
MPEP §2144(II) states:
The strongest rationale for combining references is a recognition, expressly or impliedly in the prior art or drawn from a convincing line of reasoning based on established scientific principles or legal precedent, that some advantage or expected beneficial result would have been produced by their combination. In re Sernaker, 702 F.2d 989, 994-95, 217 USPQ 1, 5-6 (Fed. Cir. 1983).
Easily create and viewing a scene is an expressly recognized advantage. Examiner does not rely upon KSR in the combination of Pahud.
Applicant remarks page 25-26 further repeats argument based on analogousness which has already been discussed above.
Applicant remarks page 26 further argues:
VI. THE SPECIFIC CLAIM ELEMENT PAHUD IS CITED FOR IS NOT TAUGHT BY PAHUD IN ANY TECHNICALLY MEANINGFUL SENSE
…
Pahud's "color" is a natural language word entered by a user as part of a text description of a desired animation (e.g., "a red sunset"). Pahud's NLP component parses this text and maps the word "red" to a graphical asset in a consumer animation library. This is not a "semantically described parameter" in the technical sense used in the claimed subject matter.
This argument is unpersuasive. Applicant’s argument here relies upon claim interpretation of “semantic-data” already discussed above. Examiner has found natural language text, such as ‘red,’ is a semantic description of a color parameter. This is within the scope and consistent with Examiner’s claim interpretation discussed above. Accordingly, Examiner finds Pahud teaches the respective “color” claim limitation.
Applicant remarks page 29 et seq. discusses the instant amendments. Because this discussion relates to new claim scope it does not directly discuss Examiner’s current rejection of this claim scope as presented below. For details regarding Examiner’s current prior art rejection(s), see below.
Examiner Comment
Examiner notes the amendment filed 17 March 2026 omits strikethrough notation for claim language removed from claim 19 in the instant claims, but which was present in the amendment dated 25 November 2025. Examiner reminds Applicant of the appropriate markings required by 37 C.F.R. 1.121 as outlined in MPEP §714(II)(C).
Claim Interpretation
The claims recite “semantic-data dataset” and “semantically described parameters.” The broadest reasonable interpretation (see MPEP §2111) of “semantic-data dataset” is textual data. In this context semantic data is data in text form, also known as a natural language description. A further example is that an adjective is semantic data because adjectives are words (semantic) which provide a description (data).
Claim Rejections - 35 USC § 112(a) – New Matter
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-23 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1 and 19 recite semantically described parameters “stored as individually named and typed data fields.” The Specification does not disclose “individually named and typed data fields.” Specification page 12 lines 2-3 state “processor 204 stores the enhanced semantic-data dataset on at least one digital memory 206.” However, this disclosure does not specify the type or format in which the semantic data is stored on the memory.
Figure 6 showing “OSI::GroundTruth” and “OSI:SensorData” does not expressly depict semantic-data. Furthermore, even if OSI were considered sufficiently linked to the claim term “semantic-data” the figure fails to provide written description that semantically described parameters are “stored as individually named and typed data fields.”
Dependent claims 2-18 and 20-23 are rejected for depending from a rejected claim.
Claim Rejections - 35 USC § 112 – Indefiniteness
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-23 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claims 1 and 19 recite “a structured semantic-data dataset.” It is unclear what attributes or standard being “structured” means in the context of semantic data. Given some set of textual data, natural language description, or other semantic-data (see claim interpretation above) a person of ordinary skill in the art would be unable to determine whether or not that particular semantic data was sufficiently structured or not.
Dependent claims 2-18 and 20-23 are rejected for depending from a rejected claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 4, 5, and 10-19
Claims 1, 2, 4, 5, and 10-19 are rejected under 35 U.S.C. 103 as being unpatentable over US patent 10,678,244 B2 Iandola, et al. (cited in IDS dated 5 December 2023) [herein “Iandola”] in view of Ratrout, N., et al. “Calibration of PARAMICS Model: Application of Artificial Intelligence-Based Approach” Arabian J. for Science & Engineering, vol. 40, pp. 3459-3468 (2015) (cited in IDS dated 5 December 2023) [herein “Ratrout”], US patent 7,512,537 B2 Pahud, et al. [herein “Pahud”], and US patent 6,405,195 B1 Ahlberg [herein “Ahlberg”].
Claim 1 recites “1. A computer implemented method of creating data for a host vehicle simulation.” Iandola column 2 lines 10-17 disclose:
The autonomous control systems use the synthetic data to train computer models for various detection and control algorithms. In general, this allows autonomous control systems to augment training data to improve performance of computer models, simulate scenarios that are not included existing training data, and/or train computer models that remove unwanted effects or occlusions from sensor data of the environment.
Synthetic data for simulation scenarios is created data for autonomous control systems. Iandola column 1 lines 23-25 defines “Autonomous control systems are systems that guide vehicles (e.g., automobiles, trucks, vans) without direct guidance by human operators.”
Claim 1 further recites “comprising: in each of a plurality of iterations of a host vehicle simulation engine.” Iandola column 13 lines 28-30 disclose “The detection models may perform further functionalities, such as tracking objects across multiple time steps of data.” Each time step is a corresponding time frame of an iteration.
Claim 1 further recites “simulating a certain geographic area.” Iandola column 10 lines 21-24 disclose “Based on the simulated scenes, the data synthesizing module 325 simulates sensor data that capture how different types of sensors would have perceived the environment if the scenes occurred in the physical world.” The environment as if in the physical world is a representation of geographical area. The physical world is geographical areas.
Claim 1 further recites “using at least one processor.” Iandola column 18 line 23 discloses “instructions for execution on a processor.” A processor is at least one processor.
Claim 1 further recites “for: obtaining from an environment simulation engine a structured semantic-data dataset representing a plurality of scene objects present in said geographical area.” Iandola column 5 lines 54-56 disclose “the synthetic data represents sensor data of simulated scenarios in the environment from the perspective of one or more sensors included in the autonomous control system 110.” Iandola column 5 lines 63-64 disclose “simulated objects such as pedestrians.” Simulated objects are scene objects. See further Iandola column 13 lines 17-19.
Iandola column 10 lines 21-24 disclose “Based on the simulated scenes, the data synthesizing module 325 simulates sensor data that capture how different types of sensors would have perceived the environment if the scenes occurred in the physical world.” The environment as if in the physical world is a representation of geographical area. The physical world is geographical areas.
Iandola column 15 lines 4-8 disclose:
The behavior modeling system 620 generates a scene by placing objects in the environment and modeling the behavioral motion of objects in the environment. The behavior modeling system 620 contains a behavior model for each object that is in the object database 650.
The behavior models for each object in the behavior modeling system as a group is a dataset of semantic-data. The behavior models being “for” a respective object is the semantic data representing the respective behavior for the respective object.
Claim 1 further recites “each one of said plurality of scene objects is individually described in a respective discrete object record within said structured semantic-data dataset.” Iandola column 31 lines 34-37 disclose “The detection module 316 trains the detection models that detect presence and location of specific categories of objects given an instance of sensor data.” The locations of the objects correspond with location coordinates for the object.
Iandola column 15 lines 4-8 disclose:
The behavior modeling system 620 generates a scene by placing objects in the environment and modeling the behavioral motion of objects in the environment. The behavior modeling system 620 contains a behavior model for each object that is in the object database 650.
The object database corresponds with the discrete object records.
Claim 1 further recites “wherein a description of each of the plurality of objects in the respective discrete object record comprises at least object location coordinates and a plurality of values of semantically described parameters stored as individually named and typed data fields within said respective discrete object record.” Iandola column 15 lines 4-8 disclose:
The behavior modeling system 620 generates a scene by placing objects in the environment and modeling the behavioral motion of objects in the environment. The behavior modeling system 620 contains a behavior model for each object that is in the object database 650.
The behavior model is a set of semantically described parameters. A behavior model for each object is describing the respective individual object with corresponding data. The object database corresponds with the discrete object records.
Iandola column 15 lines 1-3 disclose “The object database 650 contains 3D models of objects, such as variants of pedestrians, vehicles, road debris, which can be placed into the scene.”
Iandola, Ratrout, and Pahud does not explicitly disclose individually named data fields; however, in analogous art of database structure, Ahlberg column 3 lines 16-22 teaches:
The invention may operate with data base data stored or arranged according to any known data structure. In the preferred embodiment of the invention, however, the data base data is structured into records, each record having one or more fields. Each field contains field data, has a field name and one of a plurality of data types.
Data records within the database having each field have a field name corresponds with parameters being individually named. Each field having a data type is a respective typing of the data fields.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, and Ahlberg. One having ordinary skill in the art would have found motivation to use conventional database record structure into the system of simulating a vehicle environment for the advantageous purpose of a system which can display the data logically consistently to a user. See Ahlberg column 2 lines 22-27.
Claim 1 further recites “said semantically described parameters are indicative of at least one of color, size, shape, text on signboards and states of traffic lights.” From the above list of alternatives the Examiner is selecting “color.”
Neither Iandola nor Ratrout explicitly disclose semantically described parameters indicating a color; however, in analogous art of simulating using text-based input, Pahud column 1 lines 59-60 teaches “generating a scene or animation based at least in part upon text entered in a natural language form.” Pahud column 2 lines 13-20 teach:
the input including, but not limited to mood, color, dimension, or size. … the appropriate image(s) can be accessed from a graphics library and assembled to create a scene that is representative of the input.
Creating a scene from input text entered in natural language form to indicate a color is a semantically described parameter describing color. See further Pahud column 5 lines 30-55.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, and Pahud. One having ordinary skill in the art would have found motivation to use natural language text input to create a simulated scene into the system of simulating a vehicle environment for the advantageous purpose of easily create and view an animated scene. See Pahud column 5 lines 52-55.
Claim 1 further recites “creating a virtual three dimensional (3D) visual realistic scene emulating said geographical area according to said structured semantic-data dataset, by placing individually, at least some of said objects in said virtual 3D visual realistic scene.” Iandola column 5 lines 54-56 disclose “the synthetic data represents sensor data of simulated scenarios in the environment from the perspective of one or more sensors included in the autonomous control system 110.” Iandola column 10 lines 21-24 disclose “Based on the simulated scenes, the data synthesizing module 325 simulates sensor data that capture how different types of sensors would have perceived the environment if the scenes occurred in the physical world.” The simulated scenes simulating sensor data as it would have been perceived in the environment of the physical world is creating a realistic scene emulating the real world according to the synthetic data. The synthetic data includes the semantic data, specifically the modeled behavioral model. The physical world is 3D.
Iandola column 15 lines 4-8 disclose:
The behavior modeling system 620 generates a scene by placing objects in the environment and modeling the behavioral motion of objects in the environment. The behavior modeling system 620 contains a behavior model for each object that is in the object database 650.
Placing objects in the environment is placing objects in said virtual 3D visual realistic scene.
Neither Iandola nor Ratrout explicitly disclose semantic-data is structured; however, in analogous art of simulating using text-based input, Pahud column 6 lines 1-3 teach “The NLP component 210 can understand the basic semantic structure, or logical form, of a given sentence.” The semantic structure corresponds with at least some structure of respective natural language data.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, and Pahud. One having ordinary skill in the art would have found motivation to use natural language text input to create a simulated scene into the system of simulating a vehicle environment for the advantageous purpose of easily create and view an animated scene. See Pahud column 5 lines 52-55.
Claim 1 further recites “and by injecting one or more dynamic or moving objects into said virtual 3D visual realistic scene.” Iandola column 13 lines 15-19 disclose:
Objects may include both stationary and moving items in the scenery of the physical environment. For example, stationary objects may include guard rails, road signs, or traffic cones. As another example, moving objects may include pedestrians, bicyclists, animals, or vehicles.
Moving objects of bicyclists and vehicles are injected moving objects into the scene. Iandola column 2 lines 61-64 define “vehicles” as including “automobiles, trucks, vans, …, and the like.”
Claim 1 further recites “applying at least one noise pattern associated with physical sensing behavior characteristics of at least one sensor of a vehicle simulated by said host vehicle simulation engine on said virtual 3D visual realistic scene to create sensory ranging data simulation of said geographical area.” Iandola column 5 lines 57-67 disclose:
synthetic data may include sensor images of the environment that are modified to introduce artifacts in the environment. Specifically, artifacts introduced into the synthetic data may virtually represent any modifications to an environment in existing sensor data, and may represent atmospheric conditions such as precipitation, iciness, snow, lighting conditions such as intense sunlight, darkness, simulated objects such as pedestrians, precipitation, puddles, and the like in the surrounding environment. As another example, synthetic data may include sensor data of the environment that simulate partial sensor malfunction.
Simulating a partial sensor malfunction is applying a noise pattern associated with a physical sensing behavior characteristic. Iandola column 12 lines 49-56 disclose:
the reconstruction module 315 may request synthesized sensor data that include data that reflect reduced image quality due to the simulated artifacts. For example, the reconstruction module 315 may request synthesized sensor data that simulate artifacts such as a dark-colored car, precipitation, and puddles on the road that generate incomplete sensor data in at least some portions of the image due to these artifacts.
These artifacts with reduced image quality representing various modifications are each respective noise patterns. Modifying the synthetic data to introduce artifacts is applying at least one noise pattern to create sensory ranging data of the environment.
Claim 1 further recites “wherein said sensory ranging data simulation represents said geographical area as perceived by said at least one sensor including effects of said noise pattern on object detection.” Iandola column 13 lines 4-5 disclose “The detection module 316 trains detection models that detect objects in the scene based on sensor data.” The detection model being trained on the sensor data corresponds with the object detection including effects of the sensor noise.
Claim 1 further recites “converting said sensory ranging data simulation to an enhanced structured semantic-data dataset emulating said geographical area by parsing said sensory ranging data simulation to identify said plurality of scene objects therein.” Iandola column 13 lines 4-5 disclose “The detection module 316 trains detection models that detect objects in the scene based on sensor data.” Detecting objects based on the sensor data corresponds with parsing said sensory ranging to identify respective scene objects therein.
Claim 1 further recites “and enhancing the discrete object records of said plurality of scene objects in said structured semantic-data dataset comprising: adjusting the object location coordinates stored in said discrete object records based on object attribute values derived from said created sensory ranging data; and adapting respective one or more of the values of said semantically described parameters stored in said discrete object records, based on object attribute values derived from said created sensory ranging data.” Iandola column 12 lines 49-56 disclose:
the reconstruction module 315 may request synthesized sensor data that include data that reflect reduced image quality due to the simulated artifacts. For example, the reconstruction module 315 may request synthesized sensor data that simulate artifacts such as a dark-colored car, precipitation, and puddles on the road that generate incomplete sensor data in at least some portions of the image due to these artifacts.
The simulated artifacts alter the sensor data correspond with adjusted simulated object sensed locations and adapted parameter values. The dark-colored car being affected by these artifacts is a color being adapted according to the artifacts. Color is a semantically described parameter.
As discussed above, Iandola column 15 lines 4-8 disclose behavioral models. Iandola column 15 lines 8-12 continues:
In one implementation, the behavioral engine is parameterized such that "conservative" or "risky" behavior can be selected. For "risky" behavior (e.g. pedestrians that run into the road at random, cars that frequently run traffic lights)
The parameterization of the behavioral model is an adapting of the respective semantically described parameters of the behavioral model.
Claim 1 further recites “and providing said enhanced structured semantic-data dataset to said host vehicle simulation engine for updating a simulation of said vehicle in said geographical area.” Iandola column 2 lines 10-17 disclose:
The autonomous control systems use the synthetic data to train computer models for various detection and control algorithms. In general, this allows autonomous control systems to augment training data to improve performance of computer models, simulate scenarios that are not included existing training data, and/or train computer models that remove unwanted effects or occlusions from sensor data of the environment.
Training the computer models with the synthetic data to simulate scenarios is providing the enhanced semantic data of the synthetic data to the system performing the simulation for training. The simulated scenarios of synthetic data are the updated simulation of the geographic environment.
Claim 2 further recites “2. The method of claim 1, wherein creating said virtual 3D visual realistic scene comprises executing a neural network; wherein said neural network receives said semantic-data dataset; and wherein said neural network generates said virtual 3D visual realistic scene according to said semantic-data dataset.” Iandola column 16 lines 1-6 disclose:
In another implementation, the sensor modeling system 630 refines synthetic LIDAR data that was generated using another approach. The sensor modeling system 630 performs this refinement using a convolutional neural network (CNN), deep neural network (DNN), generative adversarial network (GAN), or the like.
Refining the synthetic data is receiving the synthetic data and generating a respective 3D visual realistic scene. The pre-refined data is the received semantic data. The refined data is the corresponding refined 3D visual scene of the sensor modeling.
Claim 4 further recites “4. The method of claim 2, wherein said neural network is a generator network of a Generative Adversarial Neural Network (GAN) or of a Conditional Generative Adversarial Neural Network (cGAN).” From the above list of alternatives the Examiner is selecting “a Generative Adversarial Neural Network (GAN).”
Iandola column 16 lines 3-6 disclose “The sensor modeling system 630 performs this refinement using a convolutional neural network (CNN), deep neural network (DNN), generative adversarial network (GAN), or the like.”
Claim 5 further recites “5. The method of claim 1, wherein said at least one sensor of said vehicle simulated by said host vehicle simulation engine is selected from a group of sensors consisting of: a camera, a video camera, an infrared camera, a night vision sensor, a Light Detection and Ranging (LIDAR) sensor, a radar, and an ultra-sonic sensor.” From the above list of alternatives the Examiner is selecting “a Light Detection and Ranging (LIDAR) sensor.”
Iandola column 8 lines 3-5 disclose “Active sensors can include ultrasound sensors, RADAR sensors, active infrared (IR) sensors, LIDAR sensors, and the like.”
Iandola column 15 lines 62-65 disclose “the physics of the LIDAR is simulated in sufficient detail such that the limitations of the LIDAR, such as range, photons absorbed by dark objects result from the physics simulation.”
Furthermore, Iandola column 15 lines 23-26 disclose “when the sensor is a camera with specific parameters such as focal length and field of view, the sensor modeling system 630 simulates the optics of the camera.”
Claim 10 further recites “10. The method of claim 1, further comprising using the at least one processor for: generating report data comprising at least one of analysis report data and analytics report data; and outputting said report data.” Iandola column 6 lines 36-42 disclose “generates a set of predictions by applying the computer model…The model training system 140 determines a loss function that indicates a difference between the set of predictions and the set of target outputs. The set of parameters are updated to reduce the loss function.” The performance of the applied computer models making the predictions is analysis report data and analytics. The loss function is a measure of performance of the respective computer model predictions in comparison to the target output. Updating the parameters to reduce the loss function is outputting the loss function measure of performance to update the parameters.
Note: The claimed “outputting” is not required to be output to a user; accordingly, output to another function for updating the parameters is an “outputting” of the loss function analytics under the broadest reasonable interpretation of “outputting” as claimed.
Claim 11 further recites “11. The method of claim 1, wherein said semantically described parameters are further indicative of at least one of velocity, movement parameters and behavior parameters.” From the above list of alternatives the Examiner is selecting “behavior parameters.”
Iandola column 15 lines 4-8 disclose:
The behavior modeling system 620 generates a scene by placing objects in the environment and modeling the behavioral motion of objects in the environment. The behavior modeling system 620 contains a behavior model for each object that is in the object database 650.
The behavior model is a set of behavioral parameters.
Alternatively, Pahud as discussed above teaches “color” as a member. Color remains one of the possible members of the group of semantically described parameters listed.
Claim 12 further recites “12. The method of claim 1, wherein at least one of said one or more dynamic or moving objects is a ground vehicle.” Iandola column 13 lines 15-19 disclose:
Objects may include both stationary and moving items in the scenery of the physical environment. For example, stationary objects may include guard rails, road signs, or traffic cones. As another example, moving objects may include pedestrians, bicyclists, animals, or vehicles.
Moving objects of bicyclists and vehicles are injected moving objects into the scene. Iandola column 2 lines 61-64 define “vehicles” as including “automobiles, trucks, vans, …, and the like.”
Claim 12 further recites “wherein emulating movement of said ground vehicle is controlled according to driver behavior data received from a driver behavior simulator.” As discussed above, Iandola column 15 lines 4-8 disclose behavioral models. Iandola column 15 lines 8-12 continues:
In one implementation, the behavioral engine is parameterized such that "conservative" or "risky" behavior can be selected. For "risky" behavior (e.g. pedestrians that run into the road at random, cars that frequently run traffic lights)
The behavioral engine parameterization of behavior of “cars” is emulating ground vehicles according to a driver behavior simulator.
Claim 12 further recites “and adjusted according to one or more driver behavior patterns and/or driver behavior classes; and wherein said one or more driver behavior patterns and said driver behavior classes are identified through big-data analysis over a large data set of sensory data collected from a plurality of drivers having driver behavior patterns and/or driver behavior classes typical to said geographical area.” Iandola does not explicitly disclose driver behavior patterns identified through data analysis of a large data set of sensory data collected from a plurality of drivers typical of a geographical area; however, in analogous art of simulating driving behavior in traffic, Ratrout page 3462 section 2.3 step 7 teaches:
Step 7 Use an optimization tool such as [genetic algorithm (GA)] model to determine the appropriate values of [mean target headway (MTH)] and [mean reaction time (MRT)] for the given network. The model determines the desired MTH and MRT, which ensure minimum difference with the measured values of queue lengths and the ANN output (i.e., queue lengths).
Parameterizing mean target headway (MTH) and mean reaction time (MRT) by using a genetic algorithm is a big data analysis of a large data set. Parameterizing the model is identifying respective driving behavior and driving behavior classes. The measured values of queue lengths is a large data set of sensory data collected for the geographic area.
Ratrout abstract lines 5-7 teach “calibration of microscopic model incorporating the driving behavior for the local traffic conditions in the Kingdom of Saudi Arabia.” Incorporating driving behavior for local traffic conditions is driving behavior of a geographical area. The local traffic conditions are a geographical area.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola and Ratrout. One having ordinary skill in the art would have found motivation to use calibrating a traffic microscopic simulation model into the system of simulating a vehicle environment for the advantageous purpose of incorporate diverse driving behavior to model local traffic conditions. See Ratrout page 3459 right column second paragraph.
Claim 13 further recites “13. The method of claim 1, wherein a time between consecutive iterations is determined based on one of a predefined time frame and a simulated velocity change of a vehicle simulated by said host vehicle simulation engine.” From the above list of alternatives the Examiner is selecting “a predefined time frame.”
Iandola column 13 lines 28-30 disclose “The detection models may perform further functionalities, such as tracking objects across multiple time steps of data.” Each time step is a corresponding time frame of an iteration.
Claim 14 further recites “14. The method of claim 1, wherein said creating said virtual 3D visual realistic scene comprising overlaying visual imagery of said at least some of said objects, labeled with class labels, over a geographic map of said geographical area.” Iandola column 31 lines 34-37 disclose “The detection module 316 trains the detection models that detect presence and location of specific categories of objects given an instance of sensor data.” The locations of the objects correspond with location coordinates for the object. The categories of the objects are class labels for the respective object. Iandola column 13 lines 58-60 disclose “trains one or more neural networks to perform low-level semantic segmentation, which consists of classifying the type of object or surface.” The respective processed segmentation which identifies the classification of the object is also considered a respective class label for the object(s).
Iandola column 17 lines 61-62 disclose “sensor data that correspond to locations of the simulated set of artifacts in the surrounding environment.” The simulated objects in the surrounding environment are said objects. The synthesized sensor data is overlaid visual imagery of the respective objects.
Iandola column 10 lines 21-24 disclose “Based on the simulated scenes, the data synthesizing module 325 simulates sensor data that capture how different types of sensors would have perceived the environment if the scenes occurred in the physical world.” The environment as if in the physical world is a representation of geographical area. The physical world is geographical areas.
Claim 14 further recites “each in a respective location, position, orientation and proportion identified by analyzing said geographic map.” Iandola column 31 lines 34-37 disclose “The detection module 316 trains the detection models that detect presence and location of specific categories of objects given an instance of sensor data.” The locations of the objects correspond with location. Location is a combination of position and orientation for the object. The categories of the objects are class labels for the respective object. Iandola column 13 lines 58-60 disclose “trains one or more neural networks to perform low-level semantic segmentation, which consists of classifying the type of object or surface.” The respective processed segmentation which identifies the classification of the object is also considered a respective class label for the object(s).
Iandola column 5 lines 54-56 disclose “the synthetic data represents sensor data of simulated scenarios in the environment from the perspective of one or more sensors.” Simulating from the perspective of the sensor applies corresponding proportions (i.e. image sizes) to the corresponding simulated objects in the environment.
Claim 15 further recites “15. The method of claim 12, wherein said sensory data of said large data set includes at least one of speed, acceleration, direction, orientation, elevation, space keeping and position in lane.” From the above list of alternatives the Examiner is selecting “space keeping.”
Iandola does not explicitly disclose driver behavior patterns identified through data analysis of a large data set of sensory data collected from a plurality of drivers typical of a geographical area; however, in analogous art of simulating driving behavior in traffic, Ratrout page 3462 section 2.3 step 7 teaches:
Step 7 Use an optimization tool such as [genetic algorithm (GA)] model to determine the appropriate values of [mean target headway (MTH)] and [mean reaction time (MRT)] for the given network. The model determines the desired MTH and MRT, which ensure minimum difference with the measured values of queue lengths and the ANN output (i.e., queue lengths).
Mean target headway (MTH) is a space keeping.
Furthermore, Ratrout page 3459 right column second paragraph teaches “Such models rely mainly on the use of car-following, lane-changing, and gap-acceptance rules to better describe longitudinal and lateral movements of individual vehicle.” Car-following and gap-acceptance are space keeping. Lane changing and lateral movement correspond with a position in lane.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola and Ratrout. One having ordinary skill in the art would have found motivation to use calibrating a traffic microscopic simulation model into the system of simulating a vehicle environment for the advantageous purpose of incorporate diverse driving behavior to model local traffic conditions. See Ratrout page 3459 right column second paragraph.
Claim 16 further recites “16. The method of claim 12, wherein said big-data analysis is conducted using one or more machine learning algorithms which are members of a group consisting of a neural network and a Support Vector Machine (SVM).” From the above list of alternatives the Examiner is selecting “neural network.”
Ratrout abstract lines 11-12 teaches “artificial neural network (ANN) model.” An ANN is a neural network.
Claim 17 further recites “17. The method of claim 1, further comprising adjusting the at least one noise pattern according to at least one environmental characteristic, said at least one environmental characteristic is a member a group consisting of weather, time of day and date.” From the above list of alternatives the Examiner is selecting “weather.”
Iandola column 10 lines 50-59 discloses:
Specifically, the sensor quality module 313 constructs a training data set that contains sensor data collected in a variety of weather and environmental conditions. Each instance of sensor data in the training data set is assigned a quality status label. For example, human annotators may assign a quality status label to each data sample, such as "rain," "dark," or "direct sunlight." The sensor quality models are trained to determine the quality status of sensor data based on the relationship between sensor data and status labels in the training data set.
The quality of the sensor data corresponds with a noise patterning. The weather label corresponds with a weather environmental characteristic.
Claim 18 further recites “18. The method of claim 1, further comprising adjusting said enhanced semantic-data dataset emulating said geographical area based on mounting attributes of the at least one sensor.” Iandola column 5 lines 15-19 disclose “the autonomous control system 110 may synthesize the sensor data from the set of cameras with different resolution, color, and field-of-view (e.g., front, back, left, and right sides).” Synthesizing sensor data with a respective field-of-view and/or on respective sides/front/back corresponds with adjusting semantic-data dataset based on mounting attributes of the sensor.
Examiner notes Specification page 15 lines 3-9 includes “field of view (FOV)” as a specific example of “mounting attributes.” The claim scope covers at least the specific examples enumerated in the Specification.
Claim 19 recites “19. A system for creating data for a host vehicle simulation.” Iandola column 2 lines 10-17 disclose:
The autonomous control systems use the synthetic data to train computer models for various detection and control algorithms. In general, this allows autonomous control systems to augment training data to improve performance of computer models, simulate scenarios that are not included existing training data, and/or train computer models that remove unwanted effects or occlusions from sensor data of the environment.
Synthetic data for simulation scenarios is created data for autonomous control systems. Iandola column 1 lines 23-25 defines “Autonomous control systems are systems that guide vehicles (e.g., automobiles, trucks, vans) without direct guidance by human operators.”
Claim 19 further recites “comprising: an input interface for obtaining from an environment simulation engine a structured semantic-data dataset representing a plurality of scene objects present in said geographical area.” Iandola column 5 lines 54-56 disclose “the synthetic data represents sensor data of simulated scenarios in the environment from the perspective of one or more sensors included in the autonomous control system 110.” Iandola column 5 lines 63-64 disclose “simulated objects such as pedestrians.” Simulated objects are scene objects. See further Iandola column 13 lines 17-19.
Iandola column 10 lines 21-24 disclose “Based on the simulated scenes, the data synthesizing module 325 simulates sensor data that capture how different types of sensors would have perceived the environment if the scenes occurred in the physical world.” The environment as if in the physical world is a representation of geographical area. The physical world is geographical areas.
Iandola column 15 lines 4-8 disclose:
The behavior modeling system 620 generates a scene by placing objects in the environment and modeling the behavioral motion of objects in the environment. The behavior modeling system 620 contains a behavior model for each object that is in the object database 650.
The behavior models for each object in the behavior modeling system as a group is a dataset of semantic-data. The behavior models being “for” a respective object is the semantic data representing the respective behavior for the respective object.
Claim 19 further recites “each one of said plurality of scene objects is individually described in a respective discrete object record within said structured semantic-data dataset.” Iandola column 31 lines 34-37 disclose “The detection module 316 trains the detection models that detect presence and location of specific categories of objects given an instance of sensor data.” The locations of the objects correspond with location coordinates for the object.
Iandola column 15 lines 4-8 disclose:
The behavior modeling system 620 generates a scene by placing objects in the environment and modeling the behavioral motion of objects in the environment. The behavior modeling system 620 contains a behavior model for each object that is in the object database 650.
The object database corresponds with the discrete object records.
Claim 19 further recites “wherein a description of each of the plurality of objects in the respective discrete object record comprises at least object location coordinates and a plurality of values of semantically described parameters stored as individually named and typed data fields within said respective discrete object record.” Iandola column 15 lines 4-8 disclose:
The behavior modeling system 620 generates a scene by placing objects in the environment and modeling the behavioral motion of objects in the environment. The behavior modeling system 620 contains a behavior model for each object that is in the object database 650.
The behavior model is a set of semantically described parameters. A behavior model for each object is describing the respective individual object with corresponding data. The object database corresponds with the discrete object records.
Iandola column 15 lines 1-3 disclose “The object database 650 contains 3D models of objects, such as variants of pedestrians, vehicles, road debris, which can be placed into the scene.”
Iandola, Ratrout, and Pahud does not explicitly disclose individually named data fields; however, in analogous art of database structure, Ahlberg column 3 lines 16-22 teaches:
The invention may operate with data base data stored or arranged according to any known data structure. In the preferred embodiment of the invention, however, the data base data is structured into records, each record having one or more fields. Each field contains field data, has a field name and one of a plurality of data types.
Data records within the database having each field have a field name corresponds with parameters being individually named. Each field having a data type is a respective typing of the data fields.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, and Ahlberg. One having ordinary skill in the art would have found motivation to use conventional database record structure into the system of simulating a vehicle environment for the advantageous purpose of a system which can display the data logically consistently to a user. See Ahlberg column 2 lines 22-27.
Claim 19 further recites “said semantically described parameters are indicative of at least one of color, size, shape, text on signboards and states of traffic lights.” From the above list of alternatives the Examiner is selecting “color.”
Neither Iandola nor Ratrout explicitly disclose semantically described parameters indicating a color; however, in analogous art of simulating using text-based input, Pahud column 1 lines 59-60 teaches “generating a scene or animation based at least in part upon text entered in a natural language form.” Pahud column 2 lines 13-20 teach:
the input including, but not limited to mood, color, dimension, or size. … the appropriate image(s) can be accessed from a graphics library and assembled to create a scene that is representative of the input.
Creating a scene from input text entered in natural language form to indicate a color is a semantically described parameter describing color. See further Pahud column 5 lines 30-55.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, and Pahud. One having ordinary skill in the art would have found motivation to use natural language text input to create a simulated scene into the system of simulating a vehicle environment for the advantageous purpose of easily create and view an animated scene. See Pahud column 5 lines 52-55.
Claim 19 further recites “at least one processor for conducting in each of said plurality of iterations.” Iandola column 18 line 23 discloses “instructions for execution on a processor.” A processor is at least one processor.
Claim 19 further recites “creating a virtual three dimensional (3D) visual realistic scene emulating said geographical area according to said structured semantic-data dataset, by placing individually, at least some of said objects in said virtual 3D visual realistic scene.” Iandola column 5 lines 54-56 disclose “the synthetic data represents sensor data of simulated scenarios in the environment from the perspective of one or more sensors included in the autonomous control system 110.” Iandola column 10 lines 21-24 disclose “Based on the simulated scenes, the data synthesizing module 325 simulates sensor data that capture how different types of sensors would have perceived the environment if the scenes occurred in the physical world.” The simulated scenes simulating sensor data as it would have been perceived in the environment of the physical world is creating a realistic scene emulating the real world according to the synthetic data. The synthetic data includes the semantic data, specifically the modeled behavioral model. The physical world is 3D.
Iandola column 15 lines 4-8 disclose:
The behavior modeling system 620 generates a scene by placing objects in the environment and modeling the behavioral motion of objects in the environment. The behavior modeling system 620 contains a behavior model for each object that is in the object database 650.
Placing objects in the environment is placing objects in said virtual 3D visual realistic scene.
Claim 19 further recites “and by injecting one or more dynamic or moving objects into said virtual 3D visual realistic scene.” Iandola column 13 lines 15-19 disclose:
Objects may include both stationary and moving items in the scenery of the physical environment. For example, stationary objects may include guard rails, road signs, or traffic cones. As another example, moving objects may include pedestrians, bicyclists, animals, or vehicles.
Moving objects of bicyclists and vehicles are injected moving objects into the scene. Iandola column 2 lines 61-64 define “vehicles” as including “automobiles, trucks, vans, …, and the like.”
Claim 19 further recites “applying at least one noise pattern associated with physical sensing behavior characteristics of at least one sensor of a vehicle simulated by a host vehicle simulation engine on said virtual 3D visual realistic scene to create sensory ranging data simulation of said geographical area.” Iandola column 5 lines 57-67 disclose:
synthetic data may include sensor images of the environment that are modified to introduce artifacts in the environment. Specifically, artifacts introduced into the synthetic data may virtually represent any modifications to an environment in existing sensor data, and may represent atmospheric conditions such as precipitation, iciness, snow, lighting conditions such as intense sunlight, darkness, simulated objects such as pedestrians, precipitation, puddles, and the like in the surrounding environment. As another example, synthetic data may include sensor data of the environment that simulate partial sensor malfunction.
Simulating a partial sensor malfunction is applying a noise pattern associated with a physical sensing behavior characteristic. Iandola column 12 lines 49-56 disclose:
the reconstruction module 315 may request synthesized sensor data that include data that reflect reduced image quality due to the simulated artifacts. For example, the reconstruction module 315 may request synthesized sensor data that simulate artifacts such as a dark-colored car, precipitation, and puddles on the road that generate incomplete sensor data in at least some portions of the image due to these artifacts.
These artifacts with reduced image quality representing various modifications are each respective noise patterns. Modifying the synthetic data to introduce artifacts is applying at least one noise pattern to create sensory ranging data of the environment.
Claim 19 further recites “wherein said sensory ranging data simulation represents said geographical area as perceived by said at least one sensor including effects of said noise pattern on object detection.” Iandola column 13 lines 4-5 disclose “The detection module 316 trains detection models that detect objects in the scene based on sensor data.” The detection model being trained on the sensor data corresponds with the object detection including effects of the sensor noise.
Claim 19 further recites “converting said sensory ranging data simulation to an enhanced structured semantic-data dataset emulating said geographical area by parsing said sensory ranging data simulation to identify said plurality of scene objects therein.” Iandola column 13 lines 4-5 disclose “The detection module 316 trains detection models that detect objects in the scene based on sensor data.” Detecting objects based on the sensor data corresponds with parsing said sensory ranging to identify respective scene objects therein.
Claim 19 further recites “and enhancing the discrete object records of said plurality of scene objects in said structured semantic-data dataset comprising: adjusting the object location coordinates stored in said discrete object records based on object attributed values derived from said created sensory ranging data; and adapting respective one or more of the values of said semantically described parameters stored in said discrete object records, based on object attribute values derived from said created sensory ranging data.” Iandola column 12 lines 49-56 disclose:
the reconstruction module 315 may request synthesized sensor data that include data that reflect reduced image quality due to the simulated artifacts. For example, the reconstruction module 315 may request synthesized sensor data that simulate artifacts such as a dark-colored car, precipitation, and puddles on the road that generate incomplete sensor data in at least some portions of the image due to these artifacts.
The simulated artifacts alter the sensor data correspond with adjusted simulated object sensed locations and adapted parameter values. The dark-colored car being affected by these artifacts is a color being adapted according to the artifacts. Color is a semantically described parameter.
As discussed above, Iandola column 15 lines 4-8 disclose behavioral models. Iandola column 15 lines 8-12 continues:
In one implementation, the behavioral engine is parameterized such that "conservative" or "risky" behavior can be selected. For "risky" behavior (e.g. pedestrians that run into the road at random, cars that frequently run traffic lights)
The parameterization of the behavioral model is an adapting of the respective semantically described parameters of the behavioral model.
Claim 19 further recites “and an output interface for providing said enhanced structured semantic-data dataset to said host vehicle simulation engine for updating a simulation of said vehicle in said geographical area.” Iandola column 2 lines 10-17 disclose:
The autonomous control systems use the synthetic data to train computer models for various detection and control algorithms. In general, this allows autonomous control systems to augment training data to improve performance of computer models, simulate scenarios that are not included existing training data, and/or train computer models that remove unwanted effects or occlusions from sensor data of the environment.
Training the computer models with the synthetic data to simulate scenarios is providing the enhanced semantic data of the synthetic data to the system performing the simulation for training. The simulated scenarios of synthetic data are the updated simulation of the geographic environment.
Dependent Claim 3
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Iandola, Ratrout, Pahud, and Ahlberg as applied to claim 2 above, and further in view of Pan, T., et al. “Perceptual Loss with Fully Convolutional for Image Residual Denoising” Pattern Recognition, pp. 122-132, Communications in Computer & Information Science, vol. 663, Springer Singapore (October 2016) (cited in IDS dated 5 December 2023) [herein “Pan”].
Claim 3 further recites “3. The method of claim 2, wherein said neural network is trained using a perceptual loss function.” Iandola nor Ratrout explicitly disclose using a perceptual loss function; however, in analogous art of image processing, Pan title teaches “Perceptual Loss with Fully Convolutional for Image Residual Denoising.” Pan page 130 section 5 last paragraph teaches “We train image transformation networks with per-pixel loss layer and perceptual loss layer. … we show that train with a mix loss layers allows the model to better restore fine details and edges.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, Ahlberg, and Pan. One having ordinary skill in the art would have found motivation to use Denoising with perceptual loss functions into the system of data synthesis for evaluating autonomous control systems for the advantageous purpose of “better restoring fine details and edges” (Pan page 131) and because “the
l
f
e
a
t
model may be more aware of image semantics.” See Pan page 130 first paragraph last sentence.
Dependent Claims 6-8, 20, and 21
Claims 6-8, 20, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Iandola, Ratrout, Pahud, and Ahlberg as applied to claims 1 and 11 above, and further in view of Jayaraman, et al. “Creating 3D Virtual Driving Environments for Simulation-Aided Development of Autonomous Driving and Active Safety”, SAE Technical Paper Series, SAE Int’l., XP055518353 (March 2017) (cited in IDS dated 5 December 2023) [herein “Jayaraman”] (cited in IDS dated 19 December 2018).
Claim 6 further recites “6. The method of claim 1, wherein providing said enhanced semantic-data dataset to said host vehicle simulation engine comprises sending a stream of data to at least one other processor via at least one digital communication network interface connected to said at least one processor.” Iandola nor Ratrout explicitly disclose a stream of data for simulation; however, in analogous art of vehicle sensor simulation environments, Jayaraman page 2 right column first paragraph lines 11-13 teach “Note that TCP/IP and UDP communication are still useful in cases where MATLAB and the Unreal Engine project is not running processes on the same machine.” Not on “the same machine” means at least one other machine with respective at least one other processor. TCP/IP and UDP communication are network communication interface. See further Jayaraman page 4 right column.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, Ahlberg, and Jayaraman. One having ordinary skill in the art would have found motivation to use multiple computers/machines for implementing the system of data synthesis for evaluating autonomous control systems for the advantageous purpose of processing the perception and control algorithms (i.e. MATLAB) separate from the virtual simulator (i.e. Unreal Engine). See Jayaraman page 2 right column first paragraph lines 11-13. Jayaraman abstract last sentence teaches “demonstrated to run at sufficient frame rates for real-time computations.” Frame rates indicates a video stream of the respective frame rate. The video stream is a stream of data.
Claim 7 further recites “7. The method of claim 1, wherein providing said enhanced semantic-data dataset to said host vehicle simulation engine comprises storing a file on a shared access memory accessible by said host vehicle simulation engine.” Iandola nor Ratrout explicitly disclose shared access memory; however, in analogous art of vehicle sensor simulation environments, Jayaraman page 2 right column first paragraph lines 8-11 teaches “The shared-memory interface was the only interface that was fast enough to facilitate data-exchange rates that allowed the Unreal project to run at 30 FPS or faster while still being able to process camera and LiDAR data in MATLAB.” Jayaraman abstract last sentence teaches “demonstrated to run at sufficient frame rates for real-time computations.” The shared-memory interface is a shared access memory accessible by the vehicle simulation engine. The Unreal engine is a vehicle simulation engine.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, Ahlberg, and Jayaraman. One having ordinary skill in the art would have found motivation to use shared memory interface for data exchange for implementing the system of data synthesis for evaluating autonomous control systems for the advantageous purpose of allowing the simulation to run at 30 FPS or faster for real-time computations. See Jayaraman page 2 right column first paragraph lines 8-11 and abstract.
Claim 8 further recites “8. The method of claim 1, wherein providing said enhanced semantic-data dataset to said host vehicle simulation engine comprises storing a file on a digital data storage.” Iandola nor Ratrout explicitly disclose shared access memory; however, in analogous art of vehicle sensor simulation environments, Jayaraman page 2 right column first paragraph lines 8-11 teaches “The shared-memory interface was the only interface that was fast enough to facilitate data-exchange rates that allowed the Unreal project to run at 30 FPS or faster while still being able to process camera and LiDAR data in MATLAB.” The shared-memory is a digital data storage. Specifically, memory is digital data storage.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, Ahlberg, and Jayaraman. One having ordinary skill in the art would have found motivation to use shared memory interface for data exchange for implementing the system of data synthesis for evaluating autonomous control systems for the advantageous purpose of allowing the simulation to run at 30 FPS or faster for real-time computations. See Jayaraman page 2 right column first paragraph lines 8-11 and abstract.
Claim 20 further recites “20. The system of claim 19, wherein said output interface is a digital communication network interface.” Iandola nor Ratrout explicitly disclose a stream of data for simulation; however, in analogous art of vehicle sensor simulation environments, Jayaraman page 2 right column first paragraph lines 11-13 teach “Note that TCP/IP and UDP communication are still useful in cases where MATLAB and the Unreal Engine project is not running processes on the same machine.” Not on “the same machine” means at least one other machine with respective at least one other processor. TCP/IP and UDP communication are network communication interface. See further Jayaraman page 4 right column.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, Ahlberg, and Jayaraman. One having ordinary skill in the art would have found motivation to use multiple computers/machines for implementing the system of data synthesis for evaluating autonomous control systems for the advantageous purpose of processing the perception and control algorithms (i.e. MATLAB) separate from the virtual simulator (i.e. Unreal Engine). See Jayaraman page 2 right column first paragraph lines 11-13. Jayaraman abstract last sentence teaches “demonstrated to run at sufficient frame rates for real-time computations.” Frame rates indicates a video stream of the respective frame rate. The video stream is a stream of data.
Claim 21 further recites “21. The system of claim 19, further comprising a digital memory for at least one of storing code and storing an enhanced semantic-data dataset.” Iandola nor Ratrout explicitly disclose a memory; however, in analogous art of vehicle sensor simulation environments, Jayaraman page 2 right column first paragraph lines 8-11 teaches “The shared-memory interface was the only interface that was fast enough to facilitate data-exchange rates that allowed the Unreal project to run at 30 FPS or faster while still being able to process camera and LiDAR data in MATLAB.” Jayaraman abstract last sentence teaches “demonstrated to run at sufficient frame rates for real-time computations.” The shared-memory interface is a memory storing the synthetic image data. The Unreal engine is a vehicle simulation engine.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, Ahlberg, and Jayaraman. One having ordinary skill in the art would have found motivation to use shared memory interface for data exchange for implementing the system of data synthesis for evaluating autonomous control systems for the advantageous purpose of allowing the simulation to run at 30 FPS or faster for real-time computations. See Jayaraman page 2 right column first paragraph lines 8-11 and abstract.
Dependent Claim 9
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Iandola, Ratrout, Pahud, and Ahlberg as applied to claim 2 above, and further in view of Biedermann, D., et al. “COnGRATS: Realistic Simulation of Traffic Sequences for Autonomous Driving” IEEE 2015 Int’l Conf. on Image & Vision Computing New Zealand, IVCNZ (2015) (cited in IDS dated 5 December 2023) [herein “Biedermann”].
Claim 9 further recites “9. The method of claim 2, wherein said neural network is trained using optical flow estimation to reduce temporal inconsistency between consecutive frames of a created virtual 3D visual realistic scene.” Iandola nor Ratrout explicitly disclose optical flow estimation; however, in analogous art of creating synthetic datasets for vehicle scenes, Biedermann abstract first sentence teaches “For evaluating or training different kinds of vision algorithms, a large amount of precise and reliable data is needed.” Biedermann page 5 section 3.3 last paragraph teaches “Since the rendering pass that is the source of the optical flow data is implemented in Blender for the use in motion blur, we also utilize this in our sequences to further increase the realism of our image sequences.” Rendering optical flow data is using optical flow estimation to increase realism. Motion blur is a temporal consistency/inconsistency between consecutive frames of the created virtual 3D created scene.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, Ahlberg, and Biedermann. One having ordinary skill in the art would have found motivation to use motion blur rendering of optical flow data into the system of data synthesis for evaluating autonomous control systems for the advantageous purpose of increasing “the realism of our image sequences.” See Biedermann page 5 section 3.3 last paragraph.
Dependent Claims 22 and 23
Claims 22 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Iandola, Ratrout, Pahud, and Ahlberg as applied to claim 11 above, and further in view of US 2019/0294742 A1 Zhao, et al. (cited in IDS dated 5 December 2023) [herein “Zhao”].
Claim 22 further recites “22. The system of claim 19, further comprising a digital data storage connected to said at least one processor via said output interface.” Iandola nor Ratrout explicitly disclose digital data storage; however, in analogous art of simulating visual data for machine learning, Zhao paragraph 54 teaches:
The storage unit 13 may store data captured, processed, generated, simulated, rendered, or otherwise used by the movable object 300. In various embodiments, the storage unit 13 …and may include flash memory, USB drives, memory cards, solid-state drives (SSDs), hard disk drives (HDDs), floppy disks, optical disks, magnetic tapes, and the like.
A hard disk storage unit is a digital data storage. Zhao paragraph 48 teaches “The processing unit 204 may include one or more processors … the simulation engine 214, may be configured to execute one or more instructions stored in the storage unit 203.” The processor is a processor connected to the storage unit.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, Ahlberg, and Zhao. One having ordinary skill in the art would have found motivation to use a storage unit to store the data captured, processed, generated, simulated, rendered, or otherwise used by the system of data synthesis for evaluating autonomous control systems for the advantageous purpose of storing the used data. See Zhao paragraph 54.
Claim 23 further recites “23. The system of claim 19, wherein said digital data storage is selected from a group consisting of: a storage area network, a network attached storage, a hard disk drive, an optical disk, and a solid-state storage.” From the above list of alternatives the Examiner is selecting “a hard disk drive.”
Iandola does not explicitly disclose digital data storage; however, in analogous art of simulating visual data for machine learning, Zhao paragraph 54 teaches:
The storage unit 13 may store data captured, processed, generated, simulated, rendered, or otherwise used by the movable object 300. In various embodiments, the storage unit 13 …and may include flash memory, USB drives, memory cards, solid-state drives (SSDs), hard disk drives (HDDs), floppy disks, optical disks, magnetic tapes, and the like.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Iandola, Ratrout, Pahud, Ahlberg, and Zhao. One having ordinary skill in the art would have found motivation to use a storage unit to store the data captured, processed, generated, simulated, rendered, or otherwise used by the system of data synthesis for evaluating autonomous control systems for the advantageous purpose of storing the used data. See Zhao paragraph 54.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jay B Hann whose telephone number is (571)272-3330. The examiner can normally be reached M-F 10am-7pm EDT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Renee Chavez can be reached at (571) 270-1104. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jay Hann/Primary Examiner, Art Unit 2186 9 May 2026
Prosecution Insights