DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Amendments and arguments filed on 12/04/25 have been received and considered. Claims 1-20 are pending. Claims 1-10, 12-15, and 18-20 have been amended. No claim has been added. No claim has been cancelled.
Response to Arguments
Applicant states:
PNG
media_image1.png
378
794
media_image1.png
Greyscale
Remarks p. 10.
The Examiner disagrees with Applicant reasoning.
[BRI on record]
With respect to “a . . . semantic layer,” the Examiner is reading the limitation to mean a portion of a model, wherein attribute(s) that controls the portion’s appearance could be set individually compared to other portion(s) of the model when there are other portion(s) of the model. This interpretation is made in light of the specification. Spec. ¶¶ 35, 55.
[0035] Models 114 may include semantic layers 110, to which semantic data can be applied to change the visual characteristics of the model 114. Semantic layers 110 may be/include individual portions of each model (e.g., a predetermined set of polygons, vertices, etc.) to which any type of semantic data (e.g., randomly selected textures 116, materials 118, or patterns 120) may be applied. Semantic layers 110 can enable a model 114 to include multiple portions that have different appearances when placed within a scene, rather than having a texture 116, material 118, and/or pattern 120 applied uniformly across the entire surface of the model. The semantic layers 110 of different models may be randomized independently according to the configuration data 104, and therefore enable a greater degree of diversity and control even when a relatively smaller set of models 114 are selected (e.g., as the selected models 108) for a scene 106.
[0055] As described herein, 3D models may include semantic layers (e.g., semantic layers 110), which may be randomized by applying selected textures, patterns, and/or materials to the semantic layers of the 3D models. In one example, the 3D models may include 3D models of people, and the semantic layers may correspond to one or more of articles of clothing, skin color, hair color, eye color, or other visual aspects of the 3D models. Semantic layers may include predetermined sets of polygons, vertices, or portions of a 3D model to which different textures, patterns, colors, and/or materials may be applied relative to other polygons, vertices, or portions of the 3D model. The 3D models described herein may include multiple semantic layers may each be randomized differently, enabling a greater degree of diversity even when using a limited pool of assets.
With respect to “randomization for a . . . semantic layer,” the Examiner is reading the limitation to mean randomly select attribute(s) that controls a portion’s appearance. This interpretation is in light of the specification. Spec. ¶¶ 4, 25.
[0004] At least one aspect relates to a processor. The processor can include one or more circuits. The one or more circuits can receive a configuration (e.g., via a configuration file) that specifies a level or degree of randomization for a semantic layer of a model (e.g., a three-dimensional model) for a scene.
[0025] . . . Enhancing randomization and diversity of simulated datasets can be achieved in part by targeting randomization at different, specific regions or sub-elements of models placed within the simulation. In one example, the clothing of a model of a person may be randomized across a collection of clothing textures, colors, or materials, while skin or hair can be alternatively randomized across a different range of colors. Partial randomization of models, materials, and/or textures results in a greater degree of diversity even when using a limited set of three-dimensional (3D) assets, while still maintaining a degree of realism that is sufficient to generalize artificial intelligence models to real-world images.
[Mapping Analysis]
Tobin teaches randomization, stating “Bridging the ‘reality gap’ that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator.” Abstract.
PNG
media_image2.png
454
414
media_image2.png
Greyscale
, which shows scenes generated based on the randomization. Tobin provides a list of domains for the randomization, and the list includes attributes that controls the appearance of a three-dimensional (3D) . . . asset, mapped to Tobin’s a table and/or objects placed on the table. These attributes include the texture of objects of the model within the scene.
Tobin teaches the asset could be mesh asset, stating “We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator.” Tobin IV.A. Experimental Setup.
Tobin:
PNG
media_image3.png
486
426
media_image3.png
Greyscale
Here, for example, the claimed “semantic layer” could be mapped to each object on the table, or the table. Fig. 1 shows that attributes of each semantic layer, like an object on table, could be controlled individually. Because there are multiple objects, a first semantic layer and a second semantic layer are taught. An asset could comprise one or more objects.
As figs. 1, 7 show each object (semantic layer) could be textured with different colors.
Compact Prosecution
With respect to Claim Interpretation, the Examiner has provided some notes regarding “[BRI on the record]” throughout the Office Action, so that the record is clear about the scope of the claimed invention, and the record is also clear about the basis for the Examiner’s analyses. A clear record of the claim interpretation could expedite the examination by creating the condition to allow the examination to focus on Applicant’s inventive concept and its comparison with related prior art.
If there are disagreements, Applicant may present an alternative interpretation based on MPEP 2111. The Examiner will adopt Applicant’s interpretation on the record, if Applicant’s interpretation is reasonable and/or arguments are persuasive.
Applicant may amend claims relying on the Examiner’s claim interpretation provided on the record.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-9, 11, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Tobin et al. (“Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”) in view of Mazumder et al. (US 11748664 B1).
Regarding Claim 1, Tobin teaches A processor comprising: one or more circuits to:
receive a configuration that specifies randomization for a first semantic layer and a second semantic layer of a three-dimensional (3D) mesh asset for a scene (
[BRI on record]
With respect to “a . . . semantic layer,” the Examiner is reading the limitation to mean a portion of a model/asset, wherein attribute(s) that controls the portion’s appearance could be set individually compared to other portion(s) of the model/asset when there are other portion(s) of the model/asset. This interpretation is made in light of the specification. Spec. ¶¶ 35, 55.
[0035] Models 114 may include semantic layers 110, to which semantic data can be applied to change the visual characteristics of the model 114. Semantic layers 110 may be/include individual portions of each model (e.g., a predetermined set of polygons, vertices, etc.) to which any type of semantic data (e.g., randomly selected textures 116, materials 118, or patterns 120) may be applied. Semantic layers 110 can enable a model 114 to include multiple portions that have different appearances when placed within a scene, rather than having a texture 116, material 118, and/or pattern 120 applied uniformly across the entire surface of the model. The semantic layers 110 of different models may be randomized independently according to the configuration data 104, and therefore enable a greater degree of diversity and control even when a relatively smaller set of models 114 are selected (e.g., as the selected models 108) for a scene 106.
[0055] As described herein, 3D models may include semantic layers (e.g., semantic layers 110), which may be randomized by applying selected textures, patterns, and/or materials to the semantic layers of the 3D models. In one example, the 3D models may include 3D models of people, and the semantic layers may correspond to one or more of articles of clothing, skin color, hair color, eye color, or other visual aspects of the 3D models. Semantic layers may include predetermined sets of polygons, vertices, or portions of a 3D model to which different textures, patterns, colors, and/or materials may be applied relative to other polygons, vertices, or portions of the 3D model. The 3D models described herein may include multiple semantic layers may each be randomized differently, enabling a greater degree of diversity even when using a limited pool of assets.
With respect to “randomization for a semantic layer,” the Examiner is reading the limitation to mean randomly select attribute(s) that controls a portion’s appearance. This interpretation is in light of the specification. Spec. ¶¶ 4, 25.
[0004] At least one aspect relates to a processor. The processor can include one or more circuits. The one or more circuits can receive a configuration (e.g., via a configuration file) that specifies a level or degree of randomization for a semantic layer of a model (e.g., a three-dimensional model) for a scene.
[0025] . . . Enhancing randomization and diversity of simulated datasets can be achieved in part by targeting randomization at different, specific regions or sub-elements of models placed within the simulation. In one example, the clothing of a model of a person may be randomized across a collection of clothing textures, colors, or materials, while skin or hair can be alternatively randomized across a different range of colors. Partial randomization of models, materials, and/or textures results in a greater degree of diversity even when using a limited set of three-dimensional (3D) assets, while still maintaining a degree of realism that is sufficient to generalize artificial intelligence models to real-world images.
[Mapping Analysis]
Tobin teaches randomization, stating “Bridging the ‘reality gap’ that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator.” Abstract.
PNG
media_image2.png
454
414
media_image2.png
Greyscale
, which shows scenes generated based on the randomization. Tobin provides a list of domains for the randomization, and the list includes attributes that controls the appearance of items/objects of a three-dimensional (3D) . . . asset, mapped to Tobin’s a table and/or objects placed on the table. These attributes include the texture of objects of the model within the scene.
Tobin teaches the asset could be mesh asset, stating “We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator.” Tobin IV.A. Experimental Setup.
Tobin:
PNG
media_image3.png
486
426
media_image3.png
Greyscale
Here, for example, the claimed “semantic layer” could be mapped to each object on the table, or the table. Fig. 1 shows that attributes of each semantic layer, like an object on table, could be controlled individually. Because there are multiple objects of an asset, a first semantic layer and a second semantic layer are taught.
The claimed configuration comprises the settings for abovementioned randomized parameters for color, texture, and etc. The configuration is received by the system to generate simulated images as shown in figs. 1, 7.);
sample a distribution according to the randomization to select first data for the first semantic layer of the 3D mesh asset and second data for the second semantic layer of the 3D mesh asset (
[BRI on record]
With respect to “a distribution,” the scope of the term includes a range of options in light of the specification. Spec. ¶¶ 46-47. Particularly, the specification allows “choice distribution” as an example of a “distribution.” The “choice distribution” is a range of options.
[0046] Some parameters specified in the parameter file 202 may be specified via distributions 208, which may be utilized to automatically generate one or more random values for an associated parameter. The distributions 208 may include, but are not limited to, uniform distributions (which may return a floating point value between specified minimum and maximum values), normal distributions (based at least on a specified mean and standard deviation), a range distribution (which may return an integer value between specified minimum and maximum integer values), choice distributions (which may return an element from a list of elements, such as a list of assets in one or more asset lists 204), or a walk distribution (which may be a choice distribution without replacement).
[0047] In some implementations, distributions, including the choice distribution or the walk distribution, may be specified in the parameter file 202 to randomly select certain assets for placement in the scene. For example, the parameter file 202 may include a choice distribution that specifies random selection of a texture from a list of textures in an asset file 202 to apply to a sematic layer of a 3D model that is randomly selected for placement for the scene. Other types of distributions, such as uniform distributions and normal distributions, may be utilized to randomly generate numerical values, such as values that specify the placement coordinates and/or rotation of a 3D model within a scene. For example, rather than explicitly specifying a location of a 3D model within a scene, the parameter file 202 may specify that the position of the model is to be generated using a uniform distribution between specified minimum and maximum coordinate boundaries. Furthering this example, these boundaries may be selected based at least on an environmental 3D model for the scene, such that the 3D model is to be randomly placed within the 3D environment model.
With respect to “sample a distribution,” the scope of the term includes making a selection from a range of options in light of the specification. The Spec. ¶¶ 46-47. Particularly, the specification allows “choice distribution” as an example of a “distribution.” The “choice distribution” is a range of options.
[0052] The sampling process 212 can be executed to generate values or to select assets according to the distribution parameters 210 parsed from the parameter file 210. For example, the sampling process may select one or more random values from specified uniform distributions or normal distributions, or may select assets from one or more lists of assets (e.g., in specified asset file(s) 204) according to the choice distributions or walk distributions specified in the parameter file. Doing so may include executing one or more random number generation algorithms, including random number generation algorithms that sample from Gaussian distributions or uniform distributions. Identifiers of the selected assets (e.g., models, textures, patterns, colors, etc.), as well as any other parameters generated by sampling the distributions 210, may be provided as input to the scene generation process 214, along with any primitive (e.g., constant) parameters parsed from the parameter file 202.
[Mapping Analysis]
PNG
media_image3.png
486
426
media_image3.png
Greyscale
Here, Tobin states, “We randomize the following aspects of the domain for each sample used during training: . . ..” Each domain provides a range of options, mapped to claimed “distribution.” The sampling selects an attribute option, e.g., color, for an object on the table, mapped to “semantic layer.”);
generate the scene including the 3D mesh asset having the first data selected for the first semantic layer of the 3D mesh asset and the second data selected for the second semantic layer of the 3D mesh asset (figs. 1, 7, showing different colors for different objects.); and
render the scene including the 3D mesh asset to generate an image (figs. 1, 7) for updating a neural network (
[BRI on record]
With respect to “render the scene,” the Examiner is reading the limitation to mean: simulating the way that light travels from objects in the scene to the camera or viewport to generate images. This interpretation is in light of the specification. Spec. ¶ 40.
[0040] . . . Once the selected models 108, lights, and other visual effects have been placed in the scene 106, the data processing system 102 may place and/or navigate a virtual camera or other rendering viewport within the scene 106 to generate the output images 122. The data processing system 102 may render the scene 106 by simulating the way that light travels from objects in the scene to the camera or viewport. Parameters of the camera may include position and orientation in the scene 106, field of view, and focal length, among others. The configuration data 104 may specify the parameters for the camera, the number of output images 122 to be generated from the scene, and may define a path (e.g., a series of positions and/or orientations) within the scene 106 at which the camera is to be positioned to generate corresponding output images 122.
With respect to “an image for updating a neural network,” the Examiner is reading the limitation to mean: an image as an training image to train a neural network This interpretation is in light of the specification. Spec. ¶ 71.
[0071] The method 500, at block B508, includes rendering the scene including the model to generate an image for updating a neural network. Rendering the scene may include performing a process for generating output images (e.g., the output images 122) for a training dataset (e.g., the dataset 218).
[Mapping Analysis]
Tobin teaches using generated images for training, stating “Bridging the ‘reality gap’ that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator.” Abstract.
The training is for deep neural networks, and its title states, “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World.”).
Tobin is an academic paper and does not explicitly disclose A processor comprising: one or more circuits, even though Tobin’s teaching is within the context of computer technology.
Mazumder teaches A processor comprising: one or more circuits (
“Said processors can be implemented as any hardware capable of processing data, such as application-specific integrated circuits (ASICs), standard integrated circuits (ICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), logic circuits, or any other appropriate hardware.” Mazumder col. 42 line 58- col. 43 line 5.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mazumder’s processor and circuits with primary reference Tobin. One of ordinary skill in the art would be motivated to automate/implement Tobin’s algorithm on computing technology. Mazumder states, “A person of skill in the art will appreciate that any function or operation within such block diagrams, schematics, and flowcharts can be implemented by a wide range of hardware, software, firmware, or combination thereof. As non-limiting examples, the various embodiments herein can be implemented in one or more of: application-specific integrated circuits (ASICs), standard integrated circuits (ICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), computer programs executed by any number of computers or processors, programs executed by one or more control units or processor units, firmware, or any combination thereof.” Mazumder col. 42 lines 44-57.
Regarding Claim 2, Tobin in view of Mazumder teaches The processor of claim 1, wherein the one or more circuits are to:
generate the scene to include an environmental model (
Tobin:
PNG
media_image2.png
454
414
media_image2.png
Greyscale
, which shows the scene includes an environment that contains walls and a floor.); and
position the 3D mesh asset within the environmental model according to the configuration (The model, the table and/or objects on the table, is positioned within the environment, comprising a floor and walls.
Tobin teaches the asset could be mesh asset, stating “We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator.” Tobin IV.A. Experimental Setup.).
Regarding Claim 3, Tobin in view of Mazumder teaches The processor of claim 1, wherein the one or more circuits are to:
update a position of the 3D mesh asset in the scene according to a simulation of one or more physical constraints (
“In an exemplary scenario, the at least one processor can autonomously determine a random position within the virtual environment, and randomly determine another position within the virtual environment which is within the distance threshold from the random position. These two randomly generated positions are the first and second positions, and can be determined in either order (i.e. first position is determined first, or second position is determined first). Further, autonomous determination of positions can be constrained based on features of the virtual environment. For example, random determination of positions can be limited to positions which are on roadways of the virtual environment. Further, random determinations of first and second positions of two vehicles can be constrained to positions in a same lane of a roadway of the virtual environment.” Mazumder col. 24 lines 12-26.
Tobin teaches the asset could be mesh asset, stating “We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator.” Tobin IV.A. Experimental Setup.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mazumder’s simulation of moving vehicles with primary reference Tobin. One of ordinary skill in the art would be motivated to train machine learning models to identify vehicle position and movements. Mazumder states, “The present disclosure details systems and methods for creating training data, for training machine learning models, and for applying machine learning models, for identifying vehicle movement and positioning. The present disclosure sees particular value in detecting travel lane of vehicles, determining distance between vehicles, and identifying when a vehicle is tailgating another vehicle.” Mazumder col. 11 lines 24-30.
Regarding Claim 5, Tobin in view of Mazumder teaches The processor of claim 1,
wherein the first data selected for the first semantic layer of the 3D mesh asset comprises at least one of a color, a pattern, a texture, or a material (
Tobin:
PNG
media_image3.png
486
426
media_image3.png
Greyscale
, which at least show the selected data include color and texture.
Tobin teaches the asset could be mesh asset, stating “We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator.” Tobin IV.A. Experimental Setup.).
Regarding Claim 6, Tobin in view of Mazumder teaches The processor of claim 1, wherein the one or more circuits are to
generate a label for the image based at least on an aspect of the 3D mesh asset within the scene (
“We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator. Each training sample consists of (a) a rendered image of the object and one or more distractors (also from among the geometric object set) on a simulated tabletop and (b) a label corresponding to the Cartesian coordinates of the center of mass of the object in the world frame.” Tobin IV.A. Experimental Setup.).
Regarding Claim 7, Tobin in view of Mazumder teaches The processor of claim 1, wherein the one or more circuits are to determine a pose for the 3D mesh asset according to the configuration (
[BRI on record]
With respect to “a pose for the model,” the Examiner is reading the limitation to mean: position and orientation for the model. Spec. ¶ 58.
[0058] The scene generation process 214 may include arranging one or more 3D models placed within the scene according to animations specified via the parameter file 202. The animations may be applied to 3D models of entities in the scene through rigging, which includes posing a skeleton or wire rig of the 3D model according to one or more keyframes. Keyframes include points in time at which the joints and segments of the skeleton/rig applied to the model are in a specific pose. When a skeleton/rig is applied to a 3D model, the vertices of the 3D model are assigned to corresponding segments of the skeleton/rig, causing the 3D model to pose according to the positions and orientations of the segments specified in the keyframe. Example animations of people may include standing, sitting, walking, running, or typing on a keyboard, among others.
[Mapping Analysis]
PNG
media_image2.png
454
414
media_image2.png
Greyscale
, showing that the position and the orientation of the table and objects and those with respect to the viewpoint are determined and rendered according to the configuration for the scene.
Tobin teaches the asset could be mesh asset, stating “We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator.” Tobin IV.A. Experimental Setup.).
Regarding Claim 8, Tobin in view of Mazumder teaches The processor of claim 7, wherein the one or more circuits are to determine the pose by simulating an animation selected for the 3D mesh asset according to the configuration (
Mazumder teaches simulation animation according to configuration, stating “In order to render such data, acts 512 and 812 in methods 500 and 800 entail simulating movement of the first vehicle (vehicle 2020 in FIGS. 20A-20C) and the virtual camera over each respective moment in time represented by the plurality of images for the instance.” Mazumder col. 31 lines 20-29. Figs. 20A-C.
PNG
media_image4.png
338
496
media_image4.png
Greyscale
PNG
media_image5.png
340
498
media_image5.png
Greyscale
Here, the vehicle model has a determined pose, including position and orientation.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mazumder’s simulation of moving vehicles with primary reference Tobin. One of ordinary skill in the art would be motivated to train machine learning models to identify vehicle position and movements. Mazumder states, “The present disclosure details systems and methods for creating training data, for training machine learning models, and for applying machine learning models, for identifying vehicle movement and positioning. The present disclosure sees particular value in detecting travel lane of vehicles, determining distance between vehicles, and identifying when a vehicle is tailgating another vehicle.” Mazumder col. 11 lines 24-30.
Regarding Claim 9, Tobin inv view of Mazumder teaches The processor of claim 1, wherein the one or more circuits are to position the 3D mesh asset within the scene relative to a viewpoint used to generate the image (
Tobin:
PNG
media_image3.png
486
426
media_image3.png
Greyscale
. Note the position of the objects can be randomized. The field of view could also be randomized.
Tobin teaches the asset could be mesh asset, stating “We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator.” Tobin IV.A. Experimental Setup.).
Regarding Claim 11, Tobin in view of Mazumder teaches The processor of claim 1, wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations (Tobin et al. (“Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”). );
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for performing generative AI operations using a large language model (LLM);
a system for generating synthetic data (Tobin et al. (“Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”). The simulated data could be mapped to synthetic data.);
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
Regarding Claims 18-19, they are substantially similar to Claims 1-2. They rejection analyses for Claims 1-2 based on Tobin in view of Mazumder are applied to Claims 18-19. In addition, Claims 18-19 recite “A method” and “using one or more processors” (Mazumder col. 42 line 44 – col. 43 line 5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mazumder’s processor and circuits with primary reference Tobin. One of ordinary skill in the art would be motivated to automate/implement Tobin’s algorithm on modern computing technology.
Claims 12 and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Tobin et al. (“Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”) in view of Mazumder et al. (US 11748664 B1) and Pittman (US 20190371196 A1).
Regarding Claim 12, Tobin teaches A processor comprising: one or more circuits to:
generate a synthetic scene including a plurality of three-dimensional (3D) mesh assets positioned according to a configuration
Tobin Figs. 1, 7.
Tobin teaches the asset could be mesh asset, stating “We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator.” Tobin IV.A. Experimental Setup.
Any group of objects on the table, the table, floor, and/or wall may correspond to a 3D mesh asset.),
at least one 3D mesh asset of the plurality of 3D mesh assets comprising a first semantic layer having a first property randomized according to a first distribution specified in the configuration
Tobin teaches first and second property randomized, stating “Bridging the ‘reality gap’ that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator.” Abstract.
PNG
media_image2.png
454
414
media_image2.png
Greyscale
, which shows scenes generated based on the randomization. Tobin explain the domains for the randomization that including attributes that controls the appearance of 3D . . . assets, mapped to Tobin’s table and/or objects placed on the table. These attributes include the texture of objects of the model within the scene.
PNG
media_image3.png
486
426
media_image3.png
Greyscale
Here, Tobin states, “We randomize the following aspects of the domain for each sample used during training: . . ..” Each domain provides a range of options, mapped to claimed “distribution.” The sampling selects an attribute option, e.g., color, for an object on the table, mapped to “semantic layer of a model.” Fig. 1 shows that attributes of each semantic layer, like an object on table, could be controlled individually.
The claimed configuration comprises the settings for abovementioned randomized parameters for color, texture, …. The configuration is received by the system to generate simulated images as shown in figs. 1, 7.
Tobin teaches the asset could be mesh asset, stating “We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator.” Tobin IV.A. Experimental Setup.); and
render the synthetic scene to generate an image for updating a neural network (
Tobin teaches using generated images for training, stating “Bridging the ‘reality gap’ that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator.” Abstract.
The training is for deep neural networks, and Tobin’s title states, “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World.”).
Tobin does not explicitly disclose store the configuration in a file, and simulate movement of the at least one 3D mesh asset within the synthetic scene.
Mazumder teaches
simulate movement of the at least one 3D mesh asset within the synthetic scene (
“In an exemplary scenario, the at least one processor can autonomously determine a random position within the virtual environment, and randomly determine another position within the virtual environment which is within the distance threshold from the random position. These two randomly generated positions are the first and second positions, and can be determined in either order (i.e. first position is determined first, or second position is determined first). Further, autonomous determination of positions can be constrained based on features of the virtual environment. For example, random determination of positions can be limited to positions which are on roadways of the virtual environment. Further, random determinations of first and second positions of two vehicles can be constrained to positions in a same lane of a roadway of the virtual environment.” Mazumder col. 24 lines 12-26.
Tobin already teaches 3D mesh asset, stating “We evaluated our approach by training object detectors for each of eight geometric objects. We constructed mesh representations for each object to render in the simulator.” Tobin IV.A. Experimental Setup; figs. 1, 7.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mazumder’s simulation of moving vehicles with primary reference Tobin. One of ordinary skill in the art would be motivated to train machine learning models to identify vehicle position and movements. Mazumder states, “The present disclosure details systems and methods for creating training data, for training machine learning models, and for applying machine learning models, for identifying vehicle movement and positioning. The present disclosure sees particular value in detecting travel lane of vehicles, determining distance between vehicles, and identifying when a vehicle is tailgating another vehicle.” Mazumder col. 11 lines 24-30.
Tobin in view of Mazumder does not explicitly teach storing the configuration in a file
Pittman teaches storing information in a file (
“A memory device 70 including the memory 58 and storage 60 may also store one or more data files 72 that include information that may be used by the processor 56 when executing the computer-executable instructions for each software modules 68.” Pittman ¶ 34.
“The virtual simulation module 74 generates the virtual environment 24 including the simulated heavy equipment vehicle 26 using the object render files and vehicle attribute files stored in the memory device 70.” Pittman ¶ 36.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Pittman’s teaching of use of files with Tobin in view of Mazumder. One of ordinary skill in the art would be motivated to organize information for easier/quicker access of the information. It also improves the information organization. “A memory device 70 including the memory 58 and storage 60 may also store one or more data files 72 that include information that may be used by the processor 56 when executing the computer-executable instructions for each software modules 68.” Pittman ¶ 34.
Regarding Claim 14, Tobin in view of Mazumder and Pittman teaches The processor of claim 12, wherein the one or more circuits are to
simulate movement of the at least one 3D mesh asset by simulating a collision between the at least one 3D mesh asset and a second 3D mesh asset of the plurality of 3D mesh assets within the synthetic scene (
“In other aspects of the present invention, the methods of operating a heavy equipment simulation system may also include a method of simulating a physical interaction in a simulator. The method includes identifying simulated physical contact zones between two or more simulated objects and defining one or more interactions in the contact zones separate from simulated environmental or equipment object rules.” Pittman ¶ 12.
The claimed “collision” is mapped to the simulation contact between simulated objects.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Pittman’s simulation of object collision with Tobin in view of Mazumder. One of ordinary skill in the art would be motivated to train a machine operator in the virtual simulated environment to improve safety. Pittman discloses “Safety: The control system 22 emphasizes safety while operating the equipment to increase the Student's knowledge level and awareness of jobsite safety. The simulator system 10 tracks numerous safety violations during training lessons. Any time a safety violation is detected, a message will be displayed indicating the type of violation and any other relevant information.” Pittman ¶ 133.
Regarding Claim 15, Tobin in view of Mazumder and Pittman teaches The processor of claim 12, wherein the one or more circuits are to simulate movement of the at least one 3D mesh asset by adjusting the at least one 3D mesh asset according to an animation (“. . . autonomous determination of positions can be constrained based on features of the virtual environment. For example, random determination of positions can be limited to positions which are on roadways of the virtual environment. Further, random determinations of first and second positions of two vehicles can be constrained to positions in a same lane of a roadway of the virtual environment.” Mazumder col. 24 lines 12-26.
Mazumder teaches simulation animation, stating “In order to render such data, acts 512 and 812 in methods 500 and 800 entail simulating movement of the first vehicle (vehicle 2020 in FIGS. 20A-20C) and the virtual camera over each respective moment in time represented by the plurality of images for the instance.” Mazumder col. 31 lines 20-29.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mazumder’s simulation of moving vehicles with primary reference Tobin. One of ordinary skill in the art would be motivated to train machine learning models to automatically identify vehicle position and movements.
Regarding Claim 16, Tobin in view of Mazumder and Pittman teaches The processor of claim 15, wherein the one or more circuits are to select an animation frame of the animation according to the configuration file (
“When rendering images (as in act 513 of method 500 or act 813 of method 800), technical effects can also be simulated, which results in visible changes in the image data. Alternatively, subsets of images can be selected, and technical distortion effects applied after the image data is rendered, as discussed below with reference to FIG. 19.” Mazumder col. 28 lines 24-29.
Each image of the subset is a frame. Whether, how, and when to select the subsets are based on configuration of a system based on Tobin in view of Mazumder and Pittman. Pittman already teaches that the configuration could be saved in a file.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mazumder’s simulation of moving vehicles with primary reference Tobin. One of ordinary skill in the art would be motivated to train machine learning models to automatically identify vehicle position and movements.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Pittman’s teaching of use of files with Tobin in view of Mazumder. One of ordinary skill in the art would be motivated to organize information for easier/quicker access of the information. It also improves the information organization.
Regarding Claim 17, Tobin in view of Mazumder and Pittman teaches The processor of claim 12, wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations; a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations (Tobin et al. (“Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”).);
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for performing generative AI operations using a large language model (LLM);
a system for generating synthetic data (Tobin et al. (“Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”). The simulated data could be mapped to synthetic data.);
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Tobin in view of Mazumder and Pittman as applied to Claim 12, in further view of Shuster et al. (US 20160140752 A1).
Regarding Claim 13, Tobin in view of Mazumder and Pittman teaches The processor of claim 12.
Tobin in view of Mazumder and Pittman does not explicitly teach wherein the one or more circuits are to simulate movement of the at least one 3D mesh asset by simulating a gravitational force within the synthetic scene.
Shuster teaches wherein the one or more circuits are to
simulate movement of the at least one 3D mesh asset by simulating a gravitational force within the synthetic scene (
“In alternate embodiments, the scene may include instructions that randomize or otherwise alter the nature of each new instance.” Shuster ¶ 25.
“In order for movement of objects in the instance of the scene to appear realistic, calculations are performed to simulate physical conditions consistent with the virtual world on the objects. Physical conditions may be simulated by spawning objects and determining paths for objects through an instance of a scene that avoid other objects, applying gravitational and drag forces, applying frictional forces, determining collisions between objects and accounting for elasticity of contacting surfaces when collisions occur between objects..” Shuster ¶ 28.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shuster’s simulation based on gravitational force with Tobin in view of Mazumder and Pittman . One of ordinary skill in the art would be motivated to accurately estimate object motion path according to physical rules. “Physical conditions may be simulated by spawning objects and determining paths for objects through an instance of a scene that avoid other objects, applying gravitational and drag forces, applying frictional forces, determining collisions between objects and accounting for elasticity of contacting surfaces when collisions occur between objects..” Shuster ¶ 28.
Claims 4 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tobin in view of Mazumder as applied to Claims 3, 19, in further view of Shuster et al. (US 20160140752 A1).
Regarding Claim 4, Tobin in view of Mazumder teaches The processor of claim 3, wherein the 3D mesh asset is a first 3D mesh asset, the scene is generated to include a second 3D mesh asset (“For example, random determination of positions can be limited to positions which are on roadways of the virtual environment. Further, random determinations of first and second positions of two vehicles can be constrained to positions in a same lane of a roadway of the virtual environment.” Mazumder col. 24 lines 12-26.)
Tobin in view of Mazumder does not explicitly disclose the one or more circuits are to: simulate a collision between the first 3D mesh asset and the second 3D mesh asset.
Shuster teaches the one or more circuits are to: simulate a collision between the first 3D mesh asset and the second 3D mesh asset (
“In alternate embodiments, the scene may include instructions that randomize or otherwise alter the nature of each new instance.” Shuster ¶ 25.
“In order for movement of objects in the instance of the scene to appear realistic, calculations are performed to simulate physical conditions consistent with the virtual world on the objects. Physical conditions may be simulated by spawning objects and determining paths for objects through an instance of a scene that avoid other objects, applying gravitational and drag forces, applying frictional forces, determining collisions between objects and accounting for elasticity of contacting surfaces when collisions occur between objects..” Shuster ¶ 28.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shuster’s simulation based on gravitational force with Tobin in view of Mazumder. One of ordinary skill in the art would be motivated to accurately estimate object motion path according to physical rules. “Physical conditions may be simulated by spawning objects and determining paths for objects through an instance of a scene that avoid other objects, applying gravitational and drag forces, applying frictional forces, determining collisions between objects and accounting for elasticity of contacting surfaces when collisions occur between objects..” Shuster ¶ 28.
Regarding Claim 20, Tobin in view of Mazumder teaches The method of claim 19.
Tobin in view of Mazumder does not explicit disclose
further comprising updating, by using the one or more processors, a position of the 3D mesh asset in the scene according to a physical simulation.
Shuster teaches further comprising updating, by using the one or more processors, a position of the 3D mesh asset in the scene according to a physical simulation (
“In alternate embodiments, the scene may include instructions that randomize or otherwise alter the nature of each new instance.” Shuster ¶ 25.
“In order for movement of objects in the instance of the scene to appear realistic, calculations are performed to simulate physical conditions consistent with the virtual world on the objects. Physical conditions may be simulated by spawning objects and determining paths for objects through an instance of a scene that avoid other objects, applying gravitational and drag forces, applying frictional forces, determining collisions between objects and accounting for elasticity of contacting surfaces when collisions occur between objects..” Shuster ¶ 28.
The disclosed path is constructed based on positions of the 3D mesh asset. There is a physical simulation, because the simulation is based on physical rules as described in Shuster ¶ 28. ).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shuster’s simulation based on gravitational force with Tobin in view of Mazumder. One of ordinary skill in the art would be motivated to accurately estimate object motion path. “Physical conditions may be simulated by spawning objects and determining paths for objects through an instance of a scene that avoid other objects, applying gravitational and drag forces, applying frictional forces, determining collisions between objects and accounting for elasticity of contacting surfaces when collisions occur between objects..” Shuster ¶ 28.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Tobin in view of Mazumder as applied to Claim 1, in further view of Jeong et al. (US 20190215479 A1).
Regarding Claim 10, Tobin in view of Mazumder teaches The processor of claim 1,
wherein the one or more circuits are to: generate a plurality of scenes according to the configuration (Tobin figs. 1, 7);
generate a plurality of images using the plurality of scenes (Tobin figs. 1, 7); and
filter the plurality of images based at least on
“For example, random determination of positions can be limited to positions which are on roadways of the virtual environment. Further, random determinations of first and second positions of two vehicles can be constrained to positions in a same lane of a roadway of the virtual environment.” Mazumder col. 24 lines 12-26.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mazumder’s simulation of moving vehicles with primary reference Tobin. One of ordinary skill in the art would be motivated to train machine learning models to identify vehicle position and movements. Moving vehicles are often compliant with traffic rules, which is a realistic situation. “For example, random determination of positions can be limited to positions which are on roadways of the virtual environment. Further, random determinations of first and second positions of two vehicles can be constrained to positions in a same lane of a roadway of the virtual environment.” Mazumder col. 24 lines 12-26.
Tobin in view of Mazumder does not explicitly disclose filtering the plurality of images based on an illumination of the plurality of scenes.
Jeong teaches filtering the plurality of images based on an illumination of the plurality of scenes (
“In the ‘dense fog’, if an image filtering algorithm such as the DCP algorithm is applied to the black-and-white image output from the ISP 14, an image with clearness enough to identify an target object can be generated, but even if the image filtering algorithm is applied to the black-and-white image output from the ISP 14 in the ‘nighttime status’, the black-and-white image itself is too dark, and thus, an image with clearness enough to identify the target object may not be generated. Accordingly, in the present embodiment, in order to increase brightness of the black-and-white image output from the ISP 14 in the ‘nighttime status’, a luminance value of each pixel of the black-and-white image output from the ISP 14 is inverted. The software filter 15 subtracts the luminance value of each pixel of the black-and-white image output from the ISP 14 from a maximum luminance value and sets the subtraction result as the luminance value of each pixel of the black-and-white image output from the ISP 14, thereby, inverting the luminance value of each pixel of the black-and-white image output from the ISP 14. In a case where the entire range of the luminance values of the respective pixels is expressed as levels of 0 to 255, the maximum luminance value can be 255.” Jeong ¶ 49.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jeong’s image filtering with Tobin in view of Mazumder. One of ordinary skill in the art would be motivated to enhance the visibility and/or recognizability in an image. Jeong states, “an image quality of the black-and-white image generated from light of the infrared light band emitted from the lens unit 11 is better than an image quality of the color image generated from light of the visible light band emitted from the lens unit 11. Moreover, in a case where an image filtering algorithm such as the DCP algorithm is applied to the black-and-white image generated from the light in of the infrared light band emitted from the lens unit 11, the image quality can be further improved.” Jeong ¶ 48.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Tremblay et al. (Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization)
PNG
media_image6.png
390
764
media_image6.png
Greyscale
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENGXI LIU whose telephone number is (571)270-7509. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 571-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENGXI LIU/Primary Examiner, Art Unit 2611