Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 21-40 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-21 of U.S. Patent No. US 12190448B2
Although the conflicting claims are not identical, they are not patentably distinct from each other because the Patent US 12190448B2 contains substantially all the limitations of the instant application claims.
Application No. 19/010340
Patent No. US 12190448 B2
21. (New) A method comprising: generating, based at least on a simulated environment, an input three-dimensional (3D) representation of a surface structure of a simulated road; generating, based at least on the simulated environment, a ground truth 3D representation of the surface structure of the simulated road and a segmentation mask representing the simulated road; and training one or more neural networks (NNs) using the input 3D representation and the ground truth 3D representation of the surface structure as training data, wherein one or more updates corresponding to one or more predicted values of one or more regions outside the simulated road are discarded using the segmentation mask.
1. A method comprising:
accessing simulated data comprising simulated image data and at least one of classification data corresponding to the simulated image data or range data corresponding to the simulated image data;
generating, based at least on the simulated image data, a first three-dimensional (3D) representation of a surface structure of a simulated road represented by the simulated image data;
generating, based at least on the at least one of the classification data corresponding to the simulated image data or the range data corresponding to the simulated image data, a second 3D representation of the surface structure of the simulated road; and
training one or more neural networks (NNs) to generate a densified 3D representation of the surface structure using the first 3D representation of the surface structure as input training data and using the second 3D representation of the surface structure as ground truth training data.
2. The method of claim 1, wherein the second 3D representation of the surface structure of the simulated road comprises a two-dimensional (2D) ground truth height map representing one or more height values of the simulated road, wherein the classification data comprises a ground truth segmentation mask demarcating the simulated road, wherein the training of the one or more NNs comprises comparing a predicted 2D height map with the 2D ground truth height map and applying the ground truth segmentation mask to cancel out updates to the one or more NNs based on one or more values of the predicted 2D height map that do not correspond to a surface of the simulated road.
31. (New) One or more processors comprising processing circuitry to: generate, based at least on a simulated environment, an input three-dimensional (3D) representation of surface structure height of a simulated road; generate, based at least on the simulated environment, a ground truth 3D representation of the surface structure height of the simulated road and a segmentation mask representing the simulated road; and train one or more neural networks (NNs) using the input 3D representation and the ground truth 3D representation of the surface structure height as training data, wherein one or more updates corresponding to one or more predicted values of one or more regions outside the simulated road are discarded using the segmentation mask.
10. A processor comprising one or more circuits to:
generate simulated image data representing a simulated environment;
generate, based at least on the simulated image data, a first three-dimensional (3D) representation of a surface structure of a simulated road represented by the simulated image data;
generate, based at least on a representation of the simulated environment, a ground truth 3D representation of the surface structure of the simulated road; and
train one or more neural networks (NNs) to generate a densified 3D representation of the surface structure using the first 3D representation of the surface structure as input training data and using the ground truth 3D representation of the surface structure as ground truth training data.
11. The processor of claim 10, wherein the ground truth 3D representation of the surface structure of the simulated road comprises a two-dimensional (2D) ground truth height map representing one or more height values of the simulated road, wherein the representation of the simulated environment comprises a ground truth segmentation mask demarcating the simulated road, the one or more circuits further to train the one or more NNs based at least on comparing a predicted 2D height map with the 2D ground truth height map and applying the ground truth segmentation mask to cancel out updates to the one or more NNs based on values of the predicted 2D height map that do not correspond to a surface of the simulated road.
Dependent claims 22-30, 32-40 recites similar matter as claim 2-9, 11-21 of Patent US 12190448 B2 and are rejected for the same reason.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 21-22, 24, 29-32, 34, 36-38, 40 are rejected under 35 U.S.C. 103 as being unpatentable over Avidan et al. (US Pub 2019/0205667 A1) in view of Liu et al. (US Pub 2021/0150203 A1).
As to claim 21, Avidan discloses a method comprising: generating, based at least on a simulated environment, an input three-dimensional (3D) representation of a surface structure of a simulated road (Fig. 5, ¶0037, “multiple road geometries and features can be randomly generated by varying a set of randomizable parameters.” ¶0038, “the CNN or equivalent machine learning model can be trained using a synthetic dataset generated based on the geometry and other map-stored characteristics of a specific road, route, or location. In this way, a vehicle 101 could have further, previously unavailable insights of e.g. safety hazards specific to a route.”);
generating, based at least on the simulated environment, a ground truth 3D representation of the surface structure of the simulated road and a (Fig. 5, ¶0031, “a labeled dataset is image data (e.g., image sequences) that has been annotated with one or more labels that represent ground truth classes for the depicted action or dynamic movement.” ¶0034, “generate synthetic image data (e.g., image sequences or videos) based on user defined parameters. In one embodiment, the parameters can describe the types of actions or movements that synthetic image data is to depict. The system can then use the parameters to automatically generate and label the synthetic image data. This labeled synthetic data can be then be used for training or evaluating machine learning models (e.g., CNNs or equivalent) to predict or detect actions or dynamic movements of objects in input image sequences or videos.”); and
training one or more neural networks (NNs) using the input 3D representation and the ground truth 3D representation of the surface structure as training data (¶0034, “generate synthetic image data (e.g., image sequences or videos) based on user defined parameters. In one embodiment, the parameters can describe the types of actions or movements that synthetic image data is to depict. The system can then use the parameters to automatically generate and label the synthetic image data. This labeled synthetic data can be then be used for training or evaluating machine learning models (e.g., CNNs or equivalent) to predict or detect actions or dynamic movements of objects in input image sequences or videos.”),
wherein one or more updates corresponding to one or more predicted values of one or more regions outside the simulated road are discarded using the (¶0045, “the parameter module 201 selects the geographic area based on an area of interest, an origin-destination pair, a navigation route, a road geometry, an accident rate, or a combination thereof.” “the user can specify attributes (e.g., area of interest, origin-destination pair, route, road geometry, etc.) that can be used by the parameter module 201 to determine the appropriate geographic area. In one embodiment, the geographic area can be a real-world location or a generic location that is modeled to include the specified attributes. For example, in cases where real-world locations are used, the image generator 203 queries a geographic database for map data associated with the geographic area. The image generator 203 then renders the geographic area in the computer-generated image sequence based on the map data. The map data, for instance, can include HD surface maps of the terrain and/or locations of geographic features (e.g., roads, road objects, points of interest, etc.) that can be used to render a 3D rendering of the location in the synthetic image data.” ¶0052, “parameters describing other objects to include in the synthetic image data can be specified. These other object parameters can indicate the types of objects as well as their behavior, appearance, and/or other characteristics. These other objects can include, but are not limited to: (1) animals (e.g., wild, domestic, etc.), (2) road debris, (3) road signs, (4) road objects (e.g., lane markings, guard rails, sidewalks, etc.), (5) nearby structures; (6) nearby terrain; and/or the like.” ¶0059, “Depending on the user defined parameters, the 3D world generated by the synthetic image generator 203 can be complemented using random scenery objects such as buildings, trees, road furniture (signs, lights, etc.), parked vehicles, advertisement posters, etc. Such objects can also be obtained from a map containing 3D information, such as the geographic database 109.”).
Avidan does not explicitly disclose the word segmentation mask. However, it is quite obvious the “parameter module” can be a segmentation mask to determine the area of interest.
Liu teaches segmentation mask (Liu, ¶0018, “Semantic segmentation involves Image Classification to recognize objects and existing properties in an image; and Image Segmentation, where exactly what is in the image can be recognize and understood at a pixel level view.” ¶0073, “The CNN can take as input a perspective image with occluded regions (corresponding to foreground objects) masked out, and estimate the segmentation labels and depth values over the entire image.” ¶0075, “given the semantic segmentation, a mask of foreground pixels can be defined, where a pixel in the mask is 1 if and only if the segmentation at that pixel belongs to any of the foreground classes. Otherwise, the pixel in the mask is 0. In order to inform the CNN about which pixels have to be in-painted, we apply the mask on the input RGB image and define each pixel in the masked input.”).
Avidan and Liu are considered to be analogous art because all pertain to generate simulated road. It would have been obvious before the effective filing date of the claimed invention to have modified Avidan with the features of “segmentation mask” as taught by Liu. The suggestion/motivation would have been in order to estimate the segmentation labels and depth values over the entire image (Liu, ¶0073).
As to claim 22, claim 21 is incorporated and the combination of Avidan and Liu discloses the ground truth 3D representation of the surface structure of the simulated road comprises a two-dimensional (2D) ground truth height map representing one or more height values of the simulated road (Liu, ¶0113, “a feature map F=gi(x*)∈Rhf×wf×cf, where hf, wf, and cf denote the height, width and feature dimension that can be extracted after gi.” ¶0114, “Then both gftm(gi(xt−1)) and gi(xt) can be inputted into fcom to output a final feature, which can then be processed and subsequently the feature inputted into an LSTM module.”),
wherein the training of the one or more NNs comprises comparing a predicted 2D height map with the 2D ground truth height map and applying the segmentation mask to discard the one or more updates (Liu, ¶0075, “given the semantic segmentation, a mask of foreground pixels can be defined, where a pixel in the mask is 1 if and only if the segmentation at that pixel belongs to any of the foreground classes. Otherwise, the pixel in the mask is 0. In order to inform the CNN about which pixels have to be in-painted, we apply the mask on the input RGB image and define each pixel in the masked input.”).
As to claim 24, claim 21 is incorporated and the combination of Avidan and Liu discloses the generating of the input 3D representation of the surface structure of the simulated road comprises: generating, based at least on applying 3D structure estimation to simulated image data generated based at least on the simulated environment, a point cloud representation of the simulated image data; and projecting the point cloud representation to generate the input 3D representation of the surface structure of the simulated road (Liu, ¶0076, “a top view representation of a road can be generated. A top-view image of the road can be generated by (1) back-projecting all the road pixels into a 3D point cloud and (2) projecting all the 3D points onto the x-y plane. This mapping can be used to transfer the semantic class probability distribution of each pixel from perspective-view into top-view (bird's eye view (“bev”)), and is referred to as bev and denote as xr∈Figure H×W×C, where C is the number of semantic classes, and in various embodiments H=128 and W=64 pixels, for example, which can relate to 60×30 meters in the point cloud in a non-limiting exemplary embodiment.”).
As to claim 29, claim 21 is incorporated and the combination of Avidan and Liu discloses using the one or more NNs to perform at least one of: adapting a suspension system of an ego-object, navigating the ego-object to avoid a detected protuberance, or applying an acceleration or deceleration to the ego-object based at least on a detected surface slope (Avidan, ¶0036, “involving dynamic movement for training a CNN detector (or any other equivalent machine learning algorithm). The dataset is synthetic, i.e., generated using any rendering and/or physics engine or method known in the art. By way of example, such engines include, but are not limited to, driving simulators, game engines, and/or the like. In one embodiment, the engines provide for simulating the physics of objects, rendering a video clip, collision detection, artificial intelligence (e.g., to simulate driving behavior, pedestrian behavior, animal behavior, etc.), and/or the like.” ¶0037, “any action or situation involving a dynamic movement by one or more objects (e.g., vehicles, pedestrians, etc.) can also be simulated and automatically labeled in the same manner (e.g. dangerous driving behavior of cars nearby, potential dangerous trajectory of a pedestrian/vehicle nearby, etc.)”).
As to claim 30, claim 21 is incorporated and the combination of Avidan and Liu discloses the method is performed by at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (Avidan, ¶0036-0037, ¶0040).
As to claim 31, the combination of Avidan and Liu discloses one or more processors comprising processing circuitry to: generate, based at least on a simulated environment, an input three-dimensional (3D) representation of surface structure height of a simulated road; generate, based at least on the simulated environment, a ground truth 3D representation of the surface structure height of the simulated road and a segmentation mask representing the simulated road; and train one or more neural networks (NNs) using the input 3D representation and the ground truth 3D representation of the surface structure height as training data, wherein one or more updates corresponding to one or more predicted values of one or more regions outside the simulated road are discarded using the segmentation mask (See claim 21 for detailed analysis.).
As to claim 32, claim 31 is incorporated and the combination of Avidan and Liu discloses the ground truth 3D representation of the surface structure height of the simulated road comprises a two-dimensional (2D) ground truth height map representing one or more height values of the simulated road, and the one or more processors are further to train the one or more NNs based at least on comparing a predicted 2D height map with the 2D ground truth height map and applying the segmentation mask to discard the one or more updates (See claim 22 for detailed analysis.).
As to claim 34, claim 31 is incorporated and the combination of Avidan and Liu discloses the generating of the input 3D representation of the surface structure height of the simulated road comprises: generating, based at least on applying 3D structure estimation to simulated image data generated based at least on the simulated environment, a point cloud representation of the simulated image data; and projecting the point cloud representation to generate the input 3D representation of the surface structure height of the simulated road (See claim 24 for detailed analysis.).
As to claim 36, claim 31 is incorporated and the combination of Avidan and Liu discloses the one or more processors are comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (See claim 30 for detailed analysis.).
As to claim 37, the combination of Avidan and Liu discloses a system comprising: one or more processors to perform one or more operations using a neural network, wherein the neural network is trained i) using a dataset that comprises one or more input three-dimensional (3D) representations of one or more surface structures of a simulated road, one or more ground truth 3D representations of the surface structure of the simulated road, and one or more segmentation masks representing the simulated road (See claim 21 for detailed analysis.),
and ii) using a loss function that cancels one or more updates to the neural network corresponding to one or more predicted values of one or more regions outside the simulated road based at least on the one or more segmentation masks (Liu, ¶0074, “an adversarial loss can be employed for teaching the neural network (e.g., convolutional neural network (CNN)) a generative aspect about road layouts.” ¶0143, “The loss function can be used to compare the predictions of the scene attributes/parameters (theta—output of network h) with the ground truth (referring to information provided by direct observation (i.e. empirical evidence)). We have a data set of many pairs (input images and corresponding scene attributes/parameters). The loss function returns a single scalar that corresponds to how much error out neural network makes. This error/loss should be minimized (via gradient-based optimization here). This is standard supervised learning for neural networks.”).
As to claim 38, claim 37 is incorporated and the combination of Avidan and Liu discloses the neural network is implemented in at least one processing node of a plurality of processing nodes of a data center and accessible to one or more remote clients via at least one of an application programming interface (API), or an application plug-in (Avidan, ¶0036-0037, ¶0040).
As to claim 40, claim 37 is incorporated and the combination of Avidan and Liu discloses the one or more processors are comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (Avidan, ¶0036-0037, ¶0040).
Claims 26-28 are rejected under 35 U.S.C. 103 as being unpatentable over Avidan et al. (US Pub 2019/0205667 A1) in view of Liu et al. (US Pub 2021/0150203 A1) and Ren (US Pub 2022/0300681 A1).
As to claim 26, claim 21 is incorporated and the combination of Avidan and Liu does not disclose the generating of the ground truth 3D representation of the surface structure of the simulated road comprises filling in one or more missing values using triangulation.
Ren teaches the generating of the ground truth 3D representation of the surface structure of the simulated road comprises filling in one or more missing values using triangulation (Ren, ¶0081, “Conventional surface generation methods using polygon meshes to represent surfaces, for example greedy surface triangulation and Delaunay triangulation algorithms, yield a surface consisting of a polygon mesh with holes, which may result in scan lines missing points in an area corresponding to a hole and to points appearing in the shadow area of the surface during scanline and shadow generation (described below with reference to FIG. 5).”).
Avidan, Liu and Ren are considered to be analogous art because all pertain to generate simulated road. It would have been obvious before the effective filing date of the claimed invention to have modified Avidan with the features of “the generating of the ground truth 3D representation of the surface structure of the simulated road comprises filling in one or more missing values using triangulation” as taught by Ren. The claim would have been obvious because the technique for improving a particular class of devices was part of the ordinary capabilities of a person of ordinary skill in the art, in view of the teaching of the technique for improvement in other situations.
As to claim 27, claim 21 is incorporated and the combination of Avidan, Liu and Ren the generating of the ground truth 3D representation of the surface structure of the simulated road comprises: projecting a set of 3D points to form a first two-dimensional (2D) height map representing one or more height values of the simulated road (Liu, ¶0076, “a top view representation of a road can be generated. A top-view image of the road can be generated by (1) back-projecting all the road pixels into a 3D point cloud and (2) projecting all the 3D points onto the x-y plane. This mapping can be used to transfer the semantic class probability distribution of each pixel from perspective-view into top-view (bird's eye view (“bev”)), and is referred to as bev and denote as xr∈Figure H×W×C, where C is the number of semantic classes, and in various embodiments H=128 and W=64 pixels, for example, which can relate to 60×30 meters in the point cloud in a non-limiting exemplary embodiment.”);
performing 2D Delaunay triangulation in the first 2D height map to generate one or more triangles; and sampling a 2D ground truth height map from the one or more triangles (Ren, ¶0081, “Conventional surface generation methods using polygon meshes to represent surfaces, for example greedy surface triangulation and Delaunay triangulation algorithms, yield a surface consisting of a polygon mesh with holes, which may result in scan lines missing points in an area corresponding to a hole and to points appearing in the shadow area of the surface during scanline and shadow generation (described below with reference to FIG. 5).”).
As to claim 28, claim 21 is incorporated and the combination of Avidan, Liu and Ren the generating of the ground truth 3D representation of the surface structure of the simulated road comprises: identifying a set of 3D points that correspond to the surface structure of the simulated road (Liu, ¶0076, “a top view representation of a road can be generated. A top-view image of the road can be generated by (1) back-projecting all the road pixels into a 3D point cloud and (2) projecting all the 3D points onto the x-y plane. This mapping can be used to transfer the semantic class probability distribution of each pixel from perspective-view into top-view (bird's eye view (“bev”)), and is referred to as bev and denote as xr∈Figure H×W×C, where C is the number of semantic classes, and in various embodiments H=128 and W=64 pixels, for example, which can relate to 60×30 meters in the point cloud in a non-limiting exemplary embodiment.”);
performing 3D Delaunay triangulation to compute a surface mesh of one or more triangles surrounding the set of 3D points; sampling 3D points from the one or more triangles of the surface mesh; and projecting the 3D points to form a 2D ground truth height map (Ren, ¶0081, “Conventional surface generation methods using polygon meshes to represent surfaces, for example greedy surface triangulation and Delaunay triangulation algorithms, yield a surface consisting of a polygon mesh with holes, which may result in scan lines missing points in an area corresponding to a hole and to points appearing in the shadow area of the surface during scanline and shadow generation (described below with reference to FIG. 5).”).
Claims 39 is rejected under 35 U.S.C. 103 as being unpatentable over Avidan et al. (US Pub 2019/0205667 A1) in view of Liu et al. (US Pub 2021/0150203 A1) and Lasram et al. (US Pub 2022/0366638 A1).
As to claim 39, claim 37 is incorporated and the combination of Avidan and Liu does not disclose generate the one or more input 3D representations, the one or more ground truth 3D representations, and the one or more segmentation masks based at least on controlling, within a simulation rendered using one or more ray-tracing algorithms, one or more actions of a simulated object in a simulated environment.
Lasram teaches generate the one or more input 3D representations, the one or more ground truth 3D representations, and the one or more segmentation masks based at least on controlling, within a simulation rendered using one or more ray-tracing algorithms, one or more actions of a simulated object in a simulated environment (Lasram, ¶0031, “the sensor engine may use ray casting and/or ray tracing techniques to generate a reprojection of the 2D view for a particular sensor in operation 110.” ¶0033, “ray tracing/ray casting may be used to reproject the 2D view in operation 106. For example, ray tracing may be used to generate a point cloud that spatially aligns the simulated sensor and the 2D view at a point in time during the simulation. In addition, during the execution of the driving simulation, the simulated sensor may move based on the vehicle movement and/or may move independently of the vehicle (e.g., a spinning camera, lidar, or radar sensor). The sensor engine may be configured to cast rays at periodic time intervals after the generation of the 2D view, based on the updated positions and/or orientations of the sensor at the time intervals. Using the rays cast at each time interval, the sensor engine may generate reprojections (e.g., point clouds and/or images) of the 2D view to re-align the 2D view to correspond to the updated sensor position and angle.” ¶0045, “After generating a 2D view of the 3D simulated environment, the reprojection component 226 may be configured to use the GPU 224 to generate a reprojection (e.g., a modified 2D view) that is spatially and/or temporally aligned with a particular simulated sensor operating in the simulation. In some instances, the reprojection component 226 may perform some or all of the operations described above in reference to operation 110. For example, the reprojection component 226 may use ray casting and/or ray tracing to generate a reprojection of the rendered 2D view, by casting rays from the location of the sensor within the simulated environment toward the 2D view to determine ray intersection points. The visual and/or depth data associated with the ray intersection points may be used to align the 2D view, compute shadowing and masking information, and simulate various visual effects. Additionally, depending on the type of simulated sensor for which the reprojection is generated, the reprojection component 226 may generate a point cloud and/or a visual representation of the 2D view.”).
Avidan, Liu and Lasram are considered to be analogous art because all pertain to generate simulated road. It would have been obvious before the effective filing date of the claimed invention to have modified Avidan with the features of “generate the one or more input 3D representations, the one or more ground truth 3D representations, and the one or more segmentation masks based at least on controlling, within a simulation rendered using one or more ray-tracing algorithms, one or more actions of a simulated object in a simulated environment” as taught by Lasram. The claim would have been obvious because the technique for improving a particular class of devices was part of the ordinary capabilities of a person of ordinary skill in the art, in view of the teaching of the technique for improvement in other situations.
Allowable Subject Matter
Claims 23, 25, 33, 35 are dependent upon a rejected base claim, but would be allowable if overcame the obvious type double patenting and rewritten in independent form including all of the limitations of the base claim and any intervening claims.
23. (New) The method of claim 21, wherein the one or more NNs include a first output channel that regresses one or more height values of the simulated road and a second output channel that regresses one or more confidence values corresponding to the height values, wherein the training of the one or more NNs comprises applying a loss function, wherein the loss function compares predicted and ground truth height values and is based on predicted confidence values.
25. (New) The method of claim 21, wherein the generating of the ground truth 3D representation of the surface structure of the simulated road comprises: unprojecting one or more simulated range values generated based at least on a simulated depth map corresponding to the simulated environment into 3D points; selecting a set of the 3D points on the surface structure of the simulated road based at least on the segmentation mask; and generating the ground truth 3D representation of the surface structure of the simulated road as a projection of the set of the 3D points.
33. (New) The one or more processors of claim 31, wherein the one or more NNs include a first output channel that regresses one or more height values of the simulated road and a second output channel that regresses one or more confidence values corresponding to the height values, and the one or more processors are further to train the one or more NNs based at least on applying a loss function, wherein the loss function compares predicted and ground truth height values and is based on predicted confidence values.
35. (New) The one or more processors of claim 31, wherein the generating of the ground truth 3D representation of the surface structure height of the simulated road comprises: unprojecting one or more simulated range values generated based at least on a simulated depth map corresponding to the simulated environment into 3D points;selecting a set of the 3D points on the surface structure height of the simulated road based at least on the segmentation mask; andgenerating the ground truth 3D representation of the surface structure height of the simulated road as a projection of the set of the 3D points.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Dowdall et al. (US Pub 2023/0091924 A1) teaches the depth completion algorithm can be utilized to determine the dense depth values based on intensity values of pixels, and other depth values.
Subbiah (US Patent 11,256,958 B1) teaches training a machine learning model with simulated images.
George et al. (US Pub 2020/0293054 A1) teaches generating training data and training machine learning models.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YU CHEN whose telephone number is (571)270-7951. The examiner can normally be reached on M-F 8-5 PST Mid-day flex.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YU CHEN/Primary Examiner, Art Unit 2613
