DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-24 are currently pending in the present application, with claims 1 and 13 being independent.
Response to Arguments / Amendments
Applicant’s arguments, see pg. 9, filed 02/04/2026, with respect to claims 16, 19, and 24 have been fully considered and are persuasive. The 35 U.S.C. §112(d) rejection of claims 16, 19, and 24 have been withdrawn.
Applicant’s arguments, see pg. 9, filed 02/04/2026, with respect to claims 7 and 24 have been fully considered and are persuasive. The 35 U.S.C. §112(b) rejection of claims 7 and 24 has been withdrawn.
Applicant’s arguments with respect to claim(s) 1-24 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Regarding the remaining arguments: Applicant argues with respect to the amended claim language, which is fully addressed in the prior art rejections set forth below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 7-11, and 13-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vora et al. "Pointpainting: Sequential fusion for 3d object detection." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4604-4612. 2020, hereinafter referred to as Vora, in view of Wekel et al. (WO 2021041854), hereinafter referred to as Wekel, and in further view of Henning et al. (US 20210256738), hereinafter referred to as Henning.
Regarding claim 1, Vora discloses a computer-implemented method for creating a virtual vehicle environment for testing highly automated driving functions of a motor vehicle (Fig. 2-4 and section 2.1; in a robotics or autonomous vehicle system…Abstract; PointPainting: a sequential fusion method…), the method comprising:
providing, from a data memory, pre-acquired camera image data of a camera image and LiDAR point cloud data of a real vehicle environment (Fig. 2 and Section 2-2.3; The PointPainting architecture accepts point clouds and images as input…The image sem. Seg. Network takes in an input image… Algorithm 1…Section 3.1; The KITTI dataset [6] provides synced lidar point clouds and front-view camera images…nuScenes comprises the full autonomous vehicle data suite: synced lidar, cameras and radars with complete 360 coverage…we use lidar point clouds and RGB images from all 6 cameras…)
performing pixel-based classification of the pre-acquired camera image data (Section 2; (1) Semantic Segmentation: an image based sem. seg. network which computes the pixel wise segmentation scores. Section 2.1; The image sem. seg. network takes in an input image and outputs per pixel class scores…local, per pixel classification) using a machine learning algorithm which outputs an associated class and a confidence value corresponding to the classification for each pixel (Section 2.1-2.2; The image sem. seg. network takes in an input image and outputs per pixel class scores…Algorithm 1…Segmentation scores S ∈ RW,H,C with C classes),
projecting the pixel-based classified camera image data onto the pre-acquired LiDAR point cloud data, wherein each point of the LiDAR point cloud, superimposed by classified pixels of the camera image data or points having the same image coordinates, is assigned an identical class (Section 2.2 and Algorithm 1; Each point in the lidar point cloud is (x, y, z, r) or (x, y, z, r, t) for KITTI and nuScenes respectively…The lidar points are transformed by a homogenous transformation followed by a projection into the image…camera matrix, M, projects the points into the image…Once the lidar points are projected into the image, the segmentation scores for the relevant pixel, (h, w), are appended to the lidar point to create the painted lidar point….)
Vora does not disclose instance segmenting the classified LiDAR point cloud data to determine at least one real object comprised by a class.
In the same art of object detection and classification using LiDAR point cloud for autonomous vehicle environments, Wekel discloses instance segmenting the classified LiDAR point cloud data (Pg. 5, lines 23-28; The DNN(s) may process the LiDAR data to compute outputs corresponding to instance segmentation masks, per-class semantic segmentations masks, and/or bounding shapes (e.g., two-dimensional (2D) range image bounding boxes). These outputs may be processed into 2D bounding boxes (e.g., corresponding to the LiDAR range image) and/or three-dimensional (3D) bounding boxes (e.g., corresponding to a LiDAR point cloud used to generate the LiDAR range image) and class labels for the detected objects) to determine at least one real object comprised by a class (Pg. 23, lines 19-23; The instance segmentation mask 130 may include per-pixel values corresponding to unique actor instances detected by the DNN(s) 126. For example, each pixel associated with a first object or actor may have an associated confidence or value that indicates the value 1.0, each pixel associated with a second object or actor may have an associated confidence or value that indicates the value of 2.0, and so on)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the image-based semantic segmentation and LiDAR fusion, as taught by Vora, with the instance segmentation technique of Wekel. Doing so enables separating the semantically labeled point cloud into distinct object instances to improve granularity of the scene. The combination yields predictable results in improved object identification and tracking, virtual simulation, and environment reconstruction in autonomous vehicle perception systems.
Vora in view of Wekel does not disclose selecting and calling a stored, synthetically generated first object corresponding to the at least one real object or procedural generation of a synthetically generated second object corresponding to the at least one real object, and integrating the synthetically generated first object or the synthetically generated second object into a specified virtual vehicle environment.
In the same art of virtual vehicle environment generation using camera-image and LiDAR point-cloud data, Henning discloses selecting and calling a stored (Par. 0025; selecting the identified second feature vector and retrieving a stored synthetic object that is associated with the second feature vector and corresponds to the real object), synthetically generated first object (Par. 0022; providing a plurality of stored second feature vectors representing synthetically generated objects. Par. 0024; The synthetically generated objects are classified into different object categories and represent the objects labeled in the video image data, radar data and/or the lidar point cloud of the real vehicle environment) corresponding to the at least one real object (Par. 0021; generating a first feature vector representing a respective real object by applying a second machine learning algorithm to the respective real object and storing the first feature vector. Par. 0020; The real objects may be static objects, such as traffic signs, buildings, plants, and/or parked motor vehicles. Furthermore, the real objects may be dynamic objects, such as moving motor vehicles) or procedural generation of a synthetically generated second object corresponding to the at least one real object (Par.0025; or procedurally generating the synthetic object that corresponds to the real object, depending on the identified degree of similarity), and integrating the synthetically generated first object or the synthetically generated second object into a specified virtual vehicle environment (Par. 0025; integrating the synthetic object into a predetermined virtual vehicle environment).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the teachings of Vora and Wekel of performing pixel-based classification and instance segmentation using DNN, and projecting the classified data into a virtual space to simulate autonomous vehicle scenarios, with the teachings of Henning describing selecting and procedurally generating synthetically generated objects that correspond to real object. The motivation lies in the advantage of enabling a more efficient and scalable simulation process. Specifically, Henning teaches that such a system can automatically associate real and synthetically generated objects (Par. 0040), significantly simplify the setup of a virtual environment (Par. 0041), and retrieve synthetic objects with a high degree of similarity to real-world counterparts (Par. 0045).The motivation to combine Henning’s teachings with the object detection, classification, and projection techniques of Vora in view of Wekel, is that these enhancement creates a predictable and advantageous improvement to autonomous vehicle simulation platforms by supporting broader testing scenarios and reduced data collection.
Regarding claim 2, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1. Vora in view of Wekel does not disclose wherein for a specified first number of classes, the selection and call of the stored, synthetically generated first object corresponding to the at least one real object is carried out and for a specified second number of classes is carried out.
In the same art of virtual vehicle environment generation using camera-image and LiDAR point-cloud data, Henning discloses wherein for a specified first number of classes (Henning Par. 0024; The synthetically generated objects are classified into different object categories and represent the objects labeled in the video image data, radar data, and/or the lidar point cloud of the real vehicle environment. Henning Par. 0054; in the first step, the real objects are coarsely classified into main classes. In a second step, the respective real objects assigned to the main classes are classified to a greater level of detail in corresponding sub-classes) the selection and call of the stored (Henning Par. 0044; selecting the identified second feature vector and retrieving the stored synthetic object that is associated with the second feature vector and corresponds to the real object), synthetically generated first object (Henning Par. 0040; existing second feature vectors that represent synthetically generated objects) corresponding to the at least one real object (Henning Par. 0040; first feature vectors that represent corresponding real objects) is carried out and for a specified second number of classes (Henning Par. 0024; The synthetically generated objects are classified into different object categories and represent the objects labeled in the video image data, radar data, and/or the lidar point cloud of the real vehicle environment) is carried out (Henning Par. 0070; the synthetic object 14a, 14b, 14c, 14d that corresponds to the real object 12a, 12b, 12c, 12d is procedurally generated S6B).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the class categorization as taught by Henning into the perception framework of Vora and Wekel. Doing so enables a more flexible and scalable object representation within the virtual environment pipeline, allowing objects to be organized at varying levels. The combination yields predictable results in improving adaptability when generating/managing complex autonomous driving scenarios.
Regarding claim 7, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, and further discloses wherein the classes determined by a machine learning algorithm represent buildings, vehicles, traffic signs, traffic lights, roadways, road markings, plantings, pedestrians and/or pedestrians (Vora Section 2.2; C class scores, where for KITTI C= 4 (car, pedestrian, cyclist, background) and for nuScenes C = 11 (10 detection classes plus background)).
Vora, Wekel, and Henning are combined for the reasons set forth above with respect to claim 1.
Regarding claim 8, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1. Vora does not appear to explicitly disclose wherein respective points of the LiDAR point cloud, which are not superimposed by classified pixels of the camera image data are removed from the LiDAR point cloud.
In the same art of object detection and classification using LiDAR point cloud for autonomous vehicle environments, Wekel discloses wherein respective points of the LiDAR point cloud, which are not superimposed by classified pixels of the camera image data are removed from the LiDAR point cloud (Wekel Pg. 12, lines 14-22; one or more geometric constraints may be imposed on the labels and/or the points within the LiDAR point cloud determined therefrom…The geometric constraints may be enforced in order to filter out or remove outlier points in the LiDAR point cloud that are not likely to correspond to the same class and/or instance).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the removal of LiDAR points not corresponding to classified image data, as taught by Wekel, into the fused LiDAR representation of Vora. Doing so improves data consistency and reduce noise in the fused point cloud, yielding predictable results in a more reliable representation of the environment and enhanced accuracy of simulation processes.
Regarding claim 9, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1. Vora does not disclose wherein respective points of the LiDAR point cloud, which are superimposed by classified pixels of the camera image data, which pixels have a confidence value that is less than a predetermined threshold, are removed in order to provide reduced LiDAR point cloud data.
In the same art of object detection and classification using LiDAR point cloud for autonomous vehicle environments, Wekel discloses wherein respective points of the LiDAR point cloud, which are superimposed by classified pixels of the camera image data (Wekel Pg. 18, lines 8-15; This may include transferring the image labels 106 themselves (e.g., propagating the image labels 106 to the LiDAR point cloud to generate LiDAR labels or annotations), and/or may include transferring encoded ground truth data determined using the image labels from the image domain to the LiDAR domain (e.g., pixels of the image may be encoded with semantic and/or instance segmentation values based on the image labels 106, such as confidence values, and these encoded values may be transferred from the pixels of the image 104 to corresponding points of the LiDAR point cloud) , which pixels have a confidence value that is less than a predetermined first threshold (Wekel Pg. 40, lines 32-33 - Pg. 41, lines 1-3; confidence value enables the system to make further decisions regarding which detections should be considered as true positive detections rather than false positive detections. For example, the system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections), are removed in order to provide reduced LiDAR point cloud data (Wekel Pg. 24, lines 5-7; where no confidence values are above a threshold confidence, the pixel may be determined to not be associated with any of the classes that the DNN(s) 126 is trained to detect).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate confidence-based filtering, as taught by Wekel, into the fused LiDAR point cloud of Vora. Doing so increases robustness by prioritizing high-confidence classifications and reducing error detections, yielding predictable results in improved reliability of object detection and tracking used for further processing for autonomous vehicle systems.
Regarding claim 10, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 9. Vora does not disclose wherein the instance segmentation of the classified LiDAR point cloud data for determining the at least one real object comprised by a class is performed using the reduced point cloud data.
In the same art of object detection and classification using LiDAR point cloud for autonomous vehicle environments, Wekel discloses wherein the instance segmentation (segmentation mask) of the classified LiDAR point cloud data (3D LiDAR point cloud) for determining the at least one real object comprised by a class is performed (Wekel Pg. 25, lines 1-4; As such, the output 504B may include a bounding box 508B corresponding to the vehicle 510A, a bounding box 508C corresponding to the pedestrian 506B, and a segmentation mask (e.g., a semantic segmentation mask 132, an instance segmentation mask 130, etc.) with values indicating that the pedestrian 506B is a pedestrian, the vehicle 510A is a vehicle, and/or values indicating that the street - or portions thereof - correspond to drivable free-space.) using the reduced LiDAR point cloud data (Wekel Pg. 24, lines 9-16; some or all of the pixels may have confidence values that the pixel is a centroid of a bounding box 134, in addition to dimension and orientation information…This information may be used to reconstruct the bounding boxes 134 in the range image 102 and/or to ultimately transfer or unproject the bounding box 134 from the range).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate performing instance segmentation on reduced LiDAR point cloud, as taught by Wekel, into the fused LiDAR point cloud of Vora. Doing so improves computational efficiency while maintaining important object-level information, yielding predictable results in efficient processing and extraction of object instances for real-time virtual environment generation.
Regarding claim 11, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, and further discloses wherein the pre-acquired camera image data and LiDAR point cloud data represent the same real vehicle environment captured at the same time (Vora Section 2.2 and Algorithm 1).
Vora, Wekel, and Henning are combined for the reasons set forth above with respect to claim 1.
Regarding claim 13, claim 13 has similar limitations as of method claim 1, except it is the system claim (Vora Section 2.1; autonomous vehicle system. Section 3. Experimental setup) and is therefore rejected under the same rationale as claim 1.
Regarding claim 14, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, but Vora does not seem to explicitly disclose a computer program with program code for performing the method according to claim 1 when the computer program is executed on a computer
In the same art of object detection and classification using LiDAR point cloud for autonomous vehicle environments, Wekel discloses a computer program with program code for performing the method according to claim 1 when the computer program is executed on a computer (Wekel Pg. 35, lines 15-17; The GPU(s) 808 may use compute application programming interface(s) (API(s)). In addition, the GPU(s) 808 may use one or more parallel computing platforms and/or programming models (e.g., NVIDIA’s CUDA).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the method of Vora onto the system of Wekel comprising a processor and storage apparatus storing a program that, when executed, causes the processor to perform the method steps. The motivation lies in the advantage that computer systems with processors and storage media are a standard means of executing image and video processing methods, and would have been an obvious design choice, allowing the method to be automated, executed, and practically deployed in electronic devices.
Regarding claim 15, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, but Vora does not seem to explicitly disclose a non-transitory, computer-readable storage medium comprising program code of a computer program for performing the method according to claim 1 when the computer program is executed on a computer.
In the same art of object detection and classification using LiDAR point cloud for autonomous vehicle environments, Wekel discloses a non-transitory, computer-readable storage medium comprising program code of a computer program for performing the method according to claim 1 when the computer program is executed on a computer (Wekel Pg. 57, lines 12- 21; The computer-storage…implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and/or other data types…computer-readable instructions (e.g., that represent a program(s) and/or a program element(s), such as an operating system...which may be accessed by computing device 900).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the method of Vora onto the system of Wekel comprising a processor and storage apparatus storing a program that, when executed, causes the processor to perform the method steps. The motivation lies in the advantage that computer systems with processors and storage media are a standard means of executing image and video processing methods, and would have been an obvious design choice, allowing the method to be automated, executed, and practically deployed in electronic devices.
Regarding claim 16, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, but Vora in view of Wekel does not disclose wherein the procedural generation of the synthetically generated second object corresponds to the at least one real object.
In the same art of virtual vehicle environments using LiDAR point cloud, Henning discloses wherein the procedural generation of the synthetically generated second object corresponds to the at least one real object (Par. 0025; selecting the identified second feature vector and retrieving a stored synthetic object that is associated with the second feature vector and corresponds to the real object or procedurally generating the synthetic object that corresponds to the real object, depending on the identified degree of similarity. Par. 0069 and Fig. 2 showing multiple classified synthetic objects; identified second feature vector M2 for generating a synthetic object 14a, 14b, 14c, 14d, 14e).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Vora and Wekel’s object identification and classification LiDAR-based vehicle environment system with Henning’s synthetic object generation technique. Doing so provides automatic supplementation of detected real-world instances with synthetic counterparts, thereby yielding predictable results in improving scalability of testing scenarios, provide accurate simulations, and reduce time-consuming data collection.
Regarding claim 17, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, but Vora in view of Wekel does not disclose wherein, based on the instance segmentation of the classified LiDAR point cloud data for determining at least one real object comprised by a class, an extraction of a size and / or a radius of the at least one real object is performed.
In the same art of virtual vehicle environments using LiDAR point cloud, Henning discloses wherein, based on the instance segmentation of the classified LiDAR point cloud data (Par. 0288; perform instance segmentation 1708 on image and/or point cloud data item to identify an outline (e.g., boxes) around the objects and/or obstacles located around the autonomous vehicle) for determining at least one real object comprised by a class, an extraction of a size (Par. 0062; FIG. 2, the control subsystem 1400 extracts the features of the sensor data 216a and determines characteristics of the detected objects within the first field of view 212, such as their identifications (e.g., a vehicle, a road sign, etc.), size, width across the road 200, speed, location coordinates, etc.) and / or a radius of the at least one real object is performed.
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Vora and Wekel’s pixel-based instance segmentation to identify real objects with Henning’s extraction of real-object by size and/or radius. Doing so provides a more accurate simulation of objects by using real-world dimensional data, enabling more realistic collision-avoidance testing, or vehicle maneuver planning. Extracting size or radius of a real-world object to known segmentation outputs for post-processing yields predictable results in improved accuracy for virtual vehicle environment systems.
Regarding claim 18, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, but Vora does not disclose wherein respective points of the LiDAR point cloud, which have the same image coordinates, are removed from the LiDAR point cloud
In the same art of object detection and classification using LiDAR point cloud for autonomous vehicle environments, Wekel discloses wherein respective points of the LiDAR point cloud, which have the same image coordinates, are removed from the LiDAR point cloud (Wekel Pg. 12, lines 14-32; one or more geometric constraints may be imposed on the labels and/or the points within the LiDAR point cloud determined therefrom…The geometric constraints may be enforced in order to filter out or remove outlier points in the LiDAR point cloud that are not likely to correspond to the same class and/or instance…the points of the LiDAR point cloud 220 that fall outside of the geometric constraints may be removed. Fig. 6B; point 610A may be removed from the set of points 610).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the removal of LiDAR points, as taught by Wekel, into the fused LiDAR representation of Vora. Doing so improves data consistency and reduce noise in the fused point cloud, yielding predictable results in a more reliable representation of the environment and enhanced accuracy of simulation processes.
Regarding claim 19, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, and further discloses wherein each point of the LiDAR point cloud, superimposed by classified pixels of the camera image data and having the same image coordinates is assigned an identical class (Vora Section 2.2 and Algorithm 1; Each point in the lidar point cloud is (x, y, z, r) or (x, y, z, r,t) for KITTI and nuScenes respectively…The lidar points are transformed by a homogenous transformation followed by a projection into the image…camera matrix, M, projects the points into the image…Once the lidar points are projected into the image, the segmentation scores for the relevant pixel, (h, w), are appended to the lidar point to create the painted lidar point….)
Vora, Wekel, and Henning are combined for the reasons set forth above with respect to claim 1.
Regarding claim 20, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, and further discloses wherein said projecting the pixel-based classified camera image data onto the pre-acquired LiDAR point cloud data includes projecting a class and a color value (Vora Fig. 4; painted point cloud with the segmentation outputs used to color…points. Section 2.2; Each point in the lidar point cloud is (x, y, z, r) or (x, y, z, r,t) for KITTI and nuScenes respectively…The lidar points are transformed by a homogenous transformation followed by a projection into the image…camera matrix, M, projects the points into the image…Once the lidar points are projected into the image, the segmentation scores for the relevant pixel, (h, w), are appended to the lidar point to create the painted lidar point….)
Vora, Wekel, and Henning are combined for the reasons set forth above with respect to claim 1.
Regarding claim 21, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, and further discloses wherein said confidence value defines a probability that each pixel belongs to a specific class (Vora Section 2.2 and Algorithm 1; Segmentation scores S ∈ RW,H,C with C classes.)
Vora, Wekel, and Henning are combined for the reasons set forth above with respect to claim 1.
Regarding claim 22, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1. Vora does not disclose wherein said instance segmenting describes an assignment of each class to separate instances, and wherein, in an image including multiple vehicles, each individual vehicle is classified as one instance.
In the same art of object detection and classification using LiDAR point cloud for autonomous vehicle environments, Wekel discloses wherein said instance segmenting describes an assignment of each class to separate instances (Wekel Pg. 5, lines 23-28; The DNN(s) may process the LiDAR data to compute outputs corresponding to instance segmentation masks, per-class semantic segmentations masks, and/or bounding shapes (e.g., two-dimensional (2D) range image bounding boxes). These outputs may be processed into 2D bounding boxes (e.g., corresponding to the LiDAR range image) and/or three-dimensional (3D) bounding boxes (e.g., corresponding to a LiDAR point cloud used to generate the LiDAR range image) and class labels for the detected objects), and wherein, in an image including multiple vehicles, each individual vehicle is classified as one instance (Wekel Fig. 5 and Pg 24, lines 24-34 - Pg. 25, lines 1-24; …unique actor instance for pixels corresponding to each unique vehicle…).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate assigning classes to separate object instances, as taught by Wekel, into the fused LiDAR data of Vora. Doing so enables clear differentiation between multiple objects within a scene, yielding predictable results in improved scene interpretability, an essential benefit for accurate modeling and simulation in autonomous driving applications.
Regarding claim 23, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1. Vora in view of Wekel does not disclose wherein an object is searchable in a database based on class and instance.
In the same art of virtual vehicle environments using LiDAR point cloud, Henning discloses wherein an object is searchable in a database based on class and instance (Par. 0039; using existing data of typical traffic scene objects archived or stored in a database. This allows for easy association of the second feature vector with a corresponding synthetic object. Par. 0024; The synthetically generated objects are classified into different object categories and represent the objects labeled in the video image data, radar data and/or the lidar point cloud of the real vehicle environment).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Vora and Wekel’s object detection and classification system for autonomous machines with Henning’s stored objects in a database. Doing so provides a scalable architecture for reusing known LiDAR objects in multiple simulations. Enabling databased-based search and retrieval by class/instance is a well-known technique in the art, yielding predictable results in enhancing system usability, reducing memory usage by using previously stored environmental data, and improving object-to-synthetic replacement workflows.
Regarding claim 24, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, and further discloses wherein the classes determined by a machine learning algorithm represent buildings, vehicles, traffic signs, traffic lights, roadways, road markings, plantings, pedestrians and other objects (Vora Section 2.2; C class scores, where for KITTI C= 4 (car, pedestrian, cyclist, background) and for nuScenes C = 11 (10 detection classes plus background)).
Vora, Wekel, and Henning are combined for the reasons set forth above with respect to claim 1.
Claim(s) 3-6, and 12 is/are rejected under 35 U.S.C. 103 as being Vora et al. "Pointpainting: Sequential fusion for 3d object detection." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4604-4612. 2020, hereinafter referred to as Vora, in view of Wekel et al. (WO 2021041854), hereinafter referred to as Wekel, in further view of Henning et al. (US 20210256738), hereinafter referred to as Henning, and in further view of Brown et al. (US 20220081005), hereinafter referred to as Brown.
Regarding claim 3, Vora in view of Wekel and in further view of Henning discloses the computer-implemented method of claim 1, but does not disclose wherein, based on the instance segmentation of the classified LiDAR point cloud data for determining at least one real object comprised by a class, an extraction of features describing the at least one real object, in particular a size and / or a radius of the object, is performed.
Brown discloses wherein, based on the instance segmentation of the classified LiDAR point cloud data for determining at least one real object (Par. 0288; perform instance segmentation 1708 on image and/or point cloud data item to identify an outline (e.g., boxes) around the objects and/or obstacles located around the autonomous vehicle) comprised by a class (0060; the object classification techniques may classify objects with common features in one class. In one embodiment, the object classification techniques may be trained by a training dataset of data types representing objects, such as in images, videos, LiDAR data, radar, motion data, etc.), an extraction of features describing the at least one real object is performed (Par. 0061; determines classifications of the detected objects using any data processing module, such as image processing, LiDAR data processing, infrared data processing, etc. to extract features of the detected objects).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the combined teachings of Vora, Wekel and Henning by incorporating the teachings of Brown of extracting features from identified objects in point cloud and image data. Vora and Wekel teaches instance segmentation of objects within LiDAR point clouds, while Henning substitutes these real objects with synthetically generated objects for enhanced simulation, further needing to compare real objects with synthetic objects using a “Degree of similarity” (Henning Par. 0023). Brown’s teachings would allow the system to obtain necessary features of the object (Par. 0062; identification, size, width, speed, location etc.) of the segmented objects, allowing for matching real objects to synthetic objects, and scaling synthetic objects appropriately. The motivation lies in the advantage of improved object accuracy in virtual environment, yielding predictable results that enhance the performance and realism of the overall system.
Regarding claim 4, Vora in view of Wekel, in further view of Henning, and in further view of Brown discloses the computer-implemented method of claim 3, and Henning further discloses the procedural generation of the synthetically generated second object corresponding to the at least one real object is carried out (Henning Par. 0048-0049; performing the procedural generation of the synthetic object using the first feature vector generated by the second machine learning algorithm. Thus, it is advantageously possible to ensure that the synthetic object generated in this way will largely match the underlying real object).
However, Henning fails to specifically disclose procedural generation based on the extracted features.
Brown further teaches wherein based on the extracted features (Brown Par. 0061-0062; determines classifications of the detected objects using any data processing module, such as image processing, LiDAR data processing, infrared data processing, etc. to extract features of the detected objects…identifications, size, width across the road, speed, location coordinates, etc.), generating second object (Par. 0061; The control subsystem 1400 may generate a class of similar objects when it encounters at least a threshold number of those objects) corresponding to at least on real object (Par. 0062; The control subsystem 1400 then compares the extracted features of the sensor data 216a with (previously extracted) features of the expected objects in the map data 1510. In this particular example, the control system 1400 identifies that the objects 206 belong to a class of vehicles because their extracted features correspond to the features associated with the class of vehicles.),” (Examiner notes: examiner is interpreting procedural generation as method steps of the algorithm to classify and generate a synthetic object.)
The motivation to combine would've been the same as that set forth in claim 3.
Regarding claim 5, Vora in view of Wekel, in further view of Henning, and in further view of Brown discloses the computer-implemented method of claim 3, and further discloses wherein based on the extracted features (Brown Par. 0061-0062; determines classifications of the detected objects using any data processing module, such as image processing, LiDAR data processing, infrared data processing, etc. to extract features of the detected objects…identifications, size, width across the road, speed, location coordinates, etc.), a comparison of the segmented, at least one real object of a class with a plurality of stored, synthetically generated objects is performed (Brown Par. 0062; The control subsystem 1400 then compares the extracted features of the sensor data 216a with (previously extracted) features of the expected objects in the map data 1510. Henning Par. 0040; by comparison of the first feature vectors that represent corresponding real objects to existing second feature vectors that represent synthetically generated objects, it is advantageously possible to automatically associate corresponding real and synthetically generated objects with one another, so that the synthetically generated objects identified in this way can then be integrated into the virtual vehicle environment. Par. 0044-0046; degree of similarity).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the combined teachings of Vora, Wekel and Henning disclosing the procedural object generation capabilities, to incorporate the teachings of Brown of comparison techniques based on feature extraction. (Examiner notes: examiner is interpreting procedural generation as method steps of the algorithm to classify and generate a synthetic object, and that the comparison based on extracted features from Brown, can be interpreted as the comparison based on the degree of similarity of Henning)
Specifically, Henning provides a method for generating synthetic objects corresponding to real world objects based on a “Degree of similarity”, but does not disclose obtaining specific features to determine a degree of similarity. It would have been obvious to extract features of an object in order to compare real objects to stored synthetic representations, doing so would yield predictable results, such as improved accuracy in matching real and synthetic objects.
Regarding claim 6, Vora in view of Wekel, in further view of Henning, and in further view of Brown discloses the computer-implemented method of claim 5, and further discloses wherein based on the comparison of the segmented, at least one real object of a class with a plurality of stored, synthetically generated objects (Henning Par. 0022; the method includes providing a plurality of stored second feature vectors representing synthetically generated objects), a stored, synthetically generated first object having a specified similarity measure (Henning Par. 0022-0023; as well as identifying a second feature vector having a greatest degree of similarity to the first feature vector. The degree of similarity may be defined based on predetermined properties of the first feature vector and the second feature vector) is selected and called (Henning Par. 0044-0046; degree of similarity of the identified second feature vector to the first feature vector is greater than or equal to the predetermined threshold, selecting the identified second feature vector and retrieving the stored synthetic object that is associated with the second feature vector and corresponds to the real object…).
The motivation to combine would’ve been the same as that set forth in claim 5.
Regarding claim 12, Vora in view of Wekel, in further view of Henning, and in further view of Brown discloses the computer-implemented method of claim 3, and further discloses wherein the features describing the at least one real object are extracted by a further machine learning algorithm (Brown Par. 0243; The obstruction detection instructions 1416 may be implemented by the processor 1402 executing software instructions 1408, and is generally configured to detect objects and their characteristics, such as their identification (e.g., a vehicle, an animal, a person, a tree, a traffic light, etc.), speed, among other characteristics. The obstruction detection instructions 1416 may be implemented using neural networks and/or machine learning algorithms for detecting objects from images, videos, infrared images, point clouds, radar data, etc).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate a machine learning algorithm to perform feature extraction as taught by Brown, into the system of Vora, Wekel, and Henning. The motivation lies in the advantage of machine learning techniques being an obvious improvement, widely used to improve accuracy and efficiency in handling complex, 3D data such as LiDAR point cloud data, to automate learning features from training data, yielding predictable results.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNY NGAN TRAN whose telephone number is (571)272-6888. The examiner can normally be reached Mon-Thurs 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNY N TRAN/Examiner, Art Unit 2615
/ALICIA M HARRINGTON/Supervisory Patent Examiner, Art Unit 2615