Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
This application claims benefit of foreign priority under 35 U.S.C. 119(a)-(d) of EP24 15 7955.6, filed on 2/15/2024.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 2/10/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered and attached by the examiner.
Claim Objections
Claim 3 is objected to because of the following informalities: The phrasing of “determining the values of the predetermined feature for the pixel a further machine learning model” is unclear. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-9 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al., “NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields,” AAAI Conference on Artificial Intelligence 2023, 2023, pages 1-15 hereinafter referred to as (Zhang) in view of Megaro (U.S. Patent Pub. No. 2025/0104343).
Regarding Claim 1, Zhang teaches a method for generating training data for an object detector, comprising (Abstract: Labelling LiDAR point clouds for training autonomous driving is extremely expensive and difficult... Neural Radiance Fields (NeRF) have been proposed for novel view synthesis using implicit reconstruction of 3D scenes. Inspired by this, we present NeRF-LIDAR, a novel LiDAR simulation method that leverages real-world information to generate realistic LIDAR point clouds:)
receiving a plurality of optical images of a scene, each showing the scene from a respective viewing direction of a plurality of different viewing directions (Section 3.2 NeRF Reconstruction: we present a new method takes multi-view images and sparse LiDAR signals to reconstruct the street scenes and represent the 3D scenes as an implicit NeRF model. We propose to use the driving-scene data to learn the NeRF reconstruction;)
receiving a plurality of sensor data elements, each sensor data element including sensor data other than optical image data of the scene from a respective sensing direction of a plurality of different sensing directions (Section 3.2 NeRF Reconstruction: we present a new method takes multi-view images and sparse LiDAR signals to reconstruct the street scenes and represent the 3D scenes as an implicit NeRF model. We propose to use the driving-scene data to learn the NeRF reconstruction;)
training a first neural radiance field using the plurality of optical images to generate, for each 3D point of the scene, a respective value of a predetermined feature (Introduction: we proposed to learn a NeRF representation for real-world scenes and render LiDAR point clouds along with accurate semantic labels (predetermined feature);)
training a second neural radiance field using the plurality of sensor data elements to generate, for each 3D point of the scene, a respective sensor data value; and (Section 3.2 NeRF Reconstruction: We reconstruct the NeRF representation based on the multi-view images and leverage the LiDAR points to provide extra depth supervision to create more accurate 3D geometries. Besides, the real LiDAR point clouds are used as supervision to learn more realistic simulated LiDAR (respective sensor data value) data.)
generating multiple training data elements for the object detector by, for each training data element, generating training input using the second neural radiance field and ground truth information for the training input using the first neural radiance field (Section 3.4 Label generation: we can also leverage the existing ground-truth labels for more robust label generation; Section Learning Raydrop & Alignment: NeRF-LiDAR allows us to render depths (3D points) at arbitrary positions and directions. We use the ground-truth LiDAR frames as supervision to learn the raydrop. Given one ground-truth LiDAR frame P, we render the simulated LiDAR frame ˆP at the same location accordingly.)
Zhang does not explicitly disclose training a first neural radiance field using the plurality of optical images to generate, for each 3D point of the scene, a respective value of a predetermined feature.
Megaro is in the same field of art of image analysis. Further, Megaro teaches training a first neural radiance field using the plurality of optical images to generate, for each 3D point of the scene, a respective value of a predetermined feature (Megaro, Fig. 4; ¶49 In operation 406, scene decomposition engine 122 chooses a pair (3D location, viewing ray) for which training data 210 includes ground truth color information. For each NeRF in machine learning model 230, volume rendering engine 280 generates a color value (feature value) for the chosen pair (3D location, viewing ray) based on the NeRF's associated radiance field function.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Zhang by generating for each point a feature value that is taught by Megaro; thus, one of ordinary skilled in the art would be motivated to combine the references for more effective techniques for the representation of multiple 3D objects in a scene (Megaro ¶6).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 2, Zhang in view of Megaro discloses the method of claim 1, wherein the training of the first neural radiance field includes determining values of the predetermined feature for pixels of the optical images and training the first neural radiance field to determine values of the predetermined feature for the pixels (Megaro, Fig. 4; ¶49 In operation 406, scene decomposition engine 122 chooses a pair (3D location, viewing ray) for which training data 210 includes ground truth color information. For each NeRF in machine learning model 230, volume rendering engine 280 generates a color value for the chosen pair (3D location, viewing ray) based on the NeRF's associated radiance field function.)
The reasons for combining Zhang and Megaro are similar to that stated in the rejection of claim 1. In addition, this same reasoning is pertinent and applicable to the rejections of claim 3 below.
Regarding Claim 3, Zhang in view of Megaro discloses the method of claim 2, further comprising determining the values of the predetermined feature for the pixel a further machine learning model (Megaro, Fig. 4; ¶49 In operation 406, scene decomposition engine 122 chooses a pair (3D location, viewing ray) for which training data 210 includes ground truth color information. For each NeRF in machine learning model 230, volume rendering engine 280 generates a color value for the chosen pair (3D location, viewing ray) based on the NeRF's associated radiance field function; ¶27 In various embodiments, the number of NeRFs in machine learning model 230 is equal to the number of discrete objects in the 3D scene.)
Regarding Claim 4, Zhang in view of Megaro discloses the method of claim 1, wherein the second neural radiance field is trained using the plurality of sensor data elements to generate, for each 3D point of the scene and for each of a plurality of sensor data types, a respective sensor data value (Zhang, Section 3.2 NeRF Reconstruction: We reconstruct the NeRF representation based on the multi-view images and leverage the LiDAR points to provide extra depth supervision to create more accurate 3D geometries. Besides, the real LiDAR point clouds are used as supervision to learn more realistic simulated LiDAR (respective sensor data value) data.)
Regarding Claim 5, Zhang in view of Megaro discloses the method of claim 1, wherein the predetermined feature is a text embedding (Zhang, Introduction: we proposed to learn a NeRF representation for real-world scenes and render LiDAR point clouds along with accurate semantic labels (predetermined feature); Section 3.4 Label generation: we can also leverage the existing ground-truth labels for more robust label generation.)
Regarding Claim 6, Zhang in view of Megaro discloses the method of claim 1, further comprising: training the object detector using the generated training data elements (Zhang, Conclusion: The effectiveness of our NeRF-LiDAR is verified by training 3D segmentation neural networks. It’s shown that the 3D segmentation models trained on the generated LiDAR data can achieve similar mIoU as those models trained on the real LiDAR data.)
Regarding claim 7, claim 7 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Zhang further teaching on: A method for controlling a technical system, comprising the following steps; training an object detector using the generated training data elements; receiving sensor data of a scene in which the technical system is to be controlled; performing object detection using the trained object detector; and controlling the technical system according to a result of the object detection (Zhang, Conclusion: The effectiveness of our NeRF-LiDAR is verified by training 3D segmentation neural networks. It’s shown that the 3D seg mentation models trained on the generated LiDAR data can achieve similar mIoU as those models trained on the real LiDAR data; Zhang states in the Introduction this training data is used for autonomous driving, therefore one with ordinary skill in the art could use the trained device to control an autonomous vehicle.)
Regarding claim 8, claim 8 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Megaro further teaching on: A data processing device (Megaro, Fig. 1, 100 computing device)
Regarding claim 9, claim 9 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Megaro further teaching on: A non-transitory computer-readable medium on which are stored instructions, the instructions, when executed by a computer, causing the computer to perform the steps (Megaro, ¶72 one or more non-transitory computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform the steps)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUSTIN BILODEAU whose telephone number is (571)272-1032. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUSTIN BILODEAU/Examiner, Art Unit 2664
Prosecution Insights