DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The IDS(s) has/have been considered and placed in the application file.
Election/Restrictions
Applicant’s election without traverse of Claims 1-20 in the reply filed on 9/17/2025 is acknowledged.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Herau et al. (WO 2024/179658 – hereinafter “Herau”).
Claims 1 and 11.
Herau discloses an apparatus, comprising:
a memory storing processor-readable code; and
at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including (Page 7, lines 22-26: “a computer program product comprising computer readable
instructions for, when run on a computer ….”):
receiving first kinematic information associated with an image sensor of a camera (Page 15: “To compute pose priors of each sensor device … To compute pose priors of each sensor device …
The trajectory Tcw of a main sensor device … in the common coordinate frame which can readily be obtained by a Global Navigation Satellite System (GNSS), by Simultaneous Localization and Mapping (SLAM) – Herau discloses receiving trajectory/kinematic data (poses) for the main sensor (camera/LiDAR) to establish the coordinate frame);
receiving point cloud data from a light detection and ranging (LiDAR) sensor (Page 2, Lines 28-30:" ... obtaining by the first sensor device first sensor data .. . and obtaining by each of the second sensor devices second sensor data ... "; Page 8, Lines 11-12: "The first sensor device may be a camera device or a Light Detection and Ranging, LIDAR, device ... " Herau discloses obtaining sensor data from LiDAR sensors (point clouds).);
generating first image data that is time-synchronized with the point cloud data based on the first kinematic information and a neural radiance fields (NeRF) model (Page 4, Lines 16-20:" ... the neural network is trained by c) optimizing time shifts of the second timeframes ... conjointly with the optimizing of the weights ... "; Page 5, Line 5: “the neural network is trained based on the Neural Radiance Field (NeRF) technique”; Page 6, Lines 21-25 (Equation):" ...
PNG
media_image1.png
90
678
media_image1.png
Greyscale
"; Page 13, Lines 10-15:" ... determine the constant time shift dt... and observations ... by the other sensor device at k'=k+dt." Herau teaches generating (rendering via Zθ) data at a specifically calculated timestamp (time+dt) to achieve synchronization.); and
generating fused data that combines the first image data and the point cloud data (Page 3, Lines 1-5: " ... mapping the first sensor data and the second sensor data ... into a common coordinate system to obtain mapped first and second sensor data." Page 17, Lines 14-15: " ... camera and LIDAR observations are tightly coupled during the optimization process." The "mapped" data constitutes fused data where both modalities share a common coordinate frame and timeline).
Claims 2 and 12.
Herau discloses the apparatus of claim 11, wherein the operations further include determining a pose of the image sensor based on the first kinematic information, wherein the point cloud data is captured by the LiDAR sensor at a first time point, wherein the pose corresponds to the first time point, and wherein generating the first image data includes inputting the pose into the NeRF model such that the NeRF model outputs the first image data (Page 16, Lines 7-9: "The pose priors for each observation of each sensor device i in the common frame can be computed (see 51 and 52 in Figure 5) based on these inputs ... " Page 16, Line 10 (Equation 3): "Tciw,t=k' …" Herau calculates the specific pose for every specific timestamp (time point) using the kinematic trajectory and inputs this into the model for processing.).
Claims 3 and 13.
Herau discloses the apparatus of claim 11, wherein the first kinematic information is received from an inertial navigation system including a motion sensor and a rotation sensor (Page 2, Line 8:" ... inertial measurement units ... " Page 12, Lines 2-3:" Rotation can include three rotational movements, i.e., roll, yaw and pitch ").
Claims 4 and 14.
Herau discloses the apparatus of claim 11, wherein the operations further include: receiving second kinematic information associated with the LiDAR sensor (Page 15, Lines 29:" ... The priors of the extrinsic calibrations .. . i.e. the relative transformations ... "); and generating modified point cloud data based on the point cloud data and the second kinematic information (Page 13, Lines 10-14:" ... a continuous differentiable trajectory interpolation function ... [allows] interpolated observations .. . [at] t timestamps k'“. Herau receives kinematics for all sensors to map them to the common frame. "Interpolating" the trajectory to shift data points generates modified data based on kinematics.).
Claims 5 and 15.
Herau discloses the apparatus of claim 14, wherein the fused data combines the first image data and the modified point cloud data (Page 3, Lines 1-5:" ... mapping the first sensor data and the second sensor data ... to obtain mapped first and second sensor data. The "mapped" data (which has been modified/interpolated to fit the common frame) is the data used for the fusion.).
Claims 6 and 16.
Herau discloses the apparatus of claim 14, wherein generating the modified point cloud data includes modifying a position of a plurality of points of the point cloud data based on the second kinematic information (Page 3, Lines 1-5:" .. . mapping the first sensor data and the second sensor data ... into a common coordinate system ... "Mapping point clouds from a local sensor frame to a common global frame inherently involves "modifying the position" of the points via matrix multiplication (rotation/translation).).
Claims 7 and 17.
Herau discloses the apparatus of claim 11, wherein the NeRF model is trained on second image data that is not spatially or temporally synchronized with the point cloud data such that the NeRF model generates the first image data based on the first kinematic information (Page 2, Lines 10-12:" ... if the individual sensor devices are not synchronized with each other ... temporal calibration is also needed."; Page 4, Lines 14-16:" ... optimizing time shifts ...
conjointly with the optimizing of the weights ... .” Herau's is designed to train on and resolve data that is initially not synchronized (hence the need to optimize "time shifts").).
Claims 8 and 18.
Herau discloses the apparatus of claim 11, wherein the image sensor is configured to capture data at a higher frequency than the LiDAR sensor (Page 13, Lines 10-11:" ... generates measurements at timestamps k', k'≠ k ... " Page 1, Lines 22-23:"Multimodal sensor calibration is demanding since it deals with different types of data formats ... " While Herau does not strictly define "higher," it describes handling asynchronous timestamps (k'≠ k), which covers any frequency mismatch. Standard cameras (30Hz) are inherently higher frequency than LiDAR (10Hz).).
Claims 9 and 19.
Herau discloses the apparatus of claim 11, wherein the operations further include training a machine learning model with a dataset including the fused data (Page 3, Lines 4-7:" ... training a neural network based on the mapped [fused] first and second sensor data ... " The NeRF itself is a machine learning model. Furthermore, Herau trains this model using the fused (mapped) dataset.).
Claims 10 and 20.
Herau discloses the apparatus of claim 11, wherein the operations further include: detecting an object based on the fused data; and controlling a machine based on the object that is detected (Page 22, Lines 28-30:" ... facilitates fused perception of numerous sensor devices for tracking and detection verification ... "; Page 22, Line 24:" ... autonomous functionalities can be enabled by the vehicle software.").
Conclusion
The prior art made of record but not relied, yet considered pertinent to the applicant’s disclosure, is listed on the PTO-892 form.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674