Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
INFORMATION DISCLOSURE STATEMENT
The information disclosure statement (IDS) submitted on 11/14/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
FOREIGN PRIORITY
A claim for foreign priority under 35 U.S.C § 119 (a) - (d), which was contained in the Declaration and Power of Attorney filed on 11/14/2024 has been acknowledged. Acknowledgement of claimed foreign priority and receipt of priority documents is reflected in form PTO-326 Office Action Summary.
CLAIM REJECTIONS - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7 & 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over
Smolyanskiy et al. (U.S. Publication 2021/0150230) in view of Laddha et al. (U.S. Publication 2018/0348374)
As to claims 1 & 9-10, Smolyanskiy discloses at least one memory configured to store instructions; and at least one processor configured to execute instructions to ([0041] discloses a processor executing instructions stored in memory. [0042] & Fig. 4 discloses machine learning models 408 configured to detect objects based on sensor data. ): generate, from measurement data including a plurality of frames obtained by measuring a space by a sensor, object recognition information representing information of an object recognized for each of the frames; ([0043-0045] discloses object detection may be performed using LiDAR data from Lidar sensors. [0045] discloses that each detection line in the point cloud may include a three dimensional location and metadata such as reflection characteristics. [0048] & Fig. 5 discloses sensor data 402 such as lidar data may be accumulated 510 from multiple sensors and may be accumulated 510 over time with sensor detections accumulated over windows such as 0.5, 1 or 2 seconds.)([0053] discloses machine learning model extracts classification data and object instance data such as location, geometry and/or orientation data representing detected objects. ([0057-0061] discloses class confidence data including confidence maps, where each pixel stores a probability, score for corresponding class.)
Smolyanskiy partly discloses for each of the frames, generate prediction information in which situation information representing a situation at a time of measuring the space is added to the object recognition information.
([0057] & Fig. 6 discloses adding measurement context/geometry information to object recognition or classification information. See wherein the first stage extracts class confidence data, transforms it into a second view, and supplements that transformed class confidence data with geometry data 640. [0061-0064] discloses classification values may be associated with known 3D locations identified from sensor data, and that 3D locations may be labeled with classification data and projected into another view. ([0066-0068]) discloses geometry data may be generated from sensor data and may include height maps or other statistics for points in a column. Transformed classification data and geometry data are encoded together into a tensor.)
Smolyanskiy is silent to situation information representing a situation at a time of measuring the space. Also the situation information as lidar range/distance, height, intensity, absence of Lidar return or foreground background context.
However, Laddha discloses situation information representing a situation at a time of measuring the space. Also the situation information as lidar range/distance, height, intensity, absence of Lidar return or foreground background context.
([0026] discloses five channels included in a multi-channel data matrix can include range, height, intensity, absence of LIDAR return and LBS foreground. [0026] discloses how far each LIDAR point is from the vehicle (or the LIDAR sensor), the height channel indicates height above ground, the Intensity channel indicates height above ground, the intensity channel indicates returned energy, the absence-of-return channel indicates no LiDAR return, and the LBS foreground channel indicates whether a LiDAR point is foreground after LiDAR background subtraction. [0027-0029 & 0078-0079] & Fig. 6 discloses multi channel data matrix is provided as input to a machine learned model and that the model generates improved object class predictions and location/orientation estimates.)
It would have been obvious to one of ordinary skill in the art at the time of effective filing to modify (Smolyanskiy)’s disclosure to include the above limitations in order to improve object recognition accuracy by supplying richer per frame LiDAR measurement context to the machine learned model.
Smolyanskiy partly discloses generate a model that inputs, to the model, a plurality of units of the prediction information corresponding to the plurality of frames and outputs an object recognition result in the space, by machine learning using the input units of prediction information, the output object recognition result, and correct data of the object recognition result. ([0113-0116] discloses generating/training a machine learning model using input training data, output predictions, and ground truth correct data. [0048] & Fig. 5 discloses training with a plurality of sensor frames/time windows because the LiDAR data may be accumulated over time and each successive input into the machine learning model may be based on successive windows. [0124] & Fig. 5 discloses annotation information may be used to generate class confidence ground truth and object instance data matching the view, size and dimensionality of the instance regression data 412. [0126] discloses ground truth class segmentation and/or instance regression data may be used to train the machine learning models 408 and the one or more loss functions may be used to compare the accuracy of the outputs of machine learning models 408 to ground truth.)
Smolyanskiy is silent to situation information channels used in the prediction information
However, Laddha discloses situation information channels used in the prediction information.
It would have been obvious to one of ordinary skill in the art at the time of effective filing to modify (Smolyanskiy)’s disclosure to include the above limitations in order to improve object recognition accuracy by supplying richer per frame LiDAR measurement context.
As to claim 2, Smolyanskiy in view of Laddha discloses everything as disclosed in claim 1. In addition, Smolyanskiy partly discloses wherein the at least one processor is configured to execute the instructions to, for each of the frames, generate information based on the measurement data as the situation information, and generate the prediction information in which the situation information is added to the object recognition information. ([0066] discloses geometry data 640 representing object characteristics may be generated 635 from sensor data for a corresponding time slice. [0066-0068] discloses geometry data may include height maps generated from LiDAR or Radar point clouds.)
Smolyanskiy in view of Laddha is silent to identify the added situation information as the specific LiDar derived channels of Range, Height, Intensity, Absence of Lidar Return, and LBS foreground.
However, Laddha discloses to identify the added situation information as the specific LiDar derived channels of Range, Height, Intensity, Absence of Lidar Return, and LBS foreground.
([0025-0027] discloses that its multi channel data matrix is generated based at least in part on LiDAR data. [0026 & 0076] discloses also discloses that the channels include Range, Height Intensity, Absence of LiDAR return, and LBS foreground.)
It would have been obvious to one of ordinary skill in the art at the time of effective filing to modify (Smolyanskiy in view of Laddha)’s disclosure to include the above limitations in order to improve object recognition accuracy using richer measurement derived context.
As to claim 3, Smolyanskiy in view of Laddha discloses everything as disclosed in claim 2. In addition, Smolyanskiy discloses wherein the at least one processor is configured to execute the instructions to: for each of the frames, generate the object recognition information including a position of the recognized object; ([0053, 0072])
Smolyanskiy in view of Laddha partly discloses for each of the frames, generate a feature value of the measurement data at the position of the recognized object as the situation information, and generate the prediction information in which the situation information is added to the object recognition information.
Smolyanskiy ([0061]) discloses associating classification information with corresponding 3D LiDAR locations. Classification value from a predicted confidence map may be associated with a 3D location of a LiDAR detection represented by a corresponding range scan pixel. ([0063]) discloses 3D locations from sensor data may be labeled with classification data and projected into a second view. [0066-0068] discloses geometry data may be generated from sensor data for a corresponding time slice and may include height maps or other statistics for points in a column.
Smolyanskiy in view of Laddha is silent to situation information as a specific feature value of the measurement data at the position of the recognized object, such as range, height, intensity, absence of LiDAR return, or LBS foreground.
However, Laddha discloses situation information as a specific feature value of the measurement data at the position of the recognized object, such as range, height, intensity, absence of LiDAR return, or LBS foreground. ([0026-0029 & 0076-0079] & Fig. 5-6)
It would have been obvious to one of ordinary skill in the art at the time of effective filing to modify (Smolyanskiy in view of Laddha)’s disclosure to include the above limitations in order to provide localized measurement context features for improved 3D object recognition.
As to claim 4, Smolyanskiy in view of Laddha discloses everything as disclosed in claim 1. In addition, Smolyanskiy in view of Laddha partly discloses wherein the at least one processor is configured to execute the instructions to, for each of the frames, use information representing a condition of the sensor in the space as the situation information, and generate the prediction information in which the situation information is added to the object recognition information.
Smolyanskiy [0044] discloses Lidar sensor data includes sensor/measurement characteristics. Lidar reflection characteristics may include bearing, azimuth, elevation, range, intensity, reflectivity, signal to noise ratio. See reflection characteristics may depend on sensor mounting position and orientation. [0067] discloses image data from a sensor with known orientation/location may be used to identify 3D location. [0066-0069] discloses recognizing sensor condition/sensor context information such as range, intensity, SNR and sensor position/orientation and teaches combining measurement derived information with object recognition classification data.
Smolyanskiy in view of Laddha is silent to identify a condition of sensor situation channel such as absence of LiDAR return or LBS Foreground being added to the object recognition information.
However, Laddha discloses identify a condition of sensor situation channel such as absence of LiDAR return or LBS Foreground being added to the object recognition information. ([0025-0027, 0076-0079] & Fig. 5)
It would have been obvious to one of ordinary skill in the art at the time of effective filing to modify (Smolyanskiy in view of Laddha)’s disclosure to include the above limitations in order to improve object detection by informing the model with portions of the measured space correspond to missing returns and foreground LiDAR point.
As to claim 5, Smolyanskiy in view of Laddha discloses everything as disclosed in claim 4. In addition, Smolyanskiy discloses wherein the at least one processor is configured to execute the instructions to: for each of the frames, generate the object recognition information including a position of the recognized object ([0053, 0072]);
Smolyanskiy in view of Laddha partly discloses and for each of the frames, use a distance of the sensor with respect to the position of the recognized object as the situation information, and generate the prediction information in which the situation information is added to the object recognition information. ([0044, 0053, 0072])
Smolyanskiy in view of Laddha is silent to situation information is a distance of the sensor with respect to the position of the recognized object. Identify that distance as being added to the object recognition information in the claimed prediction information.
However, Laddha discloses situation information is a distance of the sensor with respect to the position of the recognized object; Identify that distance as being added to the object recognition information in the claimed prediction information. ([0026-0027, 0076-0079] & Fig. 5)
It would have been obvious to one of ordinary skill in the art at the time of effective filing to modify (Smolyanskiy in view of Laddha)’s disclosure to include the above limitations in order to provide distance from sensor context improving LiDAR object recognition accuracy.
As to claim 6, Smolyanskiy in view of Laddha discloses everything as disclosed in claim 1. In addition, Smolyanskiy discloses wherein the at least one processor is configured to execute the instructions to: for each of the frames, generate the object recognition information including a position of the recognized object ([0053, 0072]); and perform machine learning on the model by using a loss corresponding to a difference between the object recognition result output from the model and the correct data of the object recognition result, the object recognition result including at least information about whether or not the object is present and information representing the position of the object. ([0071-0072, 0126-0128])
As to claim 7, Smolyanskiy in view of Laddha discloses everything as disclosed in claim 1. In addition, Smolyanskiy discloses wherein the at least one processor is configured to execute the instructions to, when the object recognition result output from the model includes information indicating that the object is absent, perform machine learning on the model by using a loss corresponding to a difference between the object recognition result including only the information about whether or not the object is present and the correct data of the object recognition result. ([0127-0128])
CONCLUSION
No prior art has been found for claim 8 in its current form.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Stephen P Coleman whose telephone number is (571)270-5931. The examiner can normally be reached Monday-Thursday 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer can be reached at (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Stephen P. Coleman
Primary Examiner
Art Unit 2675
/STEPHEN P COLEMAN/Primary Examiner, Art Unit 2675
Prosecution Insights