DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claim(s) 1-3, 6-8, 11-15, 18-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ding (“Attention-Guided Lidar Segmentation and Odometry Using Image-to-Point Cloud Saliency Transfer”, 2023).
Claim(s) 4 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ding, in view of Xu (“Robust Self-Supervised LiDAR Odometry via Representative Structure Discovery and 3D Inherent Error Modeling”, 2022).
Claim(s) 9 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ding, in view of Xu (“Robust Self-Supervised LiDAR Odometry via Representative Structure Discovery and 3D Inherent Error Modeling”, 2022).
Claims 5 and 17 is/are allowed.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The disclosure is objected to because of the following informalities: possible typos, for example, paragraph [0017] states “may generate detect moving objects”.
Appropriate correction is required.
Claim Objections
Claim 6 and 18 objected to because of the following informalities: possible typos, for example, the second to last line states “is trained identify”, from context the statement could be read as “is trained to identify”; the claims will be interpreted as such until correction that may indicate otherwise. Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-3, 6-8, 11-15, 18-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ding (“Attention-Guided Lidar Segmentation and Odometry Using Image-to-Point Cloud Saliency Transfer”, 2023).
Regarding claims 1 and 13, Ding teaches An apparatus for detecting objects (Ding, Abstract: “our proposed SalLiDAR module. SalLiDAR is a saliency-guided 3D semantic segmentation model that integrates saliency information to improve segmentation performance”), the apparatus comprising:
at least one memory (Ding, Abstract: “our proposed SalLiDAR module. SalLiDAR is a saliency-guided 3D semantic segmentation model that integrates saliency information to improve segmentation performance”. The module is being interpreted as involving memory); and
at least one processor coupled to the at least one memory (Ding, Abstract: “our proposed SalLiDAR module. SalLiDAR is a saliency-guided 3D semantic segmentation model that integrates saliency information to improve segmentation performance”. The module is being interpreted as involving memory coupled with at least one processor) and configured to:
obtain image data representative of a scene (Ding, see Figure 2 image below, “2D Image Saliency Knowledge” shows the two images on the top-right having image data representative of a scene) and point-cloud data representative of the scene (Ding, pg 5, Figure 2, reproduced below:
PNG
media_image1.png
934
800
media_image1.png
Greyscale
. Examiner recommends viewing the color version of the figures. “3D Point Cloud Saliency Knowledge” section, the images below that show “point-cloud data representative of the scene”);
process the image data (Ding, see Figure 2 above, the top row shows image data) and the point-cloud data (Ding, see Figure 2 image above, the middle row shows point-cloud data) using a machine-learning model (Ding, see Figure 2 above, “proposed framework of image-to-LiDAR saliency knowledge transfer” is being interpreted to involve a “machine-learning model”), wherein the machine-learning model is trained (Ding, pg 15, ¶1, reproduced below:
PNG
media_image2.png
758
878
media_image2.png
Greyscale
. “Odometry model training” is being interpreted as “machine-learning model is trained”) using at least one loss function (Ding, see pg 15 image above: “odometry loss”) to detect moving objects (Ding, see pg 15 image above: “The point with a semantic class of moving or potentially moving is defined as a dynamic point”. Which is being interpreted as moving objects) represented by image data (Ding, see Figure 1 image above, which shows the top row being image data) and point-cloud data (Ding, see Figure 1 image above, which shows the middle row having point-cloud data), the at least one loss function being
based on odometry data (Ding, see pg 15 image above, “point-wise matching odometry loss” is being interpreted as involving odometry data) and at least one of training image-data features (Ding, see Figure 1 image above, which shows the top row being image data, with extracted features aiding in the training of the Ding framework) or training point-cloud-data features (Ding, see pg 15 image above, “point-wise matching odometry loss”. Point-wise is being interpreted to involve point-cloud data features; “odometry model training” is being interpreted as “training”); and
obtain, from the machine-learning model, indications of one or more objects that are moving in the scene (Ding, pg 15, Table 2, reproduced below:
PNG
media_image3.png
596
1058
media_image3.png
Greyscale
. Dynamic class is being interpreted to include moving objects).
Regarding claim 2, Ding teaches The apparatus of claim 1, wherein the at least one loss function (Ding, see pg 15 image above in claim 1, “point-wise matching odometry loss”) is based on a relationship between a change in a position of a system (Ding, see Figure 7 image below, the bottom row shows a change in a position of a system, which includes the vehicles pictured) as indicated by the odometry data (Ding, see pg 15 image above in claim 1, “point-wise matching odometry loss” is being interpreted as involving odometry data) and at least one of:
a change between a first set of training image-data features and a second set of training image-data features (Ding, pg 13, Figure 6, bottom left box for the “LiDAR to Range Image”, shows training image-data features. The subscript t and t-1 are being interpreted as first set and second set.); or
a change between a first set of training point-cloud-data features and a second set of training point-cloud-data features (Ding, pg 14, Figure 7, reproduced below:
PNG
media_image4.png
756
1068
media_image4.png
Greyscale
. The bottom row is being interpreted to be showing a change between a first and second set of training point-cloud-data features; with the first column being the first set, and the second column being a second set. Examiner recommends viewing the color version of the figure.).
Regarding claim 3, Ding teaches The apparatus of claim 2, wherein the odometry data (Ding, see pg 15 image above, “point-wise matching odometry loss” is being interpreted as involving odometry data) corresponds to at least one of the training image-data features (Ding, see Figure 1 image above, which shows the top row being image data, with extracted features aiding in the training of the Ding framework) or the training point-cloud-data features (Ding, see pg 15 image above, “point-wise matching odometry loss”. Point-wise is being interpreted to involve point-cloud data features; “odometry model training” is being interpreted as “training”).
Regarding claim 6, Ding teaches The apparatus of claim 1, where the at least one processor is further configured to:
obtain classifications of objects in the scene (Ding, pg 15, ¶1: “we first convert the predicted semantic map to a binarized mask to indicate the static points [e.g., building, road] and dynamic points [e.g., car, person]”. The static and dynamic points are being interpreted as classifications of objects in the scene); and
provide the classifications of the objects to the machine-learning model as an input (Ding, pg 13, last paragraph: “For odometry estimation, we convert and concatenate two consecutive LiDAR point clouds and their respective predicted saliency and semantic maps to range images as the input of the odometry module”. “Odometry module” is being interpreted as involving “the machine-learning model”), wherein the machine-learning model is trained identify moving objects (Ding, pg 15, ¶1 “dynamic points” are being interpreted as “moving objects” that are identified) represented by image data (Ding, pg 13, last paragraph, “range Images” are being interpreted as image data) and point-cloud data (Ding, pg 13, last paragraph: “two consecutive LiDAR point clouds”, which is interpreted as point-cloud data) further based on classifications (Ding, pg 15, ¶1: “we first convert the predicted semantic map to a binarized mask to indicate the static points [e.g., building, road] and dynamic points [e.g., car, person]”. The static and dynamic points are being interpreted as classifications of objects in the scene).
Regarding claim 7, Ding teaches The apparatus of claim 6, wherein the machine-learning model comprises a first machine-learning model and the at least one processor is further configured to:
provide at least one of the image data (Ding, see Figure 3 image below: the top-right has “RGB Source images”) or the point-cloud data (Ding, see Figure 3 image below, the second row has “Raw point Cloud” which is being interpreted as point-cloud data) to a second machine-learning model (Ding, see Figure 3, the bottom row shows the “Attention-Guided LiDAR Semantic Segmentation”, which is being interpreted as a second machine learning model) that is trained classify objects represented by at least one of image data or point-cloud data (Ding, pg 10, Figure 3, reproduced below:
PNG
media_image5.png
688
1070
media_image5.png
Greyscale
. The bottom row shows the “Attention-Guided LiDAR Semantic Segmentation”, which is being interpreted as a second machine learning model that was trained to classify objects represented by images or point-cloud data, as can be seen from the first and second row. Examiner recommends viewing the color version of the Figures); and
obtain the classifications of the objects from the second machine-learning model (Ding, see Figure 3 above, which shows the bottom row with the “Output LiDAR Semantics” as being the object classifications from the second machine-learning model).
Regarding claim 8, Ding teaches The apparatus of claim 1, wherein the machine-learning model is trained identify moving objects (Ding, pg 15, ¶1: “we first convert the predicted semantic map to a binarized mask to indicate the static points [e.g., building, road] and dynamic points [e.g., car, person]”. The binarized mask to indicate dynamic points is being interpreted identify moving objects) represented by image data (Ding, pg 5, Figure 2, the top-row shows image data) and point-cloud data (Ding, pg 5, Figure 2, the middle row shows point-cloud data) further based on classifications of objects in the scene (Ding, pg 15, ¶1: “we first convert the predicted semantic map to a binarized mask to indicate the static points [e.g., building, road] and dynamic points [e.g., car, person]”. “Semantic map” is being interpreted as object classifications in the scene).
Regarding claim 11, Ding teaches The apparatus of claim 1, wherein the indications of objects that are moving in the scene (Ding, pg 15, ¶1: “we first convert the predicted semantic map to a binarized mask to indicate the static points (e.g., building, road) and dynamic points (e.g., car, person)”. The dynamic class show indications of objects that are moving in the scene) comprise classifications of points in the scene into classes (Ding, pg 15, ¶1: “The point with a semantic class of moving or potentially moving is defined as a dynamic point”. “Semantic class” is being interpreted as involving classifications of points in the scene into classes) comprising:
Stationary (Ding, pg 15, ¶1: “we first convert the predicted semantic map to a binarized mask to indicate the static points (e.g., building, road)”);
Movable (Ding, pg 15, ¶1: “The point with a semantic class of moving or potentially moving is defined as a dynamic point”. “Potentially moving” is being interpreted to involve movable objects); or
moving (Ding, pg 15, ¶1: “we first convert the predicted semantic map to a binarized mask to indicate the static points (e.g., building, road) and dynamic points (e.g., car, person)”. “Dynamic points” are being interpreted as moving).
Regarding claim 12, Ding teaches The apparatus of claim 1, wherein the at least one processor (Ding, pg 2, Section 1, ¶1: “Simultaneous Localization and Mapping (SLAM) [1] technology plays a critical role in the perception and planning process of autonomous vehicles”. Planning process” is being interpreted to involve at least one processor) is further configured to at least one of:
control a vehicle based on the indications of objects that are moving in the scene (Ding, pg 2, Section 1, ¶1: “Simultaneous Localization and Mapping (SLAM) [1] technology plays a critical role in the perception and planning process of autonomous vehicles by constructing a map of the surrounding environment and localizing the vehicle”. “autonomous vehicles” is being interpreted as the vehicle being controlled to avoid objects in the scene, hence the usage of SLAM); or
provide information to a driver of the vehicle based on the indications of objects that are moving in the scene.
Claim 14 is rejected using the same rationale as applied to claim 2 discussed above.
Claim 15 is rejected using the same rationale as applied to claim 3 discussed above.
Claim 18 is rejected using the same rationale as applied to claim 6 discussed above.
Claim 19 is rejected using the same rationale as applied to claim 7 discussed above.
Claim 20 is rejected using the same rationale as applied to claim 8 discussed above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ding, in view of Xu (“Robust Self-Supervised LiDAR Odometry via Representative Structure Discovery and 3D Inherent Error Modeling”, 2022).
Regarding claim 4, Ding teaches wherein the at least one loss function (Ding, see pg 15 image above: “odometry loss”)
Ding does not appear to specifically teach between a magnitude of a change in a position of a system as indicated by the odometry data and at least one of: a magnitude of a change between a first set of training image-data features and a second set of training image-data features; or a magnitude of a change between a first set of training point-cloud-data features and a second set of training point-cloud-data features.
Pertaining to a similar field of endeavor, Tian teaches based on a relationship (Tian, see equation 14 image below, the optimization equation is being interpreted as a relationship) between a magnitude of a change in a position of a system as indicated by the odometry data (Tian, pg 5, column 1, equation 14, reproduced below:
PNG
media_image6.png
246
488
media_image6.png
Greyscale
. E_odo from the first line is being interpreted as odometry of a system, with the X variables being the pose of the ego vehicle, or system, support for that is in Section D on pg 4. The vertical double bars with the square term, or superscript 2, is part of the magnitude of the odometry data. Further, the optimization problem is being interpreted as involving a loss function) and at least one of:
a magnitude of a change between a first set of training image-data features and a second set of training image-data features (Examiner note: this is part of an “or” statement); or
a magnitude of a change between a first set of training point-cloud-data features (Tian, see equation 14 above, line 1, that contains the e_odo function, X_i-1 is being interpreted as a first set. The LiDAR used, from the title, is being interpreted as having point-cloud-data features) and a second set of training point-cloud-data features (Tian, see equation 14 above, line 1, that contains the e_odo function, X_i is being interpreted as the second set. The LiDAR used, from the title, is being interpreted as having point-cloud-data features).
Ding and Tian are considered to be analogous art because they are directed to odometry involving LiDAR. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for LiDAR and odometry (as taught by Ding) to include a relationship between magnitude of odometry data and magnitude between a first and second training point-cloud-data (as taught by Tian) because the combination provides to improve visual SLAM (Tian, pg 2, paragraph before section III).
Claim 16 is rejected using the same rationale and motivation as applied to claim 4 discussed above.
Claim(s) 9 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ding, in view of Xu (“Robust Self-Supervised LiDAR Odometry via Representative Structure Discovery and 3D Inherent Error Modeling”, 2022).
Regarding claim 9, Ding teaches The apparatus of claim 1, wherein the at least one loss function (Ding, see pg 15 image in claim 1, “point-wise matching odometry loss” is being interpreted as involving at least one loss function)
Ding does not appear to specifically teach an adaptive motion-consistency threshold.
Pertaining to a similar field of endeavor, Xu teaches
is further based on an adaptive motion-consistency threshold (Xu, pg 2, column 1, ¶2-4: reproduced below:
PNG
media_image7.png
1038
578
media_image7.png
Greyscale
. “Self-supervised” shows “adaptive” as the framework adapts to the data as opposed to a ground truth. “Geometric consistency loss” shows consistency. “Ego-motion” shows odometry data for the motion-consistency related to the “geometric consistency loss to check the quality of current voted ego-motion”. “Errors” show a threshold exists, otherwise an error would not be detected. Further, one of the main research topics is “how to distinguish and focus on the representative structure while adaptively downweight the unreliable regions is still an important problem in LiDAR odometry” from Section 1 Introduction on page 1).
Ding and Xu are considered to be analogous art because they are directed to odometry involving LiDAR. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for LiDAR and odometry (as taught by Ding) to include an adaptive motion-consistency threshold (as taught by Xu) because the combination provides to improve LiDAR odometry (Xu, Abstract).
Regarding claim 10, Xu teaches The apparatus of claim 9, wherein the adaptive motion-consistency threshold is dynamically adjusted (Xu, see claim 9 image, “self-supervised” is being interpreted as adaptive; “alignment errors” are dynamically adjusted based on the data instead of a ground-truth) based on at least one of:
a complexity of the scene (Xu, Abstract: “given the complex scenarios and the low-resolution LiDAR, finding reliable structures for identifying correspondences can be challenging”) or an uncertainty of the odometry data (Xu, see claim 9 image, “uncertainty-aware geometric consistency loss to check the quality of current voted ego-motion”, which is being interpreted as “uncertainty of the odometry data” of the ego-motion).
Allowable Subject Matter
Claims 5 and 17 is/are allowed.
The following is an examiner’s statement of reasons for allowance:
Regarding claims 5 and 17: The cited prior art fails to disclose, teach, or suggest: “wherein the at least one loss function is based on a relationship between: a product of a magnitude of a first change in a position of a system as indicated by the odometry data and at least one of: a magnitude of a change between a first set of training image-data features and a second set of training image-data features; or a magnitude of a change between a first set of training point-cloud-data features and a second set of training point-cloud-data features; and a product of a magnitude of a second change in the position of the system as indicated by the odometry data and at least one of: a magnitude of a change between the second set of training image-data features and a third set of training image-data features; or a magnitude of a change between the second set of training point-cloud-data features and a third set of training point-cloud-data features” in the context of the claim as a whole.
The cited prior art includes:
Tian et al (“DL-SLOT: Dynamic LiDAR SLAM and object tracking based on collaborative graph optimization”, 2022) teaches “a magnitude of a first change in a position of a system as indicated by the odometry data” (see pg 5, equation 14, line 1: the function e_odo, the ego-vehicle odometry, is being interpreted as odometry data that shows a magnitude of a change in a position of a system [see the double vertical lines and the square, or superscript 2]); Tian further teaches “a magnitude of a change between a first set of training point-cloud-data features and a second set of training point-cloud-data features” (equation 14, line 1 contains X_i and X_i-1, which are being interpreted as first and second set of training point-cloud-data, respectively); Tian further teaches a “a magnitude of a change between the second set of training point-cloud-data features and a third set of training point-cloud-data features” (equation 14, the e_cons function contains object speed variable c with i-2 being the third set and i-1 being the second set of training point-cloud-data; the LiDAR system used in the Tian reference is being interpreted to include point-cloud-data features); Tian also appears to teach second change in of the odometry data (see equation 15, involving the e_loop and t-d subscript). However, Tian does not appear to specifically teach “a product of a magnitude” of the odometry data and “a magnitude of a change between a first set of training point-cloud-data features and a second set of training point-cloud-data features”. One with ordinary skill in the art would know “a product” can involve multiplication, or dot product when used with matrix/vector mathematics; Tian instead focuses on adding up the individual elements (see equation 14, again). Further, equations 14 and 15 belong to two different optimization sections, collaborative and global; while they share similar elements, Tian does not appear to specifically teach (or reasonably derive) “a product” of magnitude of a first change for odometry data and first/second set of point-cloud-data; further, Tian does not appear to specifically teach (or reasonably derive) “a product” of magnitude of a second change for odometry data and second/third set of point-cloud-data. Further, there appears to be no motivation to derive or combine with the cited prior art.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Yang et al (US 2024/0362922 A1) discloses dynamic (or moving) object detection using images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNNY B DUONG whose telephone number is (571)272-1358. The examiner can normally be reached Monday - Thursday 10a-9p (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.B.D./Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667