Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/13/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 6, 7, and 9 are objected to because of the following informalities:
In claim 6, line 4, “the point” should be changed to --a point--
In claim 7, line 2, both “a spatial” and “a lidar” should be changed to --the special-- and --the lidar-- respectively
In claim 7, line 3, the comma “,” directly after “sensors” should be deleted
In claim 7, last line, --place-- should be inserted after “takes”
In claim 9, line 1, the dependency should be changed from “1” to --8--
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-11 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Specifically, in claim 1, line 2, “a spatial sensor, in particular a lidar sensor” renders the claim vague and indefinite since the “in particular” language makes it unclear if the scope of the claim includes a lidar sensor or just any spatial sensor, or both.
Also, in claim 1, lines 17 and 18, the recitation of “exceeds a threshold value or a threshold value of at least 0.50” renders the claim unclear as it is uncertain which threshold value is needed, i.e. a threshold of some/any value, or a threshold value of specifically at least 0.50.
Claims 2-11 are rejected likewise as being dependent from rejected claim 1 and having failed to rectify the vagueness of claim 1.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-5, 10-11 are rejected under 35 U.S.C. 102a2 as being anticipated by US 2024/0046657 to Li et al, hereinafter “Li.”
With regard to claim 1, Li discloses a computer-implemented method for automatically annotating sensor data, the method comprising: receiving a plurality of sensor data frames, including sensor data frames of a spatial sensor, in particular a lidar sensor ([0021] - sensor 128 “LIDAR”), and sensor data frames of an area sensor or a camera ([0021] - sensor 128 “set of images”), the measuring ranges of the spatial sensor and of the area sensor spatially overlapping (Figures 3A-3C); annotating the plurality of sensor data frames using at least one neural network ([0021] - 3D DNN), the annotation including recognizing objects and assigning a bounding box to each object [0021]; grouping a sensor data frame of the spatial sensor and a sensor data frame of the area sensor based on a temporal correlation of the measuring points in time; projecting at least four corners of a three-dimensional bounding box of a recognized object or a bounding box in the sensor data frame of the spatial sensor, into the image plane of the area sensor to obtain a projected rectangle [0021] and Figure 3C; checking whether the relative overlap or the intersection over union, between the projected rectangle and a neighboring two-dimensional bounding box or a bounding box in the sensor data frame of the area sensor, exceeds a threshold value or a threshold value of at least 0.50 ([0022] - mAP values compared to a threshold); linking, if the relative overlap exceeds the threshold value, the three-dimensional bounding box and the neighboring two-dimensional bounding box to the same object and carrying out attribute recognition for the object ([0022] - if the mAP values are greater than 0.50, the 2D bounding box and the corresponding 3D bounding box are sufficiently overlapped so as to properly annotate the recognized object); and correcting, if the relative overlap does not exceed the threshold value, the bounding boxes ([0022] - the controller 116 re-trains the 3D DNN for improved performance in an iterative manner).
With regard to claim 2, Li discloses the method according to claim 1, wherein prior to grouping a sensor data frame of the spatial sensor and a sensor data frame of the area sensor, based on a temporal correlation of the measuring points in time, tracking of objects in sequential sensor data frames of the spatial sensor and/or tracking of objects in sequential sensor data frames of the area sensor takes place (tracking of the objects is inherent to the perception sensors 128 affixed to the autonomous vehicle).
With regard to claim 3, Li discloses the method according to claim 1, wherein the projection of the corners of a bounding box in the sensor data frame of the spatial sensor into the image plane of the area sensor includes selection of a rectangle or the largest rectangle obtained from the projection, and a regression of the size takes place for the projected rectangle and/or the bounding box of the area sensor (see [0021]-[0022] and the re-training of the 3D DNN so as to adjust the size of the bounding boxes in order to improve performance).
With regard to claim 4, Li discloses the method according to claim 1, wherein the correction of the bounding boxes includes receiving corrected annotations for the sensor data frames of the sample and retraining the neural network, using the sensor data frames of the sample ([0022] - re-training occurs).
With regard to claim 5, Li discloses the method according to claim 1, wherein the correction of the bounding boxes when a bounding box is present in the sensor data frame of the spatial sensor includes receiving a determination of whether an incorrect object recognition was present, and if an object was actually present, projection of the corners of the three-dimensional bounding box of the object into the image plane of the area sensor takes place in order to obtain a projected rectangle, and a regression of the size of the projected rectangle is subsequently carried out ([0022] and re-training takes place of the 3D bounding box and 2D bounding boxes if they do not sufficiently overlap which would inherently imply that an incorrect object recognition took place).
With regard to claims 10 and 11, Li cites a computer readable medium, processor, memory, etc at [0023].
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of US 2022/0139094 to Yoshimi.
Li discloses the method according to claim 1, wherein the correction of the bounding boxes when a bounding box is present in the sensor data frame of the area sensor includes receiving a determination of whether an incorrect object recognition was present, and if an object was actually present, projection of measuring points of the point cloud of the spatial sensor into the image plane of the area sensor takes place (see [0022]). However, Li fails to disclose highlighting the measuring points projected that are within the 2D bounding box.
On the other hand, Yoshimi discloses in Figure 10 a projection of point cloud data points from a LIDAR sensor onto a 2D bounding box wherein the points within the 2D bounding box are highlighted (Figure 11).
Therefore, it would have been obvious before the effective filing date of the claimed invention to have provided the object recognition system of Li with the added ability to highlight point cloud data points within the corresponding 2D bounding box as taught by Yoshimi since doing this would improve the accuracy of the automatic annotation of the 3D point cloud as taught by Yoshimi at [0094]-[0095].
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of US 2023/0410534 to Kanzawa.
Li discloses the fusion of 2D imaging data with 3D point clod data for object recognition as discloses in reference to claim 1 above. However, Li fails to disclose multiple cameras being employed for the object recognition process (i.e. “at least two area sensors, or cameras, the measuring ranges of the spatial sensor and of the first area sensor spatially overlapping in a first overlap area, and the measuring ranges of the spatial sensor and of the second area sensor spatially overlapping in a second overlap area; for objects in the first overlap area an automatic annotation takes place independently of sensor data frames of the second area sensor, and for objects in the second overlap area an automatic annotation takes independently of sensor data frames of the first area sensor” as set forth in claim 7.
In the same field of endeavor, Kanazawa discloses in Figures 7 and 8 an autonomous vehicle equipped with two cameras 103 and 203 wherein the object recognition is performed between the two cameras (see [0073]-[0078]) with automatic tagging of the objects detected).
Therefore, it would have been obvious before the effective filing date of the claimed invention to have to have provided the object recognition process of Li with the addition of a second camera as taught by Kanzawa as doing so would have provided for an alternative view of the scene surrounding the autonomous vehicle, and subsequently enhancing the ability of the vehicle to “see” objects that may have been occluded within the first camera’s field of view and thus more accurately annotate the recognized objects.
Allowable Subject Matter
Claims 8 and 9 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, in addition to correction of the 35 U.S.C. 112 rejection and claim objection above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The prior art cited is reflective of the state of the art surrounding the fusion of LIDAR and camera sensors in autonomous vehicles where bounding boxes are place around annotated objects surrounding the vehicle.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID OMETZ whose telephone number is (571)272-7593. The examiner can normally be reached M-F, 8am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DAVID OMETZ
Primary Examiner
Art Unit 2672
/DAVID OMETZ/Primary Examiner, Art Unit 2672
Prosecution Insights