Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Applicants Arguments
In regards to applicants arguments that Chakravarty does not teach sensor point cloud distribution in claim 1, the examiner respectfully disagrees. As understood in the art, Chakravarty’s semantic point cloud is a point cloud distribution. The applicant continues to state that the distribution is a probability distribution and references the specification [0038] however a point cloud distribution is understood in the art to be synonymous with a point cloud as evidenced by Palais et al NPL “POINT CLOUDS: DISTRIBUTING POINTS UNIFORMLY ON A SURFACE” - pg2 ““point clouds”, i.e., relatively dense sets of points distributed evenly over the surface.”
In regards to the applicants arguments that Chakravarty does not teach receiving a general sensor point cloud distribution corresponding to the vision sensor data in claim 1, the examiner respectfully disagrees. The examiner has remapped the current prior art citation for further clarification: receiving a general sensor point cloud distribution (col11line33 “and output a semantic point cloud (SPC) 610.” The claimed receiving a point cloud is understood to be the same as outputting a point cloud in light of instant specifications [0028]) corresponding to the vision sensor data; (col11line33 “input an RGB image (RGB) 602” is understood to be the same as the claimed corresponding to the vision sensor data in light of instant specifications [0028]).
In regards to the applicants arguments that Chakravarty does not teach comparing two point cloud distributions in claim 1, the examiner respectfully disagrees. The claimed comparing is understood to be the same as training in light of instant specifications [0039] which states “method 900 includes comparing the general sensor point cloud distribution with the estimated object location sensor point cloud distribution….Accordingly, the method 900 provides for a fusion of method 300 of FIG. 3 and method 700 of FIG. 7, enabling a CNN such as CNN 104 to train a parametric estimation model”
In regards to the applicants argument that Chakravarty’s bounding boxes are not the same as point cloud distributions in claim 1, the examiner respectfully disagrees. Chakravarty states that the col12 line11 “bounding boxes for each object in a semantic point cloud 610” therefore the bounding boxes include a point cloud distribution.
In regards to the applicants argument that Chakravarty does not teach modifying in claim 5, examiner is persuaded by applicants argument that Chakravarty, Cho and Wekel do not disclose, teach or suggest the concept of “modifying the estimated object location sensor point cloud distribution or the general sensor point cloud distribution based on the comparison.”. The dependent claims 5 and 6 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regards to the applicants argument that Cho does not teach general sensor point cloud distribution and estimated object location sensor point cloud distribution includes radar points having range and azimuth information in claim 4, examiner is persuaded by applicants argument that Chakravarty, Cho and Wekel do not disclose, teach or suggest the concept of “wherein the general sensor point cloud distribution includes radar points having range and azimuth information, and the estimated object location sensor point cloud distribution includes estimated radar points having range, azimuth, elevation, and Doppler information.” The dependent claim 4 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regards to the applicants arguments that the proposed combination of Wekel and Chakravarty do not teach claim 9 due to not comparing two sensor point cloud distributions and identifying a ghost object on the spatial coincidence of low likelihood regions in both distributions, the examiner respectfully disagrees. The argument as stated is not what is claimed in claim 9. Wekel does teach claim 9 in light of broadest reasonable interpretation.
In regards to the applicants arguments that Wekel does not teach claim 11, there is no substantial argument therefore the current rejection still holds.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 7-8, 10 and 12-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Chakravarty et al. (US11189049, hereinafter “Chakravarty”)
Claim 1. (Original) Chakravarty teaches A method comprising:
receiving vision sensor data; (col2 line32 “input a color or reg, green, blue (RGB) image acquired by a sensor”)
processing the vision sensor data to produce an estimated object location sensor point cloud distribution; (col10line30 “to form a semantic point cloud 512. A semantic point cloud 504 is a point cloud image where the point cloud data corresponding to distances is also labeled to identify the type of object or region.”)
receiving a general sensor point cloud distribution (col11line33 “and output a semantic point cloud (SPC) 610.” The claimed receiving a point cloud is understood to be the same as outputting a point cloud in light of instant specifications [0028]) corresponding to the vision sensor data; (col11line33 “input an RGB image (RGB) 602” is understood to be the same as the claimed corresponding to the vision sensor data in light of instant specifications [0028])
comparing the general sensor point cloud distribution (col12line15“semantic point cloud data as ground truth.”) with the estimated object location sensor point cloud distribution; (col12line28“bounding boxes for each object in a semantic point cloud 610…The ground truth including labeled bounding boxes can be compared to the output from the VAE to train the VAE to correctly label point cloud data.” This comparing step is understood to be the same as training in light of instant specifications [0039]. ) and
identifying a vehicle, an environmental object, (col12line23 “3D object detector 704 outputs a semantic point cloud (SM3D) 706 with 3D objects corresponding to vehicles, pedestrians, traffic barriers, etc.”) or a ghost object in the general sensor point cloud distribution (col12line11“detect objects in a semantic point cloud 610”) based on the comparison. (col12line28“The ground truth including labeled bounding boxes can be compared to the output from the VAE to train the VAE to correctly label point cloud data.”)
Claim 2. (Original) Chakravarty teaches The method of claim 1, wherein the vision sensor data includes one or more camera images. (col2line27 “RGB image from a
stereo image pair.”)
Claim 3. (Original) Chakravarty teaches The method of claim 1, wherein the general sensor point cloud distribution is a radar point cloud distribution including one or more of: range, azimuth, elevation, and Doppler information. (col2line36 “A point cloud image is point cloud data that includes distances or range to points in the image.” is understood by the examiner to be the same as the claimed radar point cloud distribution including range in light of instant specifications [0019])
Claim 7. (Original) Chakravarty teaches The method of claim 1, further comprising identifying a vehicle (col1line55 “determining locations of objects in an environment around a vehicle. Objects can include other vehicles, pedestrians, traffic barriers, etc.”) in the general sensor point cloud distribution (col13line13 “semantic point clouds can be used as a guide to the location of the dynamic objects in the scene”) when the estimated object location sensor point cloud distribution includes a high likelihood region coinciding with a high likelihood region (col13line17 “reducing background clutter caused by labeled 3D regions in the semantic point cloud not related to 3D objects of interest such as buildings and foliage.” By reducing background clutter, one essentially includes a high likelihood region) of the general sensor point cloud distribution. (col13line16 “The 3D bounding box detection network can then be focused on these regions of the scene, which can speed up the algorithm up considerably”)
Claim 8. (Original) Chakravarty teaches The method of claim 1, further comprising identifying an environmental object (col13line17 “buildings and foliage.”) in the general sensor point cloud distribution when the estimated object location sensor point cloud distribution includes a low likelihood region (col13line17 “reducing background clutter caused by labeled 3D regions in the semantic point cloud not related to 3D objects of interest such as buildings and foliage.” The background clutter is the low likelihood region) coinciding with a high likelihood region of the general sensor point cloud distribution. (col13line16 “The 3D bounding box detection network can then be focused on these regions of the scene, which can speed up the algorithm up considerably”)
Claim 10. (Original) Chakravarty teaches The method of claim 1, further comprising controlling a vehicle based on the comparison. (col1line8 “operate the vehicle based on the data.”)
Claim 12. (Original) Chakravarty teaches The method of claim 1, further comprising:
processing the vision sensor data (col11line33 “input an RGB image (RGB) 602”) to estimate a vehicle location (col1line55 “determining locations of objects in an environment around a vehicle. Objects can include other vehicles, pedestrians, traffic barriers, etc.”) sensor point cloud distribution, (col11line33 “input an RGB image (RGB) 602 and output a semantic point cloud (SPC) 610.”) wherein the processing is performed using a deep learning network (col9line6 “A stereo point cloud image can also be determined by training a convolutional neural network (CNN)”) trainable using only vision sensor training imagery and other training data corresponding to the vision sensor training imagery as input training data; (col9line16 “using a training dataset that includes pairs of stereo images 302, 304 along with ground truth point cloud images 402” the ground truth point cloud images are understood to be the same as the claimed other training data in light of instant specifications [0009]) and
producing the vehicle location sensor point cloud distribution (col2line39“A semantic point cloud is a point cloud image includes labels that identify regions within the image corresponding to objects. Regions so labeled can include roadways, sidewalks, vehicles, pedestrians, buildings and foliage, etc.”) based on the estimated vehicle location sensor point cloud distribution. (col2 line39 “ using the three-dimensional point cloud image”)
Claim 13. (Original) Chakravarty teaches The method of claim 1, wherein receiving the vision sensor data and receiving the general sensor point cloud distribution corresponding to the vision sensor data include generating the vision sensor data and the general sensor point cloud distribution using sensors. (col2line33 “color or red, green, blue (RGB) image acquired by a sensor included in a vehicle and output data identifying the closest node of the topological map and a labeled point cloud image”)
Claim 14. (Original) Chakravarty teaches A non-transitory computer readable medium (col16line28 “computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium,”) embodying a set of executable instructions, the set of executable instructions to manipulate at least one processor (abstract “the memory including instructions to be executed by the processor”) to:
receive vision sensor data; (col2 line32 “input a color or reg, green, blue (RGB) image acquired by a sensor”)
process the vision sensor data to produce an estimated object location sensor point cloud distribution; (col11line33 “input an RGB image (RGB) 602 and output a semantic point cloud (SPC) 610.” And col2line39 “A semantic point cloud is a point cloud image includes labels that identify regions within the image corresponding to objects. Regions so labeled can include roadways, sidewalks, vehicles, pedestrians, buildings and foliage, etc.”)
receive a general sensor point cloud distribution corresponding to the vision sensor data; (col11line33 “input an RGB image (RGB) 602 and output a semantic point cloud (SPC) 610.”)
compare the general sensor point cloud distribution (col12line15“semantic point cloud data as ground truth.”) with the estimated object location sensor point cloud distribution; (col12line28“The ground truth including labeled bounding boxes can be compared to the output from the VAE to train the VAE to correctly label point cloud data.”)and
identify a vehicle, an environmental object, (col12line23 “3D object detector 704 outputs a semantic point cloud (SM3D) 706 with 3D objects corresponding to vehicles, pedestrians, traffic barriers, etc.”) or a ghost object in the general sensor point cloud distribution (col12line11“detect objects in a semantic point cloud 610”) based on the comparison. (col12line28“The ground truth including labeled bounding boxes can be compared to the output from the VAE to train the VAE to correctly label point cloud data.”)
Claim 15. (Original) Chakravarty teaches The non-transitory computer readable medium of claim 14, wherein the set of executable instructions further manipulate the at least one processor to:
modify the estimated object location sensor point cloud distribution (col12line15 “trained using semantic point cloud data as ground truth.” Is understood to be the same as the modifying in light of instant specifications [0016]) or the general sensor point cloud distribution(col11line20“The loss function is used to train the encoder and decoder”) based on the comparison. (col10line51 “trained by determining a loss function which measures how accurately the VAE has encoded and decoded the image data.” )
Claim 16. (Original) Chakravarty teaches The non-transitory computer readable medium of claim 15, wherein the modifying includes increasing or decreasing a confidence level associated with a region of the general sensor point cloud distribution. (col9line23 “Ground truth is used to compare to the result output from a CNN when training the CNN to determine when the CNN is outputting a correct result. ” is understood to be the same as the claimed increasing the confidence level in light of instant specifications [0036])
Claim 17. (Original) Chakravarty teaches The non-transitory computer readable medium of claim 14, wherein the set of executable instructions further manipulate the at least one processor to:
identify a vehicle (col1line55 “determining locations of objects in an environment around a vehicle. Objects can include other vehicles, pedestrians, traffic barriers, etc.”) in the general sensor point cloud distribution (col13line13 “semantic point clouds can be used as a guide to the location of the dynamic objects in the scene”) when the estimated object location sensor point cloud distribution includes a high likelihood region coinciding with a high likelihood region (col13line17 “reducing background clutter caused by labeled 3D regions in the semantic point cloud not related to 3D objects of interest such as buildings and foliage.” By reducing background clutter, one essentially includes a high likelihood region) of the general sensor point cloud distribution. (col13line16 “The 3D bounding box detection network can then be focused on these regions of the scene, which can speed up the algorithm up considerably”)
Claim 18. (Original) Chakravarty teaches The non-transitory computer readable medium of claim 14, wherein the set of executable instructions further manipulate the at least one processor to:
identify an environmental object (col13line17 “buildings and foliage.”) in the general sensor point cloud distribution when the estimated object location sensor point cloud distribution includes a low likelihood region (col13line17 “reducing background clutter caused by labeled 3D regions in the semantic point cloud not related to 3D objects of interest such as buildings and foliage.” The background clutter is the low likelihood region) coinciding with a high likelihood region of the general sensor point cloud distribution. (col13line16 “The 3D bounding box detection network can then be focused on these regions of the scene, which can speed up the algorithm up considerably”)
Claim 19. (Original) The method herein has been executed and performed by the CRM of claim 14 and is likewise rejected
Claim 20. (Original) Chakravarty teaches The method of claim 19, wherein the vision sensor data is the vision sensor training data, the method further comprising:
comparing the estimated vehicle location sensor point cloud distribution with the other training data (col12line15 “trained using semantic point cloud data as ground truth.”) corresponding to the vision sensor training data (col10line51 “image data”) to obtain a loss function; (col10line51 “trained by determining a loss function which measures how accurately the VAE has encoded and decoded the image data.”) and
updating the deep learning network based on the loss function. (col11line20“The
loss function is used to train the encoder and decoder”)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 9 and 11 are rejected under 35 U.S.C. 103 as being
unpatentable over Chakravarty et al. (US11189049, hereinafter “Chakravarty”) and
in view of Wekel et al (US20210063578, hereinafter “Wekel ”)
Claim 9. (Original) Chakravarty teaches The method of claim 1, further comprising
Chakravarty does not explicitly teach identifying a ghost object in the general sensor point cloud distribution when the estimated object location sensor point cloud distribution includes a low likelihood region coinciding with a low likelihood region of the general sensor point cloud distribution.
Wekel teaches identifying a ghost object ([0140]“false positive detections.”) in the general sensor point cloud distribution ([0086] “LIDAR point cloud”) when the estimated object location sensor point cloud distribution includes a low likelihood region ([0140] “the system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections.”) coinciding with a low likelihood region of the general sensor point cloud distribution. ([0140] “Such a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections.”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify Chakravarty to have identifying a ghost object based on a confidence level as taught by Wekel to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been to (Wekel et al Abstract “enable safe planning and control of the autonomous vehicle.”)
Claim 11. (Original) Chakravarty teaches The method of claim 1, further comprising
Chakravarty does not explicitly teach providing a notification to an occupant of a vehicle based on the comparison.
Wekel teaches providing a notification to an occupant of a vehicle ([0181] “alert the driver to a hazard, so that the driver may take corrective action.”) based on the comparison. ([0140] “Such a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections.”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify Chakravarty to have providing an alert to the driver based on a comparison as taught by Wekel to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been to (Wekel et al Abstract “enable safe planning and control of the autonomous vehicle.”)
Allowable Subject Matter
Claims 4-6 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
Wang et al US20230343026 teaches generating a 3D point cloud from 2D
images
Xu et al US20240013477 teaches inputting 2D images to output point clouds
representing a 3D scene
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OWAIS MEMON whose telephone number is (571)272-2168. The examiner can normally be reached M-F (7:00am - 4:00pm) CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OWAIS IQBAL MEMON/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698