Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Applicants Remarks/Arguments
In regards to the applicants remarks that Wekel does not teach the language in claim 9 which has now been amended into claim 1, the examiner respectfully disagrees. The examiner further cites Wekel [0029] for clarifying that Wekel does indeed teach And comparing likelihood values in the estimated sensor point cloud with a predetermined threshold ([0029] “The predictions may be compared against the encoded ground truth information using one or more loss functions applied by a DNN training engine,” is understood to be the same as the claimed comparing likelihood values with a predetermined threshold in light of instant specifications [0013])
to determine whether training of the deep learning network is complete. ([0029] “and the DNN parameters (e.g., weights and biases) may be updated accordingly until the DNN(s) converges to an acceptable level of accuracy.” )
Drawings
The drawings were received on 08/25/2023. These drawings are accepted.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 5, 8-9, 11-14, 15-17 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Chakravarty et al. (US11189049, hereinafter “Chakravarty”) and in view of Wekel et al (US20210063578, hereinafter “Wekel ”)
Claim 1. (Currently Amended) Chakravarty teaches A method comprising: receiving vision sensor data; (col2 line32 “input a color or reg, green, blue (RGB) image acquired by a sensor”)
processing the vision sensor data to estimate a sensor point cloud distribution, (col11line33 “input an RGB image (RGB) 602 and output a semantic point cloud (SPC) 610.”)
wherein the processing is performed using a deep learning network (col9line6 “A stereo point cloud image can also be determined by training a convolutional neural network (CNN)”) trainable using only vision sensor training data and other training data corresponding to the vision sensor training data as input training data; (col9 line16 “using a training dataset that includes pairs of stereo images 302, 304 along with ground truth point cloud images 402” the ground truth point cloud images are understood to be the same as the claimed other training data in light of instant specifications [0009]) and
producing an estimated sensor point cloud based on the estimated sensor point cloud distribution. (col13 line11 “outputs a semantic point cloud 610”)
Chakravarty does not explicitly teach and comparing likelihood values in the estimated sensor point cloud with a predetermined threshold to determine whether training of the deep learning network is complete.
Wekel teaches and comparing likelihood values in the estimated sensor point cloud with a predetermined threshold ([0029] “The predictions may be compared against the encoded ground truth information using one or more loss functions applied by a DNN training engine,” is understood to be the same as the claimed comparing likelihood values with a predetermined threshold in light of instant specifications [0013])
to determine whether training of the deep learning network is complete. ([0029] “and the DNN parameters (e.g., weights and biases) may be updated accordingly until the DNN(s) converges to an acceptable level of accuracy.” )
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify Chakravarty to have point cloud data that exceeds a threshold and determine if training is complete as taught by Wekel to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been to (Wekel et al Abstract “enable safe planning and control of the autonomous vehicle.”)
Claim 2. (Original) Chakravarty and Wekel teach The method of claim 1,
Chakravarty teaches wherein the vision sensor data includes one or more camera images (col2line27 “RGB image from a stereo image pair.”)
Claim 3. (Original) Chakravarty and Wekel teach The method of claim 1,
Chakravarty teaches wherein the sensor point cloud distribution is a radar point cloud distribution including one or more of: range, azimuth, elevation, and Doppler information. (col2line36 “A point cloud image is point cloud data that includes distances or range to points in the image.”)
Claim 5. (Original) Chakravarty and Wekel teach The method of claim 1,
Chakravarty teaches wherein the vision sensor data is the vision sensor training data, the method further comprising: comparing the estimated sensor point cloud distribution with the other training data (col12line15 “trained using semantic point cloud data as ground truth.”) corresponding to the vision sensor training data (col10line51 “image data”) to obtain a loss function; (col10line51 “trained by determining a loss function which measures how accurately the VAE has encoded and decoded the image data.”) and
updating the deep learning network based on the loss function. (col11line20“The loss function is used to train the encoder and decoder”)
Claim 8. (Original) Chakravarty and Wekel teach The method of claim 1, further comprising
Chakravarty does not explicitly teach generating a sensor point cloud from the estimated sensor point cloud distribution as a realization with likelihood values that exceed a threshold.
Wekel teaches generating a sensor point cloud from the estimated sensor point cloud distribution as a realization ([0140] “3D location estimates of the object obtained from … sensors (e.g., LiDAR sensor(s)”) with likelihood values that exceed a threshold. ([0140]“system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections.”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify Chakravarty to have point cloud data that exceeds a threshold as taught by Wekel to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been to (Wekel et al Abstract “enable safe planning and control of the autonomous vehicle.”)
Claim 9. (Original) Chakravarty and Wekel teach The method of claim 1,
Chakravarty teaches wherein the vision sensor data is the vision sensor training data, the method further comprising: comparing the estimated sensor point cloud distribution with the other training data (col12line15 “trained using semantic point cloud data as ground truth.”) corresponding to the vision sensor training data (col10line51 “image data”) to obtain a loss function; (col10line51 “trained by determining a loss function which measures how accurately the VAE has encoded and decoded the image data.”)
Chakravarty does not explicitly teach generating a sensor point cloud from the estimated sensor point cloud distribution as a realization; and
comparing likelihood values in the sensor point cloud with a predetermined threshold to determine whether training of the deep learning network is complete.
Wekel teaches generating a sensor point cloud from the estimated sensor point cloud distribution as a realization; ([0140] “3D location estimates of the object obtained from … sensors (e.g., LiDAR sensor(s)”) and
comparing likelihood values in the sensor point cloud with a predetermined threshold ([0029] “The predictions may be compared against the encoded ground truth information using one or more loss functions”)
to determine whether training of the deep learning network is complete. ([0029] “applied by a DNN training engine, and the DNN parameters (e.g., weights and biases) may be updated accordingly until the DNN(s) converges to an acceptable level of accuracy.” )
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify Chakravarty to have point cloud data that exceeds a threshold and determine if training is complete as taught by Wekel to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been to (Wekel et al Abstract “enable safe planning and control of the autonomous vehicle.”)
Claim 11. (Original) Chakravarty and Wekel teach The method of claim 1, further comprising:
Chakravarty teaches receiving sensor data corresponding to the vision sensor data; and
modifying the estimated sensor point cloud distribution based on the sensor data. (col2line48 “updating the topological map with new data acquired by vehicles traversing the route,”)
Claim 12. (Original) Chakravarty and Wekel teach The method of claim 11,
Chakravarty teaches wherein the modifying further includes: determining a correlation between the sensor data and the estimated sensor point cloud distribution; (col3line10 “point cloud image can be determined based on a single monocular image”) and
modifying the estimated sensor point cloud distribution based on the correlation. (col14line50 “updating … semantic point clouds 626 determined based on input RGB images 602 acquired as the vehicle 110 traverses a give route.10”)
Claim 13. (Original) Chakravarty and Wekel teach The method of claim 1,
Chakravarty teaches further comprising controlling a vehicle based on the estimated sensor point cloud distribution. (col1line8 “operate the vehicle based on the data.”)
Claim 14. (Original) Chakravarty and Wekel teach The method of claim 1, further comprising
Chakravarty does not explicitly teach providing a notification to an occupant of a vehicle based on the estimated sensor point cloud distribution.
Wekel teaches providing a notification to an occupant of a vehicle ([0181] “alert the driver to a hazard, so that the driver may take corrective action.”) based on the estimated sensor point cloud distribution. ([0100] “controller(s) 836 may receive inputs (e.g., represented by input data) from an instrument cluster 832 of the vehicle 800 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 834, an audible annunciator, a loudspeaker”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify Chakravarty to have notification to the driver based on the point cloud distribution as taught by Wekel to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been to (Wekel et al Abstract “enable safe planning and control of the autonomous vehicle.”)
Claim 15. (Currently Amended) Chakravarty teaches A non-transitory computer readable medium (col16line28 “computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium,”)embodying a set of executable instructions, the set of executable instructions to manipulate at least one processor (abstract “the memory including instructions to be executed by the processor”) to: receive vision sensor data; (col2 line32 “input a color or reg, green, blue (RGB) image acquired by a sensor”)
process the vision sensor data to estimate a sensor point cloud distribution, (col11line33 “input an RGB image (RGB) 602 and output a semantic point cloud (SPC) 610.”)
wherein the processing is performed using a deep learning network (col9line6 “A stereo point cloud image can also be determined by training a convolutional neural network (CNN)”) trainable using only vision sensor training data and other training data corresponding to the vision sensor training data as input training data; (col9 line16 “using a training dataset that includes pairs of stereo images 302, 304 along with ground truth point cloud images 402” the ground truth point cloud images are understood to be the same as the claimed other training data in light of instant specifications [0009]) and
produce an estimated sensor point cloud distribution based on the estimated sensor point cloud distribution. (col13 line11 “outputs a semantic point cloud 610”)
Chakravarty does not explicitly teach and comparing likelihood values in the estimated sensor point cloud with a predetermined threshold to determine whether training of the deep learning network is complete.
Wekel teaches and comparing likelihood values in the estimated sensor point cloud with a predetermined threshold ([0029] “The predictions may be compared against the encoded ground truth information using one or more loss functions applied by a DNN training engine,” is understood to be the same as the claimed comparing likelihood values with a predetermined threshold in light of instant specifications [0013])
to determine whether training of the deep learning network is complete. ([0029] “and the DNN parameters (e.g., weights and biases) may be updated accordingly until the DNN(s) converges to an acceptable level of accuracy.” )
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify Chakravarty to have point cloud data that exceeds a threshold and determine if training is complete as taught by Wekel to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been to (Wekel et al Abstract “enable safe planning and control of the autonomous vehicle.”)
Claim 16. (Original) Chakravarty and Wekel teach The non-transitory computer readable medium of claim 15,
Chakravarty teaches wherein the vision sensor data is the vision sensor training data and the set of executable instructions further manipulate the at least one processor to:compare the estimated sensor point cloud distribution with the other training data (col12line15 “trained using semantic point cloud data as ground truth.”) corresponding to the vision sensor training data (col10line51 “image data”) to obtain a loss function; (col10line51 “trained by determining a loss function which measures how accurately the VAE has encoded and decoded the image data.”)
and update the deep learning network based on the loss function. (col11line20“The loss function is used to train the encoder and decoder”)
Claim 17. (Original) Chakravarty and Wekel teach The non-transitory computer readable medium of claim 15,
Chakravarty teaches wherein the vision sensor data is the vision sensor training data and the set of executable instructions further manipulate the at least one processor to:compare the estimated sensor point cloud distribution with the other training data (col12line15 “trained using semantic point cloud data as ground truth.”) corresponding to the vision sensor training data (col10line51 “image data”) to obtain a loss function; (col10line51 “trained by determining a loss function which measures how accurately the VAE has encoded and decoded the image data.”)
Chakravarty does not explicitly teach generate a sensor point cloud from the estimated sensor point cloud distribution as a realization; and
compare likelihood values in the sensor point cloud with a predetermined threshold to determine whether training of the deep learning network is complete.
Wekel teaches generate a sensor point cloud from the estimated sensor point cloud distribution as a realization; ([0140] “3D location estimates of the object obtained from … sensors (e.g., LiDAR sensor(s)”) and
compare likelihood values in the sensor point cloud with a predetermined threshold ([0140]“system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections.”)
to determine whether training of the deep learning network is complete. ([0043] “LiDAR point cloud 220 that fall outside of the geometric constraints may be removed in order to generate more accurate ground truth data 110 for training the DNN(s) 126.”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify Chakravarty to have point cloud data that exceeds a threshold and determine if training is complete as taught by Wekel to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been to (Wekel et al Abstract “enable safe planning and control of the autonomous vehicle.”)
Claim 18. (Currently Amended) The device herein has been executed and performed by the CRM of claim 15 and is likewise rejected
Claim 19. (Original) The device herein has been executed and performed by the CRM of claim 16 and is likewise rejected
Claim 20. (Original) The device herein has been executed and performed by the CRM of claim 17 and is likewise rejected
Claims 4 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Chakravarty et al. (US11189049, hereinafter “Chakravarty”) and in view of Wekel et al (US20210063578, hereinafter “Wekel ”) and in view of Cho et al (US20200400810, hereinafter “Cho”)
Claim 4. (Original) Chakravarty includes a radar (Col. 6 lines 54-67) and discloses the method of Claim 3 as outlined above.
Chakravarty and Wekel do not explicitly teach wherein the other training data includes radar points having range and azimuth information, and the estimated sensor point cloud distribution includes estimated radar points having range, azimuth, elevation, and Doppler information.
Cho teaches wherein the other training data includes radar points ([0102] “training ground truth from the original raw radar data 790 based on information corresponding to at least one of dimensions defining the original raw radar data 790”) having range and azimuth information, ([0102] “raw radar data 790 may include as examples one of a … a horizonal angle, an elevation angle, and a range,”) and the estimated sensor point cloud distribution includes estimated radar points having range, azimuth, elevation, and Doppler information. ([0102] “raw radar data 790 may include as examples one of a Doppler velocity, a horizonal angle, an elevation angle, and a range,”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify the proposed combination of Chakravarty and Wekel to have training data contain radar points having range, azimuth, elevation and doppler information as taught by Cho to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been to (Cho et al [0058] “improve accuracy in generating the driving-related information and performing the object recognition.”)
Claim 7. (Original) Chakravarty and Wekel teach The method of claim 1,
Chakravarty and Wekel do not explicitly teach wherein the other training data includes point clouds at a first resolution, and the estimated sensor point cloud distribution has a second resolution higher than the first resolution.
Cho teaches wherein the other training data includes point clouds at a first resolution, ([0008] “low-resolution training input may include selecting low-resolution radar data from the original raw radar data,”) and the estimated sensor point cloud distribution has a second resolution higher than the first resolution. (Abstract “generates high-resolution output data from low-resolution input data” and [0008]“generating of the high-resolution training ground truth”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify the proposed combination of Chakravarty and Wekel to have increasing the resolution of the point cloud data as taught by Cho to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been to (Cho et al [0058] “improve accuracy in generating the driving-related information and performing the object recognition.”)
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Chakravarty et al. (US11189049, hereinafter “Chakravarty”) and in view of Wekel et al (US20210063578, hereinafter “Wekel ”) and in view of Wong et al (US20220028180, hereinafter “Wong”)
Claim 6. Chakravarty and Wekel teach The method of claim 5,
Chakravarty and Wekel do not explicitly teach wherein the loss function maximizes a posteriori probability of the other training data to the estimated sensor point cloud distribution, although producing an optimized result is the inherent reason for using a loss function in training a neural network.
Wong teaches wherein the loss function ([0069] “djusting the loss function,”) maximizes a posteriori probability ([0063] “maximizes the a posteriori probability.”) of the other training data ([0083] “The loss 914 can be backpropagated through the clustering autoencoder 902, such as for training”) to the estimated sensor point cloud distribution. ([0079] “The content 904 can include an image (e.g., a color, black and white, infrared, nighttime, a video frame, or the like), a point set (e.g., a 2D or 3D point set)”…)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify the proposed combination of Chakravarty and Wekel to have a loss function maximize a posteriori probability as taught by Wong to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been so that it would be (Wong [0027] “allows one to use an autoencoder to classify vehicles based on the generated images.”)
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Chakravarty et al. (US11189049, hereinafter “Chakravarty”) and in view of Wekel et al (US20210063578, hereinafter “Wekel ”) and in view of Doemling et al (US20200089236, hereinafter “Doemling”)
Claim 10. (Original) Chakravarty and Wekel teach The method of claim 1,
Chakravarty and Wekel do not explicitly teach wherein the estimated sensor point cloud distribution comprises Gaussian mixture model parameters.
Doemling wherein the estimated sensor point cloud distribution comprises Gaussian mixture model parameters. ([0002] “laser points… point cloud… a Normal distribution transform (NDT) map which uses … Gaussian Mixture Model to represent the environment,”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify the proposed combination of Chakravarty and Wekel to have point cloud distribution using gaussian mixture model as taught by Doemling to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been for (Doemling [0091] “improving computational efficiency in generating Gaussian Mixture Models for map elements.”)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
Wang et al US20230343026 teaches generating a 3D point cloud from 2D images
Xu et al US20240013477 teaches inputting 2D images to output point clouds representing a 3D scene
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OWAIS MEMON whose telephone number is (571)272-2168. The examiner can normally be reached M-F (7:00am - 4:00pm) CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OWAIS IQBAL MEMON/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698