DETAILED ACTION
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-4, 6-12, 14-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Chen (US 20230249712 ).
Regarding claim 1, Chen teaches a method comprising:
obtaining ultrasonic data generated using one or more ultrasonic sensors of a machine (116 in Fig. 1);
obtaining image data generated using one or more image sensors of the machine( 118, 119 in Fig. 1);
obtaining RADAR data generated using one or more RADAR sensors of the machine ( 114 in Fig. 1));
processing, using one or more neural networks, an aggregation of the ultrasonic data, the image data, and the RADAR data, the processing including:
performing first processing that processes the ultrasonic data separately from the image data and the RADAR data ( 301 in Fig. 3; [0058]-[0059], input data 301 generated by the respective sensor (e.g., night-vision camera, ultrasound sonar sensor, etc.)…Each of NNs 312-318 can process a respective modality of input data 301);
performing second processing that processes the image data separately from the RADAR data and the ultrasonic data( 306, 308, 316, 318 in Fig. 3);
performing third processing that processes the RADAR data separately the ultrasonic data and the image data(304, 314 in Fig. 3); and
performing fourth processing that processes data associated with the ultrasonic data, the image data, and the RADAR data together(330 in Fig. 3);
generating output data corresponding to one or more objects located in an area based at least on the first processing, the second processing, the third processing, and the fourth processing(331-335 in Fig. 3); and
causing a machine to perform one or more operations based at least on the output data (140 in Fig. 1).
Regarding claim 2, Chen teaches the method of claim 1, wherein the generating of the output data is based at least on the one or more neural networks processing an input data set that is generated based at least on:
first input data that is generated by the first processing based at least on the ultrasonic data( data ( 301 in Fig. 3; [0058]-[0059], input data 301 generated by the respective sensor (e.g., night-vision camera, ultrasound sonar sensor, etc.));
second input data that is generated by the second processing based at least on the image data( 306, 308 in Fig. 3;); and
third input data that is generated by the third processing based at least on the RADAR data( 304 in Fig. 3;).
and combined input data that is generated by the fourth processing based at least on aggregating the first input data, the second input data, and the third input data(330 in Fig. 3);.
Regarding claim 3, Ye teaches the method of claim 2, wherein the third input data includes one or more of:
one or more first RADAR data sets that respectively correspond to one or more individual RADAR scans( [0067], radar-based tracking may be performed …, not frame by frame, but rather using doppler analysis to track movements); or
one or more second RADAR data sets that are respectively aggregated from two or more first RADAR data sets( [0014], multiple radar devices can be mounted on a vehicle ).
Regarding claim 4, Chen teaches the method of claim 2, wherein: t
he first input data corresponds to a first map indicating respective locations in the area of one or more first objects as indicated by the ultrasonic data([0026], active sound probing of the driving environment 101, e.g., ultrasonic sonars, and one or more microphones for passive listening to the sounds of the driving environment);
the second input data corresponds to a second map indicating respective locations in the area of one or more second objects as indicated by the image data([0015], A camera (e.g., a photographic or video camera) allows high-resolution imaging of objects at both shorter and longer distances);
the third input data corresponds to a third map indicating respective locations in the area of one or more third objects as indicated by the RADAR data([0013], radars and lidars emit electromagnetic signals (radio signals or optical signals) that reflect from the objects and carry information allowing to determine distances to the objects); and
the combined input data corresponds to a fourth map indicating respective locations in the area of one or more of: the one or more first objects, the one or more second objects, or the one or more third objects as indicated by the aggregating of the ultrasonic data, the image data, and the RADAR data( [0016], With various sensors providing different benefits, an autonomous vehicle sensing system typically deploys sensors of multiple types, leveraging each sensor's advantages to obtain a more complete picture of the driving environment).
Regarding claim 6, Chen teaches the method of claim 1, wherein:
the first processing includes extracting a first feature data set based at least on first feature processing performed with respect to the ultrasonic data using a first feature extractor of the one or more neural networks; the second processing includes extracting a second feature data set based at least on second feature processing performed with respect to the image data using a second feature extractor of the one or more neural networks; the third processing includes extracting a third feature data set based at least on third feature processing performed with respect to the RADAR data using a third feature extractor of the one or more neural networks; ( [0059], Each of NNs 312-318 can process a respective modality of input data … and can output respective feature vectors)
the fourth processing includes generating a combined feature data set based at least on fourth feature processing performed with respect to the first feature data set, the second feature data set, and the third feature data set as combined ( [0060], The feature vectors output by NNs 312-318 can be combined (e.g., concatenated) into an aggregated feature vector 322) ; and
the generating of the output data is based at least on the first feature data set, the second feature data set, the third feature data set, and the combined feature data set( [0060], An output of fusion NN 330 can be provided as an input into one or more classification heads 331-335, … classification heads 331-335 may be combined into a single classification head that uses a softmax function and outputs multiple classification probabilities).
Regarding claim 7, Chen teaches the method of claim 6, wherein the fourth feature processing is performed using one or more of the first feature extractor, the second feature extractor, the third feature extractor, or a fourth feature extractor of the one or more neural networks( [0059], Each of NNs 312-318 can process a respective modality of input data 301 (e.g., lidar NN can process lidar data 302, camera NN 316 can process camera data 306, and so on).
Regarding claim 8, Chen teaches the method of claim 6, wherein the extracting of the first feature data set, the second feature data set, and the third feature data set is performed in parallel( [0059], Each of NNs 312-318 can process a respective modality of input data 301 (e.g., lidar NN can process lidar data 302, camera NN 316 can process camera data 306, and so on; Fig. 3, 312, 314, 316, 318 are all in parallel).
Regarding claim 9, Ye teaches the method of claim 1, wherein the output data includes one or more of:
an occupancy map;
an evidence grid map;
a height map; or
a distance map ([0015], With various sensors providing different benefits, an autonomous vehicle sensing system typically deploys sensors of multiple types, leveraging each sensor's advantages to obtain a more complete picture of the driving environment. For example, a lidar can accurately determine a distance to an object and the radial velocity of the object).
Claims 10-12, 14-17 recite the system for the method in claims 1-4, 6-9. Since Chen also teaches a system( Fig. 6), those claims are also rejected.
Regarding claim 18, Chen teaches the system of claim 10, wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine ( [0022], various systems and methods may be described below in conjunction with autonomous vehicles, similar techniques can be used in various driver assistance systems that do not rise to the level of fully autonomous driving systems);
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for presenting at least one of augmented reality content, virtual reality content, or mixed reality content;
a system for hosting one or more real-time streaming applications;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for performing one or more generative AI operations; a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
Claims 19-20 recite the processor(s) for the method in claims 1-2. Since Chen also teaches a processor( 602 in Fig. 6), those claims are also rejected.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANGENG SUN whose telephone number is (571)272-3712. The examiner can normally be reached 8am to 5pm, EST, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Randolph Vincent can be reached at 571 272 8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIANGENG SUN
Examiner
Art Unit 2661
/Jiangeng Sun/Examiner, Art Unit 2671
Prosecution Insights