DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The IDS(s) has/have been considered and placed in the application file.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7 and 10-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Friedman et al. (US 2022/0292705 A1 – hereinafter “Friedman”) in view of Podkamien et al. (US 2023/0168364 A1 – hereinafter “Podkamien”) in view of Gronau (US 20240265554 A1 – hereinafter “Gronau”).
Claim 1, 11, and 18.
Friedman discloses a processor comprising: one or more processing units to (FIG. 2A "Processor 152"; ¶106 discloses "The control board 150 may comprise one or more of processors 152, memory 154 and communication circuitry 156.")
generate, based at least on applying a representation of sensor data to one or more machine learning models, different types of predictions of age or presence of one or more detected occupants (Fig. 8, 820 discloses “Apply a pose detection algorithm on each of the obtained sequence of 2D images to yield a 2D skeleton representation of said one or more occupants”;
¶8 discloses “combine one or more 3D image of said sequence of 3D images with said one or more skeleton representations … to yield at least one skeleton model for each one
or more occupants … analyze the one or more skeleton models to extract one or more features of each of the one or more occupants”; ¶¶ 23, 160 discloses “In an embodiment, the pose detection algorithm is an OpenPose algorithm.”);
generate a representation of whether a child is present based at least on a combined assessment of the different types of predictions of the age or the presence of the one or more detected occupants (¶134 discloses “the integration process includes computationally combining the formed skeleton (2D skeleton) and the depth maps representation to yield the skeleton model which includes data for each key-point in the skeleton model in an (x,y,z) coordinate system.; ¶165 discloses “a mass classification (282) for each identified object in the scene, such as objects 254 and 255 may be determined in accordance with a number of pre-determined mass categories, e.g. child; teenager; adult.”) (Friedman ¶87 discloses “a face detector sensor and/or face detection and/or face recognition software module for analyzing the captured 2D and/or 3D images.”; Classifying a face into an age range is a well-understood application of facial analysis. Podkamien ¶111 discloses “The radar receiver array 310 is coupled to a
memory 326 … a neural network may be trained to identify and categorize passengers, mapping them to classifications such as age category); and
execute one or more operations based at least on the representation of whether the child is present (¶¶15-16 discloses “the output signals are associated with an operation of one or more of the vehicle's units. In an embodiment, the vehicle's units are selected from the group consisting of: airbag; Electronic Stabilization Control (ESC) Unit; safety belt.”).
Friedman discloses all of the subject matter as described above except for specifically teaching “different types” of predictions and “wherein the combined assessment is based at least on determining a confidence level.” However, Podkamien in the same field of endeavor teaches different types of predictions (Podkamien discloses radar-based system for monitoring a vehicle cabin: Abstract discloses “Vehicle cabin monitoring using a radar unit centrally positioned within the cabin to obtain image data of the vehicle cabin and a processor to generate detect occupancy of seats within the vehicle cabin, categorize occupants, detect posture, determine seatbelt status and monitor life signs of the occupants.” ¶84 discloses “Embodiments of the invention use a single sensor to track both occupancy and movements within the cabin of a vehicle.” Determining and categorizing movements, including heartbeat and breathing: ¶149 discloses “The system is also able to determine and categorize movements of each occupant, including heart beat and breathing, for example.” ¶22 discloses “In addition, the data detected which includes both macro and minor movements over time, can monitor posture, hand gestures, breathing and heart rate.” Monitoring posture and classifying occupants: ¶21 discloses “The radar sensor array is configured to monitor the cabin and the objects and passengers within the cabin, and can differentiate between different kinds of passengers, such as adults and children, babies, pets and inanimate objects.” ¶148 discloses “the signal may be analysed or compared with a signal for various targets 610, such as adults, children, pets, babies, and inanimate objects”). Neither Friedman nor Podkamien details the mathematical mechanism for combining different types of age predictions based on confidence levels. However, Gronau teaches and in-cabin vehicle monitoring system that predicts occupant characteristics, including age (Grouau ¶125). Gronau further teaches combining multiple predictions to form a combined assessment (¶¶221-222) , reading on the claimed “wherein the combined assessment is based at least on determining a confidence level.” Gronau’s ¶¶221-222 shows generating a first prediction (“current state”), generating a second prediction (“new state”), and combining them into a final assessment (“updated state”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Friedman and Podkamien before the effective filing date of the claimed invention. The motivation for this combination of references would have been to a predictable improvement to enhance Friedman’s pose-based classification system by integrating the different, complementary prediction types as taught by Podkamien (e.g., heartbeat and breathing detection). This combination would yield a more robust and reliable system, capable of confirming the presence of a living occupant and improving classification accuracy, thus rendering the claim obvious.
It would also have been obvious to one of ordinary skill in the art to modify the multi-modal occupant monitoring systems of Friedman and Podkamien to evaluate the confidence levels (i.e. uncertainties) of the different age predictions to form the combined assessment, as taught by Gronau. The motivation would be to predictably improve the reliability of the system’s final age assessment by algorithmically favoring the sensor prediction with the highest certainty, mitigating the risk of false positives from conflicting camera and radar data.
Claims 2, 12, and 19.
The combination of Friedman, Podkamien, and Gronau discloses the processor of claim 1, the one or more processing units further to generate a first prediction of the different types of predictions based at least on using a first machine learning model of the one or more machine learning models to detect a pose (Friedman Fig. 8, 820 discloses “Apply a pose detection algorithm on each of the obtained sequence of 2D images to yield a 2D skeleton representation of said one or more occupants”; ¶8 discloses “combine one or more 3D image of said sequence of 3D images with said one or more skeleton representations … to yield at least one skeleton model for each one or more occupants … analyze the one or more skeleton models to extract one or more features of each of the one or more occupants”; ¶¶ 23, 160 discloses “In an embodiment, the pose detection algorithm is an OpenPose algorithm.”), estimating limb length based at least on the detected pose, and using a second machine learning model of the one or more machine learning models to regress age based at least on the limb length.
Claims 3 and 13.
The combination of Friedman, Podkamien, and Gronau discloses the processor of claim 1, the one or more processing units further to generate a first prediction of the different types of predictions based at least on using a first machine learning model of the one or more machine learning models to detect a pose (Friedman ¶160 discloses “an OpenPose algorithm on the images and/or other DNN algorithms such as DensePose configured to extract body pose.”), estimating limb length based at least on the detected pose, and using a mapping that associates the limb length with a corresponding age range (Friedman Fig’s 4C-4G determines occupant size by first detecting features (key-points ¶133) in a 2D image, generating a 3D pose estimate (¶132), then correlating it with depth data (e.g., from a point cloud ¶134) to scale the pose estimate to real world dimensions. Limb length can be determined from the scaled 3D pose.).
Claims 4 and 14.
The combination of Friedman, Podkamien, and Gronau discloses the processor of claim 1, the one or more processing units further to generate a first prediction of the different types of predictions based at least on classifying a detected face of an occupant of the one or more detected occupants into one of a plurality of age ranges (Friedman ¶87 discloses “a face detector sensor and/or face detection and/or face recognition software module for analyzing the captured 2D and/or 3D images.”; Classifying a face into an age range is a well-understood application of facial analysis.).
Claims 5 and 15.
The combination of Friedman, Podkamien, and Gronau discloses the processor of claim 1, wherein the different types of predictions of the age of an occupant of the one or more detected occupants comprise a first estimated age predicted based at least on a detected face of the occupant (Friedman ¶87 discloses “a face detector sensor and/or face detection and/or face recognition software module for analyzing the captured 2D and/or 3D images.”; Classifying a face into an age range is a well-understood application of facial analysis.), a second estimated age predicted based at least on an estimated size of the occupant (Friedman ¶165 discloses “a mass classification (282) for each identified object in the scene, such as objects 254 and 255 may be determined in accordance with a number of pre-determined mass categories, e.g. child; teenager; adult.”), and a third estimated age predicted based at least on a RADAR classification of the occupant (Podkamien ¶111 discloses “The radar receiver array 310 is coupled to a memory 326 which stores the signals received by receiver 310 … a neural network may be trained to identify and categorize passengers, mapping them to classifications such as age category and in-position/out-of position states.”). The motivation to combine is the same as for the independent claims.
Claims 6 and 16.
The combination of Friedman, Podkamien, and Gronau discloses the processor of claim 1, wherein the different types of predictions of the presence of an occupant of the one or more detected occupants comprise a classification of the occupant as a child predicted based at least on detecting a child seat in a first slot and classifying the slot as being occupied based at least on RADAR data (Podkamien ¶111 discloses “The radar receiver array 310 is coupled to a memory 326 which stores the signals received by receiver 310 … a neural network may be trained to identify and categorize passengers, mapping them to classifications such as age category and in-position/out-of position states.”). The motivation to combine is the same as for the independent claims.
Claim 7.
The combination of Friedman, Podkamien, and Gronau discloses the processor of claim 1, the one or more processing units further to generate the representation of whether the child is present based at least on applying a representation of the different types of predictions of the age or the presence of the one or more detected occupants to one or more subsequent machine learning models to generate one or more predicted values representative of the age of the one or more detected occupants (Friedman ¶142 discloses “The mass prediction module 224 obtains the valid images of the objects from the skeleton model data … the pre-trained regression module to provide the most accurate mass prediction (e.g. estimation prediction) for each captured object (e.g. persons).).
Claims 10, 17, and 20.
The combination of Friedman, Podkamien, and Gronau discloses the processor of claim 1, wherein the processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (Friedman ¶224 discloses “In further embodiments, software modules are hosted on cloud computing platforms.”).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Friedman, Podkamien, and Gronau as applied to claim 1 above, and further in view of Weng et al. (US 11713600 B1 – hereinafter “Weng”).
Claim 8.
The combination of Friedman, Podkamien, and Gronau processor of claim 1, wherein the sensor data comprises one or more RGB images and one or more infrared images, the one or more processing units further to generate the different types of predictions of the age or the presence of the one or more detected occupants (Friedman Fig. 8, 820, ¶¶ 8, 23, 160; Podkamien Abstract, ¶¶ 21-22, 148-149);
(Friedman Fig. 8, 820, ¶¶ 8, 23, 160).
Friedman, Podkamien, and Gronau discloses all of the subject matter as described above except for specifically teaching “based at least on applying a combined representation of the one or more RGB images and the one or more infrared images.” However, Weng in the same field of endeavor teaches “based at least on applying a combined representation of the one or more RGB images and the one or more infrared images” (C8:L20-25 discloses “the sensor 140a may implement an RGB-InfraRed (RGB-IR) sensor.”).
Therefore, it would have been obvious to one of ordinary skill in the art to Friedman, Podkamien, Gronau, and Weng before the effective filing date of the claimed invention. The motivation for this combination of references would have been to incorporate both RGB and IR sensors to provide robust image analysis across all lighting conditions.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Friedman, Podkamien, and Gronau as applied to claim 1 above, and further in view of Moustafa et al. (WO 2020205597 A1 – hereinafter “Moustafa”).
Claim 9.
The combination of Friedman, Podkamien, and Gronau processor of claim 1, wherein the one or more machine learning models comprise a face-based age estimator (Friedman ¶87 discloses “a face detector sensor and/or face detection and/or face recognition software module for analyzing the captured 2D and/or 3D images.”; Classifying a face into an age range is a well-understood application of facial analysis.) (Friedman ¶165 discloses “child; teenager; adult.”)
Friedman, Podkamien, and Gronau discloses all of the subject matter as described above except for specifically teaching “trained based at least on one or more synthetic images of one or more synthetic faces generated … to one or more text-to-image generators.” However, Moustafa in the same field of endeavor teaches “trained based at least on one or more synthetic images of one or more synthetic faces generated … to one or more text-to-image generators.”(¶447 and Fig. 52 discloses a method for generating synthetic data “At 5206, the plurality of text keywords of the context are provided to synthetic image generator, the synthetic image generator to generate a plurality of images based on the plurality of text keywords of the context.”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Friedman, Podkamien, Gronau, and Moustafa before the effective filing date of the claimed invention because Friedman’s system needs accurate age classification for airbag deployment decisions and synthetic data generation solves the data scarcity problem. Moustafa provides controllable, scalable synthetic face generation. Combining these approaches would predictably improve the robustness and accuracy of age estimation in vehicle safety systems.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674