DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05/14/2026 has been entered.
Response to Amendment and Argument
Applicant’s arguments with respect to pending claims 1-20 filed on 05/14/2026 have been fully considered but the arguments have been rendered moot in view of a new ground(s) of rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wu (US 11951833 B1) in view of Zahid et al. “Vision-Language Models can Identify Distracted Driver Behavior from Naturalistic Videos” JOURNAL OF LATEX CLASS FILES, VOL. , NO. , JUNE 2023.
Regarding claim 1, Wu discloses the claim as follows:
One or more processors comprising processing circuitry to: identify one or more frames of image data that depict at least a portion of an interior space of an ego-machine (Figs. 1-4, 7-9 col. 18, lines 37-67, col. 49, lines 59-66: The example video frame 550 may be a video frame capturing a front row of seats of the interior 402 of the ego vehicle 50. The video frame 550 may comprise an example of in-cabin monitoring); generate…one or more responses indicating whether the one or more conditions associated with at least one of the operator or the occupant are detected in the one or more frames of image data (col. 50, line 51 to col. 51, line 3: The processors 106a-106n may be configured to determine the characteristics of the driver 202 and/or other occupants 502b-502n for a particular one of the seats 454a-454n. In an example, the characteristics of the driver 202 and/or the occupants 502b-502n may comprise an orientation of body parts…the decision module 158 may determine that the driver 202 is potentially distracted (e.g., the driver 202 may be looking down to text on a smartphone or providing input to the infotainment unit 352). Col. 53, lines 24-45: the processors 106a-106n may perform computer vision operations on the video frames...the CNN module 150 may extract characteristics about the objects detected…); and control one or more operations of the ego-machine based at least on the one or more responses (col. 53, lines 46 to col. 54, line 7: If the driver 202 is the person detected providing input to the infotainment unit 352, then the method 600 may move to the step 618. In the step 618, the processors 106a-106n may generate the control signal VCTRL. The signal VCTRL may be configured to cause the CPU 354 of the infotainment unit 352 to disable input (e.g., ignore the input TOUCH_DRV)).
Wu does not teach generate a multimodal prompt comprising the one or more frames of image data and a text prompt representative of a query to determine whether the one or more frames of image data depict one or more conditions associated with at least one of an operator or an occupant; generate, based at least on applying the multimodal prompt to a vision-language model (VLM), one or more responses indicating whether the one or more conditions associated with at least one of the operator or the occupant are detected in the one or more frames of image data.
Zahid teaches generate a multimodal prompt comprising the one or more frames of image data and a text prompt representative of a query to determine whether the one or more frames of image data depict one or more conditions associated with at least one of an operator or an occupant (See for example FIG. 2, illustrating inputs: “image frames” and “text prompts” and output related to a driver activity “talking”, “eating” , “yawning”, “drinking” etc.); generate, based at least on applying the multimodal prompt to a vision-language model (VLM) (i.e., inputs “image frames” and “text prompts”) one or more responses indicating whether the one or more conditions associated with at least one of the operator or the occupant are detected in the one or more frames of image data (i.e., output related to a driver activity “talking”, “eating” , “yawning”, “drinking” etc.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Wu’s in-cabin monitoring system by incorporating the teaching of Zahid, in order to improve the accuracy of the object detection system, as taught by Zahid (Abstract, Section III-V, Conclusion section).
Regarding claim 2, Wu in view of Zahid teaches the one or more processors of claim 1. Zahid further teaches wherein the VLM is updated using one or more training frames (See FIGS. 1-4, Section III: The frame-based models operate by taking inputs similar to the original CLIP model and making predictions on distracted actions…the base model can be fine-tuned for task-specific learning), at least one training frame of the one or more training frames depicting at least one observed operator or occupant, and one or more captions generated by a large language model based at least on metadata associated with the one or more training frames (See for example FIG. 2, illustrating inputs: “image frames” and “text prompts” and output related to a driver activity “talking”, “eating” , “yawning”, “drinking” etc.). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 3, Wu in view of Zahid teaches the one or more processors of claim 1. Zahid further teaches the one or more processors of claim 1, wherein the VLM is updated using one or more captions generated by a large language model (See FIGS. 1-4, Section III: The frame-based models operate by taking inputs similar to the original CLIP model and making predictions on distracted actions…the base model can be fine-tuned for task-specific learning) based at least on metadata that is associated with one or more training frames and represents at least one of: one or more driver or occupant attributes, one or more instructed actions, or one or more session attributes represented in the one or more training frames (See for example FIG. 2, illustrating inputs: “image frames” and “text prompts” and output related to a driver activity “talking”, “eating” , “yawning”, “drinking” etc.).The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 4, Wu in view of Zahid teaches the one or more processors of claim 1. Zahid further teaches the one or more processors of claim 1, wherein the VLM is updated using training data (See FIGS. 1-4, Section III: The frame-based models operate by taking inputs similar to the original CLIP model and making predictions on distracted actions…the base model can be fine-tuned for task-specific learning) corresponding to at least one of: a plurality of driver monitoring tasks or a plurality of occupant monitoring tasks (See FIGS. 1-4, illustrating dataset of multiple driver actions). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 5, Wu teaches the one or more processors of claim 1, wherein the processing circuitry is further to periodically (Col. 32, lines 30-56: Referring to FIG. 4, a diagram illustrating an object comparison between a reference video frame and a captured video frame is shown. The reference video frame 300 and the current video frame 300′ may be video frames processed by the processors 106a-106n (e.g., generated in response to the signals FRAMES_A-FRAMES_N by one of the capture devices 102a-102n)… the reference video frame 300 may be captured periodically. Col. 33, lines 26-41: the reference video frame 300 may provide a reference for when the status of the seat belt 304 is unused (e.g., not being worn by a passenger/driver)). Furthermore, Zahid teaches prompt the VLM to evaluate whether one or more subsequent frames of image data that depict at least the portion of the interior space of the ego- machine depict the one or more conditions associated with at least one of the operator or the occupant (See for example FIG. 2, illustrating inputs: “image frames” and “text prompts” and output related to a driver activity “talking”, “eating” , “yawning”, “drinking” etc.). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 6, Wu in view of Zahid teaches the one or more processors of claim 1. Zahid further teaches one or more processors of claim 1, wherein the processing circuitry is further to prompt the VLM using a representation of the one or more frames sampled from a sliding window of video frames generated using one or more cameras of a driver or occupant monitoring system (Section V: VideoCLIP uses a sliding window of 30 frames, and employs a majority voting scheme on the outputs of these small video clips). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 7, Wu in view of Zahid teaches the one or more processors of claim 1. Zahid further teaches the one or more processors of claim 1, wherein the one or more responses by the VLM indicate one or more results of at least one of driver drowsiness detection, driver distraction detection, driver or occupant out-of-position detection, driver or occupant identification, seatbelt usage detection, occupant presence detection, occupant classification, child presence detection, or gesture recognition performed by the VLM (See FIGS. 1-4, Abstract, Sections I: identifying a distracted driver behavior “talking”, “eating” , “yawning”, “drinking” etc.). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 8, Wu teaches the one or more processors of claim 1, wherein the one or more responses indicate one or more results of occlusion detection performed based at least on the (Col. 43, lines 48-54: the arm 462 may not be visibly connected to the body of the driver 202 (e.g., partially blocked by the steering wheel 408). The processors 106a-106n may determine how likely that the arm 462 is the arm of the driver 202 (e.g., the more likely that the arm 462 belongs to the driver 202, the more the confidence level may be increased)). Furthermore, Zahid teaches the VLM evaluating the one or more frames of image data representing the operator or occupant (See for example FIG. 2, illustrating inputs: “image frames” and “text prompts” and output related to a driver activity “talking”, “eating” , “yawning”, “drinking” etc.). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 9, Wu discloses wherein the one or more operations of the ego- machine comprise at least one of: issuing an audible or visual alert, adjusting one or more in-vehicle infotainment settings, or activating one or more safety systems (col. 3, lines 36-56, col. 54, LINES 1-8: If the driver 202 is the person detected providing input to the infotainment unit 352, then the method 600 may move to the step 618. In the step 618, the processors 106a-106n may generate the control signal VCTRL. The signal VCTRL may be configured to cause the CPU 354 of the infotainment unit 352 to disable input (e.g., ignore the input TOUCH_DRV)).
Regarding claim 10, Wu discloses wherein the one or more processors are comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (col. 13, lines 33-45: an autonomous vehicle implementation. Col. 10, lines 23-36, col. 25, lines 32-40: deep learning techniques).
Regarding claims 11-20, the claims are drawn to a system/method claims and recite the limitation analogous to claims 1-7 and 10, and are rejected due to the same reason set forth above with respect to claims 1-7 and 10.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NATHNAEL AYNALEM whose telephone number is (571)270-1482. The examiner can normally be reached M-F 9AM-5:30 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SATH PERUNGAVOOR can be reached at 571-272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NATHNAEL AYNALEM/ Primary Examiner, Art Unit 2488
