DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 04/08/2025 is/are being considered by the Examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 21, 23, 24, 28, 29, 31, and 35-37 is/are rejected under 35 U.S.C. 103 as being unpatentable over He et al, “A Lightweight Architecture For Driver Status Monitoring Via Convolutional Neural Networks” (published at 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 388-394, December 2019) in view of Lamb et al, U.S. Publication No. 2019/0087713.
Regarding claim 21, He teaches a system comprising:
a camera, the camera configured to capture an image of a driver of a vehicle (see He Figure 1, vision system inside of a vehicle and page 389, second column, “Firstly, the face detection model take the frames captured by near-infrared camera as input and find the faces…Considering that driver's face may not the only face in the picture, we will save the largest face bounding box found for the subsequent model”);
a distraction model (see Figure 2), the distraction model comprising a machine learning model (see page 389, first column, “Meanwhile, we proposed a lightweight architecture of DSM by integrating several proposed model and some state-of-the-art algorithms in deep learning and computer vision”) configured to receive images as input and outputs distraction classification tags (see caption for Figure 2), the machine learning model comprising a backbone network (see Figure 2, stage 1 and page 389, first column, “Firstly, the face detection model take the frames captured by near-infrared camera as input and find the faces. We chose a popular SSD-MobileNet detection model”) and at least one prediction head coupled to the backbone network (see Figure 2, stage 2 with different prediction heads for “landmark”, “head pose”, and “SSD+FPN” which the caption says is for “phone-smoke” i.e. if a phone or smoking is present in an image); and
a processor, the processor configured to load the distraction receive the image, input the image into the distraction model, and receive a distraction classification tag associated with the image (see Figure 2, stage 3 with “state judgement” and page 392, second column, “The GTX 2080Ti GPU was used for training and evaluation”), the distraction classification tag indicating whether a driver depicted in the image is operating the vehicle while distracted (see page 389, first column “Our method take the real-time video frames captured by a single infrared camera as input and detect five status (distraction, drowsiness, yawn, listening call, smoking)” and page 390, first column, “The last one is state judgement stage which can monitor the real-time status of driver through the information extracted from the second phase”).
He does not expressively teach
a computer-readable medium, the computer-readable medium storing [the] distraction model; and
the processor configured to load the distraction model.
However, Lamb in a similar invention in the same field of endeavor teaches a system comprising a model based on machine learning and a processor (see Lamb paragraph [0033]) as taught in He further comprising
a computer-readable medium, the computer-readable medium storing [the] model; and the processor configured to load the model (see paragraph [0033]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of a computer readable medium storing a model and a processor loading it as taught in Lamb with the system taught in He, the motivation being to allow different systems to utilize the same model via the medium.
Method claim 29 recites similar limitations as claim 21, and is rejected under similar rationale.
Regarding claim 36, the claim recites a non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining operations of claim 29, which He in view of Lamb further teaches (see Lamb paragraph [0159]).
Regarding claim 23, He in view of Lamb teaches all the limitations of claim 21, and further teaches wherein the backbone network comprises a convolutional neural network (see HE page 389, second column, “Firstly, the face detection model take the frames captured by near-infrared camera as input and find the faces. We chose a popular SSD-MobileNet detection model”, wherein MobileNet is well-known to be a convolutional neural network).
Regarding claim 24, He in view of Lamb teaches all the limitations of claim 21, and further teaches wherein the at least one prediction head includes a convolutional layer to receive an output of the backbone network, the convolutional layer applying a filter to the output of the backbone network to generate a convolutional output (see He Figure 4 which is an embodiment of the “landmark model” shown in Figure 2, “conv 2x1” layer. Convolutional layers are well-known to filter inputs).
Claims 31 and 37 recite similar limitations as claim 24, and are rejected under similar rationale.
Regarding claim 28, He in view of Lamb teaches all the limitations of claim 21, and further teaches all the limitations of claim 21, and further teaches wherein the machine learning model is trained using a multi-task training process (see He page 391, section A which shows that the different heads of stage 2 of Figure 2 and trained in a multi-task way).
Claim 35 recite similar limitations as claim 28, and are rejected under similar rationale.
Claim(s) 22 and 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over He et al, “A Lightweight Architecture For Driver Status Monitoring Via Convolutional Neural Networks” (published at 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 388-394, December 2019) in view of Lamb et al, U.S. Publication No. 2019/0087713 and Hamalainen, U.S. Publication No. 2013/0208134.
Regarding claim 22, He in view of Lamb teaches all the limitations of claim 21, and but does not expressively teach wherein the camera is configured to pre- process the image before transmitting the image to the processor, wherein pre- processing the image comprises one or more of performing a crop, down- sample, or grayscale conversion operation.
However, Hamalainen in a similar invention in the same field of endeavor teaches a system comprising a camera to output an image (see Hamalainen Figure 4, camera module 20 and paragraph [0056]) and a processor (see Figure 5, ISP 32 and paragraph [0070]) as taught in He in view of Lamb wherein
the camera is configured to pre- process the image before transmitting the image to the processor, wherein pre- processing the image comprises one or more of performing a crop (see Figure 4, cropping circuitry 26 and paragraph [0058]), down- sample, or grayscale conversion operation.
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of cropping an image prior to sending it to a processor as taught in Hamalainen with the system taught in He in view of Lamb, the motivation being to decrease cost in the system via decreasing the required bandwidth associated with transport, processing and storing high resolution frames (see Hamalainen paragraph [0036]).
Method claim 30 recites similar limitations as claim 22, and is rejected under similar rationale.
Claim(s) 25-27, 32-34, and 38-40 is/are rejected under 35 U.S.C. 103 as being unpatentable over He et al, “A Lightweight Architecture For Driver Status Monitoring Via Convolutional Neural Networks” (published at 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 388-394, December 2019) in view of Lamb et al, U.S. Publication No. 2019/0087713 and Mau et al, WO 2017/17134519 A1 (citations will be given to U.S. Publication No. 2020/0401851 which is the national stage entry of the reference).
Regarding claim 25, He in view of Lamb teaches all the limitations of claim 25, and further teaches wherein the at least one prediction head includes a fully connected layer, the fully connected layer receiving the convolutional output (see He Figure 4, final stage of the model and caption which indicates that the final stage is a fully connected layer. While He shows layers between the convolutional layer and fully connected layer, this is consistent with the present application as Figure 5 and claim 26 indicate there is at least one layer between the convolutional layer and fully connected layer in the Applicant’s invention).
He in view of Lamb does not expressively teach wherein the fully connected layers is configured to classify the convolutional output with the distraction classification tag.
However, Mau in a similar invention in the same field of endeavor teaches a system comprising a prediction head (see Mau Figure 3) configured to output a classification tag (see paragraph [0050]) comprising a convolutional layer configured to have a convolutional output (see Figure 3, convolutional and pooling layers 306) and a fully connected layer receiving the convolutional output (see Figure 3, fully connected layer 310 and paragraph [0050]) as taught in He in view of Lamb wherein
the fully connected layer is configured to classify the convolutional output with the classification tag (see paragraph [0050]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of a fully connected layer to formulate a classification tag as taught in Mau with the system taught in He in view of Lamb, the motivation being decentralize the system overall via allowing each prediction head to independently output a tag.
Claims 32 and 38 recite similar limitations as claim 25, and are rejected under similar rationale.
Regarding claim 26, He in view of Lamb and Mau teaches all the limitations of claim 25, and further teaches wherein the at least one prediction head includes a pooling layer configured to process the convolutional output before the convolutional output is input into the fully connected layer (see Mau Figure 3, convolution and pooling layers 306).
He in view of Lamb and Mau does not expressively teach that the pooling layer is an average pooling layer.
However, Mau goes on to teach that their pooling layer can use any statistical aggregation (see Mau paragraph [0046]). Therefore, one of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substitution to replace the pooling layers of He in view of Lamb and Mau with those claimed to yield the predictable results of successfully processing outputs of the convolutional layer.
Claims 33 and 39 recite similar limitations as claim 25, and are rejected under similar rationale.
Regarding claim 27, He in view of Lamb and Mau teaches all the limitations of claim 26, and further teaches wherein the at least one prediction head includes a sigmoid activation layer communicatively coupled to the output of the fully connected layer and configured to convert an output of the fully connected layer to a value between 0 and 1, the value between 0 and 1 comprising the distraction classification tag (see Mau Figure 3, logits layer 312 and paragraph [0051]. Soft-sigmoid functions are well known to output values between 0 and 1).
Claims 34 and 40 recite similar limitations as claim 25, and are rejected under similar rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CASEY L KRETZER whose telephone number is (571)272-5639. The examiner can normally be reached M-F 10:00-7:00 PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Payne can be reached at (571)272-3024. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CASEY L KRETZER/Primary Examiner, Art Unit 2635
Prosecution Insights