Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner note
The prior art of record does not teach or make obvious claim 9, specifically spreading the different tasks over separate threads. Therefore, claims 9-19 don’t have an art rejection.
Drawings
The drawings are objected to because figures 5 and 9 are illegible. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 2 recites the limitation "the knowledge distillation technique". There is insufficient antecedent basis for this limitation in the claim.
Claims 9-19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 recites, “a second thread configured to do inference a face and a hand of a driver, a phone, and a sunglass from the image frames preprocessed by the first thread to get a plurality of bounding boxes corresponding to the face, the hand, the phone, and the sunglass…” The phrase “to do inference a face” causes confusion.
Claim 10 recites “wherein each of the first to sixth threads is simultaneously processed in communication with each other.” This is not possible to do simultaneously when the input of threads 4-6 are dependent on previous threads.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 are rejected under 35 U.S.C. 103 as being unpatentable over US20210403004A1 to Alvarez et al, US20220261599A1 to Kastaniotis et al (Kast) and US20050262510A1 to Parameswaran et al (Param).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over US20210403004A1 to Alvarez et al, US20220261599A1 to Kastaniotis et al (Kast), US20050262510A1 to Parameswaran et al (Param) and US20080147267A1 to Plante et al.
Claims 4-8 are rejected under 35 U.S.C. 103 as being unpatentable over US20210403004A1 to Alvarez et al, US20220261599A1 to Kastaniotis et al (Kast), US20050262510A1 to Parameswaran et al (Param), US20080147267A1 to Plante et al and US20170083829A1 to Kang et al.
Alvarez teaches claim 1. A driver monitor system comprising: (Alvarez title “Driver monitoring system…”)
an image data acquiring module configured to acquire a plurality of image data from a data collection module; (Alvarez para 153 “machine learning trained models may be implemented for various in-vehicle monitoring functions, which may be trained using image, video, and/or audio data, among other types of data that form what is referred to as a training dataset.”)
a training module configured to train a plurality of teacher models to obtain a plurality of feature groups using the plurality of image data, (Alvarez para 153 “(Alvarez para 153 “machine learning trained models may be implemented for various in-vehicle monitoring functions …”)
at least one edge device comprising the plurality of student models configured (Alvarez para 2 “Based upon the particular implementation, a DMS may issue a warning when an unsafe condition is detected, which is intended to alert the driver and to ensure the driver's focus remains on the road.”) Alvarez doesn’t have a knowledge transfer.
However, Kast teaches transfer a plurality of pieces of knowledge obtained from the plurality of feature groups to a plurality of student models, respectively; and (Kast para 13 “The Teacher network is then able to annotate new data, performing in this way an automatic annotation process that is useful if data from other sources are available or data from the field are getting collected. The Teacher network is also used to train the final CNN model that will be released in production (at the edge devices) via knowledge transfer using a distillation approach in a Teacher-Student scheme.”
the plurality of student models configured to use a pipeline design pattern (Kast para 26 “FIG. 10 is a flowchart showing the object localization process pipeline of the LSTN approach.”)
Kast, Alvarez and the claims are all machine learning algorithms. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use Kast’s transfer learning to “take advantage of prior knowledge provided from a “teacher” … model…” Kast para 5.
Kast and Alvarez don’t teach multithreads.
However, Param teaches pipeline design pattern with multiple threads. (Param abs “provides a multi-threaded processing pipeline that is applicable in a System-on-Chip (SoC) using a DSP and shared resources such as DMA controller and on-chip memory…”)
Param, Kast, Alvarez and the claims all use semiconductor architecture to implement processes. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use Param’s multithreading “for increasing the throughput.” Param abs.
Kast teaches claim 2. The system of claim 1, wherein the plurality of student models receive the transferred knowledge using the knowledge distillation technique. (Kast para 13 “knowledge transfer using a distillation approach in a Teacher-Student scheme.”)
Kast teaches claim 3. The system of claim 2, wherein the plurality of student models perform inference based on the transferred knowledge to get a vision-based information including information on confidence of landmarker points, (Kast para 13 “knowledge transfer using a distillation approach in a Teacher-Student scheme.” Kast fig. 1 shows that this is for a car. Kast abs “processing loop with new annotated objects detected from images captured at the field.” Kast para 50 “The images presenting the higher confidence score (above a specific fixed threshold) are qualified and stored in the initial object database…” The object is the landmarker point.) (Alvarez para 116 “detecting if a driver is drowsy while controlling the vehicle by analyzing eyes and facial expressions…”) and information of a mouth state, (Alvarez para 171 “This overlaying process may therefore include mapping points in the 3D mesh that are identified via the collected sensor data to matching points in the reference 3D model (such as eyes, nose, mouth, hair, etc.) to further alter the 3D and texture parameters of the 3D mesh to create a refined user-specific 3D model…”) and make a warning based on the vision-based information and a car-based information. (Alvarez para 2 “a DMS may issue a warning when an unsafe condition is detected, which is intended to alert the driver and to ensure the driver's focus remains on the road…”) Alvarez and Kast don’t teach a usage of sunglasses and a phone.
However, Plante teaches considering usage of a sunglass and a phone. (Plante para 77 “Checkboxes may be used indicate binary conditions such as whether or not a driver is using a cell phone, is smoking, is alert, wearing sunglasses, made error, is using a seat belt properly, is distracted, for example. It is easily appreciated that these are merely illustrative examples, one would certainly devise many alternative and equally interesting characterizations associated with a driver and driver performance in fully qualified systems.”)
Alvarez, Kast, Plante and the claims are all driving metrics algorithms. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to include data like usage of sunglasses and phones in order to track “driver performance in fully qualified systems.” Plante para 77.
Alvarez teaches claim 4. The system of claim 3, wherein the plurality of image data includes at least one of a facial image data, a cropped face image data, a cropped eye image data, a hand image data, a phone image data, and a sunglass image data, and (Alvarez para 116 “detecting if a driver is drowsy while controlling the vehicle by analyzing eyes and facial expressions…”) Alvarez doesn’t teach a third teacher model.
However, Kang teaches wherein the plurality of teacher models include a first teacher model, a second teacher model and a third teacher model. (Kang para 53 “The plurality of teacher models may include first teacher model through N-th teacher model (e.g., first teacher model 205, second teacher model 210, third teacher model 215, N-th teacher model 225).”)
Kang, Alvarez and the claims are all machine learning models. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to minimize “reduction in a recognition rate while miniaturizing an artificial neural network.” Kang para 6.
Alvarez teaches claim 5. The system of claim 4, wherein the plurality of feature groups include a first feature group, a second feature group and a third feature group, and (Alvarez para 164 “This may include the use of cameras, LIDAR, gyroscope data, accelerometer data, etc., to obtain a 3D mesh of the user…”)
wherein the first teacher model is trained based on the facial image data, (Alvarez abs “use of personalized training data to supplement machine learning trained models for Driver Monitoring System (DMS)…” Alvarez para 116 “detecting if a driver is drowsy while controlling the vehicle by analyzing eyes and facial expressions…”) (Alvarez para 116 “detecting if a driver is drowsy while controlling the vehicle by analyzing eyes and facial expressions…”) Alvarez doesn’t teach training on the other data.
However, Plante teaches hand image data, the phone image data and the sunglass image data to acquire the first feature group including a face detection feature, a hand detection feature, a phone detection feature, and a sunglass detection feature. (Plante para 77 “Checkboxes may be used indicate binary conditions such as whether or not a driver is using a cell phone, is smoking, is alert, wearing sunglasses, made error, is using a seat belt properly, is distracted, for example. It is easily appreciated that these are merely illustrative examples, one would certainly devise many alternative and equally interesting characterizations associated with a driver and driver performance in fully qualified systems.” The checkbox is the feature, the data is the checkbox value. Smoking is hand image data. Plante para 6 “important image information may be captured by video cameras installed on the police cruiser.”)
Alvarez teaches claim 6. The system of claim 5, wherein the including a plurality of facial landmarks. (Alvarez para 116 “detecting if a driver is drowsy while controlling the vehicle by analyzing eyes and facial expressions…” The facial expression is the feature, and the cropped face image data including facial landmarks.) Alvarez doesn’t have a second teacher model.
However, Kang teaches a second teacher model. (Kang para 53 “The plurality of teacher models may include first teacher model through N-th teacher model (e.g., first teacher model 205, second teacher model 210, third teacher model 215, N-th teacher model 225).”)
Alvarez teaches claim 7. The system of claim 6, wherein the (Alvarez para 116 “detecting if a driver is drowsy while controlling the vehicle by analyzing eyes and facial expressions…” Eye data is the cropped eeye image data, the eyes are the eye-staate detection and gaze feature) Alvarez doesn’t have a third teacher model.
However, Kang teaches a third teacher model. (Kang para 53 “The plurality of teacher models may include first teacher model through N-th teacher model (e.g., first teacher model 205, second teacher model 210, third teacher model 215, N-th teacher model 225).”)
Kast teaches claim 8. The system of claim 7, wherein the plurality of pieces of knowledge include a first knowledge, a second knowledge and a third knowledge,
wherein the plurality of student models include:
a first student model trained based on the first knowledge transferred from the first teacher model;
a second student model trained based on the second knowledge transferred from the
a third student model trained based on the third knowledge transferred from the
wherein the knowledge transferring to the first, second, and third student models are executed using the knowledge distillation technique. (Kast para 82 “Knowledge distillation training can be applied on object detection tasks by forcing the student networks to imitate the teacher's network response in the regions where the objects appear.” The disclosed “student networks” teaches almost any amount of claimed student models.) Kast doesn’t have multiple teachers.
However, Kang teaches a first, second and third teacher model. (Kang para 53 “The plurality of teacher models may include first teacher model through N-th teacher model (e.g., first teacher model 205, second teacher model 210, third teacher model 215, N-th teacher model 225).”)
Notices of Reference cited not relied upon
US 10,417,486 to Ren et al teaches “A driver behavior monitoring system includes: a video camera; an image processing system; and a driver alert system.”
US8954340B2 to Sanchez teaches “The server then assigns an indication of risk to at least some of the movement categories and combines the motion sensing data from a plurality of movement categories to generate a collective measure of risk associated with the driver of the vehicle.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2124