DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Currently, claims 1-23 are pending in the application. Claims , 3-6, 9, 11-16, and 18-19 are amended. Claims 21-23 are added.
Response to Arguments / Amendments
Applicant’s arguments have been fully considered but are rendered moot in view of the new ground of rejection necessitated by amendments initiated by the applicant.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5-10, 13-17 & 19-23 are rejected under 35 U.S.C. 103 as being unpatentable over Rougeaux et al. (US 20190156100, hereinafter Rougeaux in view of Arora et al. (US 20230088021, hereinafter Arora) and Heck et al. ( US 20230166743, hereinafter Heck)
Regarding Claim 1, Rougeaux discloses a method for determining driver surprise, comprising:
receiving image data of a driver of the vehicle's pupils and video data of a face of the driver over a time interval ([0053], [0093], FIG. 1, an imaging device flight camera 104 captures time separated images of the subject's face 106 including one or both of the subject's eyes 108 and 110 in gaze tracking system of driver monitoring system for a vehicle; [0016]);
determining a diameter of the driver's pupils over the time interval based on the image data ([0094], determine pupil/iris boundary when detecting three dimensional position of corneal reflections of the light source in conjunction with detecting the two dimensional position of one or more reference eye features such as a pupil center, iris center, pupil/iris boundary, iris/sclera boundary, eyelids or eye corners; [0100] FIG. 7) and generating a pupil confidence value based on the diameter over the time interval ([0098], Circle recognition through a 2D Hough transform and radius histogramming; [0099], FIG. 6, Step 604, the two dimensional positions of the corneal reflections and the pupil position are fitted to a three dimensional cornea model having a known cornea center and pupil position to determine three dimensional positions of the corneal reflections and pupil center);
extracting one or more facial features from the video data ([0109], FIG. 8, Step 802, the image is processed to identify reference facial features such as eye corners, nostrils, mouth corners, ears or any other small recognizable areas on the face, and extract the three dimensional positions of those features) and generating a facial confidence value based on the one or more facial features ([0090] Several examples of shape regression are known, such as Supervised Descent Method (SDM) or Convolutional Neural Network (CNN) Or perform similar shape regression algorithm to identify other facial features to estimate a confidence that the region around the facial landmarks has the appearance of a face; [0110], FIG. 8, Step 803, the three dimensional positions of the facial features are fitted to a three dimensional model which is formed of a mesh structure that is deformable to account for the different profiles of individuals).
Rougeaux does not explicitly disclose identifying an upcoming hazardous driving event for a vehicle; applying a first weight to the pupil confidence value and applying a second weight to the facial confidence value to determine the driver's emotional response caused by the hazardous driving event; and determining whether the driver is surprised by the hazardous driving event based on the weighted pupil confidence value and the weighted facial confidence value and determining a time between the driver's emotional response and the vehicle's arrival at the upcoming hazardous driving event.
Arora teaches applying a first weight to the pupil confidence value and applying a second weight to the facial confidence value ([0021], FIGS. 6A-6E, apply a trained machine learning model to the image to determine the facial plane using images of other people having the facial landmarks already identified and the facial plane angle already determined and rank or adjust the parameters of the machine learning model based on the first initial 3D eye gaze vector determined using eye tracking as compared to the second 3D eye gaze vector determined using the facial plane; [0022], vehicle computing system 104 may determine 3D eye gaze vector 112 using an average of the first and second initial 3D eye gaze vectors. Vehicle computing system 104 may, in other examples, apply a weighting to one or more of the first and second 3D eye gaze vectors and use the weighted values to determine 3D eye gaze vector 112. Vehicle computing system 104 may determine weights to apply to the first and second initial 3D eye gaze vectors based on a confidence that the first or second initial 3D eye gaze vector was accurately determined );
Arora further teaches weighted pupil confidence value and the weighted facial confidence value ([0040], vehicle computing system 104 determines that the driver is likely providing the user inputs. If the driver provides more than a threshold number of user inputs within a predetermined period of time or continues to interact with the infotainment system for greater than a predetermined period of time, vehicle computing system 104 may take various actions to encourage the driver to resume paying attention to the road such as output a message reminding the driver to pay attention to the road; [0060]).
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of applying weight to the pupil and to the facial confidence value as taught by Arora ([0021]) into the imaging system of Rougeaux in order to provide systems for determining that a driver's eyes are open, analyzing facial expressions, or blink speed enables the computing system to more precisely determine a location within the vehicle the user is looking and determines what is physically located at that the location within the vehicle (Arora, [0004]).
Rougeaux and Arora do not explicitly disclose identifying an upcoming hazardous driving event for a vehicle; determine the driver's emotional response caused by the hazardous driving event; and determining whether the driver is surprised by the hazardous driving event based on the weighted pupil confidence value and the weighted facial confidence value and determining a time between the driver's emotional response and the vehicle's arrival at the upcoming hazardous driving event.
Heck teaches applying identifying an upcoming hazardous driving event for a vehicle ([0254], to detect one or more objects in the environment outside the vehicle based on one or more images provided by the first camera 202 such as object(s) being detected may be a vehicle (e.g., car, motorcycle, etc.), a lane boundary, human, bicycle, an animal, a road sign (e.g., stop sign, street sign, no turn sign, etc.), a traffic light, a road marking (e.g., stop line, lane divider, text painted on road, etc.), etc., or vehicle being detected may be a lead vehicle, which is a vehicle in front of the subject vehicle that is traveling in the same lane as the subject vehicle); determine the driver's emotional response caused by the hazardous driving event ([0261], determine whether to generate a control signal based on output from the collision predictor 218, and output from the driver monitoring module. Alternatively or additionally, the signal generation controller 224 is configured to determine whether to generate a control signal based on output from the intersection violation predictor 222, and optionally also based on output from the driver monitoring module); and
PNG
media_image1.png
736
510
media_image1.png
Greyscale
determining whether the driver is surprised by the hazardous driving event based on the weighted pupil confidence value and the weighted facial confidence value ([0315] determine that the driver is drowsy with a drowsiness metric that indicates the proportion of time in a minute that the eyes are at least 80 percent closed. PERCLOS is the percentage of eyelid closure over the pupil over time and reflects slow eyelid closures rather than blinks)and determining a time between the driver's emotional response and the vehicle's arrival at the upcoming hazardous driving event ([0387], FIG. 19C also shows how much the vehicle would have travelled based on a driver's reaction time of 1.5 seconds. For example, for a vehicle traveling at 40 km/hr, it would travel 17 meters in about 1.5 seconds (driver's reaction time) before the driver applies the brake. Thus, the total distance it would take for a vehicle traveling at 40 km/hr to stop (and considering reaction time of the driver) will be 26 meters in dry road condition and 30 meters in wet road condition.).
PNG
media_image2.png
426
570
media_image2.png
Greyscale
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of determine the driver's emotional response to a driving event as taught by Heck ([0254]) into the imaging system of Rougeaux & Arora in order to provide systems for issuing warning for a driver and automatically operates the vehicle in response to predicted risk, provides output indicating better and bad driving for training drivers and fleet management in an effective manner (Heck, [0002]).
Regarding Claim 2, Rougeaux in view of Arora and Heck discloses the method of claim 1,
Rougeaux discloses wherein the video data comprises video collected from a video camera centered on the driver's face ([0093], FIG. 1, an imaging device flight camera 104 captures time separated images of the subject's face 106 including one or both of the subject's eyes 108 and 110 in gaze tracking system of driver monitoring system for a vehicle).
Regarding Claim 7, Rougeaux in view of Arora and Heck discloses the method of claim 1,
Rougeaux discloses further comprising determining a classification performance value indicating an accuracy of determining whether the driver is expressing surprise ([0014] upon detection of no corneal reflections, determining three dimensional positions of one or more reference facial features of the subject and performing a second eye gaze tracking procedure on one or both eyes of the subject based on the estimation of head pose of the subject from the positions of the one or more detected reference facial features).
Regarding Claim 8, Rougeaux in view of Arora and Heck discloses the method of claim 1,
Arora discloses wherein determining whether the driver is expressing surprise comprises calculating a weighted average of the weighted pupil confidence value and the weighted facial confidence value ([0022], vehicle computing system 104 may determine 3D eye gaze vector 112 using an average of the first and second initial 3D eye gaze vectors. Vehicle computing system 104 may, in other examples, apply a weighting to one or more of the first and second 3D eye gaze vectors and use the weighted values to determine 3D eye gaze vector 112. Vehicle computing system 104 may determine weights to apply to the first and second initial 3D eye gaze vectors based on a confidence that the first or second initial 3D eye gaze vector was accurately determined). The same reason or rational of obviousness motivation applied as used above in claim 1.
Regarding Claims 9-10 & 13-15, system claims 9-10 & 13-15 of using the corresponding method claimed in claims 1-2 & 5-8, and the rejections of which are incorporated herein for the same reasons as used above.
Regarding Claims 16-17 & 19-20, Computer machine-readable medium claims 16-17 & 19-20 of using the corresponding method claimed in claims 1-2 & 5-8, and the rejections of which are incorporated herein for the same reasons as used above.
Regarding Claim 21, Rougeaux in view of Arora and Heck discloses the method of claim 1,
Heck discloses further comprising determining a time between the driver's emotional response and the vehicle's arrival at the upcoming hazardous event ([0387], FIG. 19C also shows how much the vehicle would have travelled based on a driver's reaction time).
The same reason or rational of obviousness motivation applied as used above in claim 1.
Regarding Claim 22-23 Analogous rejection as the rejection of Claim 21 applies.
Claims 3-6, 11-12 & 18 are rejected under 35 U.S.C. 103 as being unpatentable over Rougeaux et al. (US 20190156100, hereinafter Rougeaux in view of Arora et al. (US 20230088021, hereinafter Arora), Heck et al. ( US 20220402517, hereinafter Heck) and Raz et al. (US 20230294706, hereinafter Raz)
Regarding Claim 3, Rougeaux in view of Arora and Heck discloses the method of claim 1, but does not explicitly disclose further comprising determining a latency between the driver expressing surprise and the hazardous driving event occurring in front of the vehicle , wherein the first and second weights are determined based on the latency.
Raz teaches determining a latency between the driver expressing surprise and the hazardous driving event occurring in front of the vehicle , wherein the first and second weights are determined based on the latency ([0020], FIGS. 1-4, determine the alertness or attentiveness of a driver in combination with an exterior monitoring device 13 (see FIG. 3) that detects event or object in the local environment of the vehicle. The interior monitoring device 12 may detect the response of the driver 11 to the detected event or object based on the reaction delay, timing, or response of the driver 11, the system 10 may determine an alertness level or state of the driver 11 based on an aggregated reaction time (e.g., an average reaction time, a median reaction time, etc.) for multiple objects or events presented during a driving trip. In addition to the response of the driver 11, the attentiveness may be determined based on a number of factors including the nature or relative significance of the events or objects detected in the local environment to improve the accuracy of the assessment of the driver 11; [0031] FIGS. 2A and 2B, determine that the point of interest likely corresponds to the sole event detected in the exterior image data 29c. Further, the controller 28 may track movement of the eyes 20 over multiple frames to determine that the eyes 20 are focused on or are directed to an object moving relative to the vehicle (e.g., due to the vehicle approaching the event)).
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of latency between the driver expressing surprise and an event occurring in front of a vehicle as taught by Raz ([0020]) into the imaging system of Rougeaux & Arora in order to provide systems for improving reaction time of the driver by, communicating instructions for modifying a target distance from another vehicle for an adaptive cruise control function, increasing sensitivity of a blind zone detection function, and modifying heat distribution or an illumination of a panel of a console (Raz, [0004]).
Regarding Claim 4, Rougeaux in view of Arora, Heck and Raz discloses the method of claim 3,
Raz discloses wherein the latency is based on an action threshold associated with vehicle data of the hazardous driving event and a detection threshold associated with the image data ([0020], detect the response of the driver 11 to the detected event or object based on the reaction delay, timing, or response of the driver 11, the system 10 may determine an alertness level or state of the driver 11 based on an aggregated reaction time (e.g., an average reaction time, a median reaction time, etc.) for multiple objects or events presented during a driving trip). The same reason or rational of obviousness motivation applied as used above in claim 3.
Regarding Claim 5, Rougeaux in view of Arora, Heck and Raz discloses the method of claim 3,
Arora discloses further comprising receiving video of the front of the vehicle and determining the event based on the video ([0002] Vehicles with semi-autonomous driving features that includes features that help keep the vehicle within lane boundaries, guide the vehicle around corners, or automatically accelerate and brake based on the presence of other vehicles).
Regarding Claim 6, Rougeaux in view of Arora, Heck and Raz discloses the method of claim 5,
Rougeaux discloses wherein the hazardous driving event comprises a safety hazard ([0002], vehicle driver drowsiness and attention monitoring systems). The same reason or rational of obviousness motivation applied as used above in claim 1.
Regarding Claims 11-12, system claims 11-12 of using the corresponding method claimed in claims 3-4, and the rejections of which are incorporated herein for the same reasons as used above.
Regarding Claim 18, Computer machine-readable medium claim 18 of using the corresponding method claimed in claims 3-4, and the rejections of which are incorporated herein for the same reasons as used above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samuel D Fereja whose telephone number is (469)295-9243. The examiner can normally be reached 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DAVID CZEKAJ can be reached at (571) 272-7327. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAMUEL D FEREJA/Primary Examiner, Art Unit 2487
