DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim status
Claims 1-14 are currently pending for examination.
Claim Objections
Claim 1 is objected to because of the following informalities: “of vehicle driver” in line 1 should be “of a vehicle driver”. Appropriate correction is required.
Claim 1 is objected to because of the following informalities: “an” in line 12 should be “and”. Appropriate correction is required.
Claim 1 is objected to because of the following informalities: “a auditory reminder” in line 13 should be “an auditory reminder”. Appropriate correction is required.
Claim 9 is objected to because of the following informalities: “head up display” should be “head-up display”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
4. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6. Claims 1-3, 5, 8-10 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over by Sicconi et al. (Sicconi; US 2019/0213429) in view of Gupta et al. (Gupta; US 2022/0383883).
For claim 1, Sicconi discloses a method for determining visual and auditory attentiveness of vehicle driver, comprising:
determining a visual attentiveness of a driver according to an image of the driver's head captured by a camera installed in a vehicle [E.g. 0056: FIG. 5 shows how the driver facing camera extracts face contours, identifying eyes, nose, mouth to evaluate yaw, pitch, roll of the face, eye gaze direction and eye lid closing patterns. A Neural Network is used to analyze the parameters and determine distraction and drowsiness conditions. The attention monitoring system 502 detects the face 503 and the hands 504 of the driver. Then it identifies facial landmarks and special regions 505 like eyes, nose, mouth to estimate head pose and eye gaze direction 506, together with information about hands holding the steering wheel. If head and/or eyes are directed away from the road signaling distraction 507. The system monitors the attention level of the driver 513 against a personalized behavior model 515 and permissible thresholds (duration, frequency, patterns) compatible 517 with driving risk computed from the driving context. If the safety margin is inadequate 519 warning alerts 500 are sent to the driver immediately. If the driver is not found to be distracted but shows signs of drowsiness 509, the system starts evaluation of driver attention 513 against user behavioral models 515 and safety margins following the same flow used for distracted driving monitoring. When the driver is not found to be distracted 507 nor drowsy 511 the system returns to observing the driver's face 503];
determining sounds generated inside the vehicle and captured by a microphone installed in the vehicle [E.g. 0072: Speech Engines (reco, synthesis) and Dialog Management 1152 analyze voice and sound picked up by microphones 1144 and generate audio and verbal prompts via speakers 1145, 0062: in-cabin noise or voices; 0014-0016];
deciding whether to issue a reminder to the driver by determining the driver's visual and attentiveness [E.g. 0014: receiving extracted features from a driver-facing camera and from a road as viewed by a road-facing camera; the computer device further receiving extracted features reflecting the driver's behavior including head and eyes movement, speech and gestures; the computer device further receiving extracted telemetry features from a vehicle; the computer device still further receiving extracted features reflecting the driver's biometrics; and a decision engine receiving information from the computer device representing each of the extracted features of the driver, wherein a driver's attention and emotional state is determined to evaluate risks associated to moving vehicles and the driver's ability to deal with any projected risks]; and
executing one or a combination of reminder steps if it is necessary to remind the driver after determining the driver's visual and attentiveness [E.g. 0054: monitors a driver 305 and the driver's behavior 307. The invention observes the face and eyes orientation using a camera 308 pointed at the driver. Direction of driver's attention is classified by rotation angles (yaw, pitch, roll and eyes lateral movements) to analyze the driver's attention 309 (road ahead, left mirror, right mirror, central rearview mirror, instrument cluster, center dash, passenger seat, phone in hand, etc.) and model driver behavior 310. Car dynamics data is evaluated 311 (acceleration, speed, RPM, engine load, . . . ) are collected from sensors (embedded in unit or in associated phone, road-facing camera with object detection and distance evaluation capabilities) and optionally from vehicle buses (OBDII, CAN), complemented with dynamic trip information 312 (e.g. traffic, weather from associated phone) to evaluate driving risk at any given time using a model driving context 313. An Al powered decision engine 315 determines the type of driver of inattention: if distracted, it alerts the driver with sounds or voice prompts 321 selected and paced based on the level of urgency. If drowsy, the system warns the driver using verbal interaction 327 and proposes attention-engaging brief dialogs stimulating mind exercises 322. Responsiveness of the user is tracked 325 to determine length and richness of the dialog with the driver 327. A microphone (optionally array of microphones) 329 and a speaker 328 (optionally a wireless speakerphone) are used to verbally communicate with the driver. If biosensors 314 to monitors heart rate and galvanic skin response are installed in the vehicle or worn by the driver, data is wirelessly transferred to a stress/fatigue monitoring device 316 or algorithm in the system, to provide additional physical model driver state information 317 which is transferred to the Decision Engine 315];
wherein the reminder steps comprising:
issuing a visual reminder by a display device in the vehicle [E.g. 0064: A Decision Engine 1020 evaluates attention 1015 vs. risk 1016 and historic short- and long-term data 1018 about driver's performance in past similar situations to decide the type of feedback to provide to the driver. Historic geo data 1019 is updated from the Cloud 1021 when vehicle connectivity permits it. If the attention level is normal 1022, limited (e.g. green light—1030) information is conveyed 1025 to avoid distracting the driver. If the attention level is marginal 1023, acoustic feedback is added to the lights 1031 to call driver's attention 1026. If attention is insufficient 1029 a pattern of audible and visual alerts 1032 are produced using an alarm driver alert 1027, escalating if the condition persists. Depending on the urgency and severity a dialog interaction 1028 is used to quickly communicate the problem detected and the identified countermeasure offered to the driver, 0125, Fig. 11: see element 1142]; an
issuing a auditory reminder by a speaker in the vehicle [E.g. 0064: A Decision Engine 1020 evaluates attention 1015 vs. risk 1016 and historic short- and long-term data 1018 about driver's performance in past similar situations to decide the type of feedback to provide to the driver. Historic geo data 1019 is updated from the Cloud 1021 when vehicle connectivity permits it. If the attention level is normal 1022, limited (e.g. green light—1030) information is conveyed 1025 to avoid distracting the driver. If the attention level is marginal 1023, acoustic feedback is added to the lights 1031 to call driver's attention 1026. If attention is insufficient 1029 a pattern of audible and visual alerts 1032 are produced using an alarm driver alert 1027, escalating if the condition persists. Depending on the urgency and severity a dialog interaction 1028 is used to quickly communicate the problem detected and the identified countermeasure offered to the driver, Fig. 11: see element 1145].
Sicconi fails to expressly disclose determining a recognition attentiveness of the driver according to sounds generated inside the vehicle and deciding whether to issue a reminder to the driver by determining recognition attentiveness and executing a reminder steps if it is necessary to remind the driver after determining the recognition attentiveness.
However, as shown by Gupta, it was well known in the art of vehicles to include determining a recognition attentiveness of a driver according to sounds generated inside the vehicle and deciding whether to issue a reminder to the driver by determining recognition attentiveness and executing a reminder steps if it is necessary to remind the driver after determining the recognition attentiveness [E.g. 0059-0061, 0065].
It would have been obvious to one of ordinary skill in the art of vehicles before the effective filling date of the claimed invention modify Sicconi with the teaching of Gupta in order to alert the driver that the audio within the vehicle is distracting so that the driver can take an action to reduce distraction cause within the vehicle and thereby enhance the overall safety on the road.
For claim 2, Sicconi discloses wherein the step of determining the visual attentiveness of the driver further comprising:
detecting a face on the image of the driver's head and marking multiple facial features on the image of the driver's head by a neural network [E.g. 0056: FIG. 5 shows how the driver facing camera extracts face contours, identifying eyes, nose, mouth to evaluate yaw, pitch, roll of the face, eye gaze direction and eye lid closing patterns. A Neural Network is used to analyze the parameters and determine distraction and drowsiness conditions. The attention monitoring system 502 detects the face 503 and the hands 504 of the driver. Then it identifies facial landmarks and special regions 505 like eyes, nose, mouth to estimate head pose and eye gaze direction 506, together with information about hands holding the steering wheel. If head and/or eyes are directed away from the road signaling distraction 507. The system monitors the attention level of the driver 513 against a personalized behavior model 515 and permissible thresholds (duration, frequency, patterns) compatible 517 with driving risk computed from the driving context. If the safety margin is inadequate 519 warning alerts 500 are sent to the driver immediately. If the driver is not found to be distracted but shows signs of drowsiness 509, the system starts evaluation of driver attention 513 against user behavioral models 515 and safety margins following the same flow used for distracted driving monitoring. When the driver is not found to be distracted 507 nor drowsy 511 the system returns to observing the driver's face 503];
estimating facial vectors of the driver according to positions of the multiple facial features [E.g. 0056-0057, 0063]; and
determining that the driver's visual attentiveness is inattentive when the facial vectors are outside a predetermined range [E.g. 0056].
For claim 3, Sicconi discloses determining whether both of the driver's eyes are open according to the multiple facial features when the facial vectors are within the predetermined range [E.g. 0056, 0063]; and
determining that the driver's visual attentiveness is inattentive when both of the driver's eyes are closed [E.g. 0056, 0063].
For claim 5, Sicconi discloses wherein the facial vectors of the driver are obtained from the relationship between positions of multiple features of a head model and the positions of the multiple facial features on the image of the driver's head, and represented by three Euler angles. [E.g. 0057]
For claim 8, Sicconi discloses wherein the image of the driver's head is captured by the camera in an infrared wavelength [E.g. 0058-0060].
For claim 9, Sicconi discloses wherein the display device includes a head up display in front of a driving seat in the vehicle [E.g. 0125].
For claim 10, Although Sicconi fails to expressly disclose wherein the speaker is located in a headrest of a driving seat in the vehicle, Sicconi teaches a speaker located inside the vehicle [0064, Fig. 11: element 1145]. However, having the speaker is located in a headrest of a driving seat in the vehicle fails to yield unexpected results; it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sicconi to include the speaker in a headrest of a driving seat in order to satisfy system needs and/or environment requirement, also because such modification would have been considered a mere design consideration which fails to patentable distinguish over Sicconi.
For claim 13, is interpreted and rejected as discussed with respect to claim 1.
For claim 14, is interpreted and rejected as discussed with respect to claim 1.
7. Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over by Sicconi in view of Gupta and further in view of Herbert et al. (Herbert; US 2023/0230396).
For claim 4, Sicconi in view of Gupta fails to expressly disclose marking positions of two pupils of the driver and estimating two pupil vectors of the driver when it is determined that both of the driver's eyes are open; and determining the driver's visual attentiveness according to the facial vectors and the two pupil vectors.
However, as shown by Gupta, it was well known in the art of drivers monitoring to include marking positions of two pupils of the driver and estimating two pupil vectors of the driver when it is determined that both of the driver's eyes are open; and determining the driver's visual attentiveness according to the facial vectors and the two pupil vectors [E.g. 0050-0051, 0060, 0074].
It would have been obvious to one of ordinary skill in the art of drivers monitoring before the effective filling date of the claimed invention modify Sicconi in view of Gupta with the teaching of Herbert in order to improve the accuracy of the driver attentiveness monitoring by using the driver pupil vectors and thereby enhance the overall safety on the road by providing a richer understanding of a driver's state.
8. Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over by Sicconi in view of Gupta further in view of Herbert and further in view of Arora et al. (Arora; US 2021/0397859).
For claim 6, Sicconi in view of Gupta and Herbert fails to expressly disclose wherein the two pupil vectors of the driver are represented by three Euler angles.
However, as shown by Arora, it was well known in the art of drivers monitoring to include two pupil vectors of the driver are represented by three Euler angles [E.g. 0018-0019, 0021, 0056].
It would have been obvious to one of ordinary skill in the art of drivers monitoring before the effective filling date of the claimed invention modify Sicconi in view of Gupta and Herbert with the teaching of Arora in order to improve the accuracy of the driver attentiveness monitoring and thereby enhance the overall safety on the road.
9. Claims 7 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over by Sicconi in view of Gupta and further in view of Official Notice.
For claim 7, Sicconi in view of Gupta fails to expressly disclose wherein the step of determining the visual attentiveness of the driver further comprising: filtering the facial vectors through one or any combination of time-series filter, Kalman filter, and low-pass filter.
However, examiner takes official notice that determining the visual attentiveness of the driver further comprising: filtering the facial vectors through one or any combination of time-series filter, Kalman filter, and low-pass filter is well-known in the art of filtering and would have been obvious to one of ordinary skill in the art in order to improve accuracy and noise reduction.
For claim 11, Sicconi in view of Gupta fails to expressly disclose executing one or a combination of the following steps when it is not necessary to issue the reminder to the driver, wherein the following steps comprising: stopping the display device in the vehicle from issuing the visual reminder; and stopping the speaker in the vehicle from issuing the auditory reminder.
However, examiner takes official notice that executing one or a combination of the following steps when it is not necessary to issue the reminder to the driver, wherein the following steps comprising: stopping the display device in the vehicle from issuing the visual reminder; and stopping the speaker in the vehicle from issuing the auditory reminder is well-known in the art of reminding a driver and would have been obvious to one of ordinary skill in the art in order to reduce the chance that the reminders overwhelm the driver.
10. Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over by Sicconi in view of Gupta and further Gross (US 2018/0108369).
For claim 12, Sicconi in view of Gupta fails to expressly disclose wherein the sounds inside the vehicle is represented by a spectrogram, the step of determining the recognition attentiveness of the driver further comprising: analyzing the spectrogram by a neural network to determine whether the sounds inside the vehicle contain ambient noises and prominent noise that distracts the driver.
However, as shown by Gupta, it was well known in the art of vehicles noise monitoring to include wherein sounds inside the vehicle is represented by a spectrogram, the step of determining the recognition attentiveness of the driver further comprising: analyzing the spectrogram by a neural network to determine whether the sounds inside the vehicle contain ambient noises and prominent noise that distracts the driver [E.g. 0031-0032, 0036, 0043, 0013].
It would have been obvious to one of ordinary skill in the art of vehicles noise monitoring before the effective filling date of the claimed invention modify Sicconi in view of Gupta with the teaching of Herbert in order to effectively detect ambient sound that can distract the driver so that the driver is alerted to take an appropriate action and thereby enhance the overall safety on the road.
Conclusion
11. The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure: see PTO-892 Notice of Reference Cited.
12. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED BARAKAT whose telephone number is (571)270-3696. The examiner can normally be reached on 9:00am-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Davetta Goins can be reached on (571) 272-2957. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED BARAKAT/
Primary Examiner, Art Unit 2689