DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
Claims 2-5, 7-8, 11-14 and 16-21 are rejected under 35 U.S.C. 103 as being unpatentable over USPN 2022/0215200 to Alpert et al.
With regard to claim 2, Alpert discloses a computer-implemented method, comprising:
generating a first prediction corresponding to a drowsiness of an operator of a machine based at least on one or more environmental conditions associated with operation of the machine by the operator (paragraphs [0042], [0048], [0069] and [0079], Environment context is gathered related to the area outside the vehicle, as well as driver conditions within the vehicle such as the driver’s pose and/or head position) and one or more variations in eye activity corresponding to the operator over a period of time (paragraphs [0051] and [0056]-[0061], Alpert discloses determining eye closure values over a period of time to determine a blink rate and/or PERCLOS metric as a measure of a certain amount of time. See Figs. 5 and 6);
generating a second prediction corresponding to the drowsiness of the operator based on a frequency of eye activity corresponding to the operator over the period of time (paragraphs [0012] and [0058]-[0059], [0069] a blink rate over time is determined and used to decide if a threshold is exceeded indicating that the driver is drowsy);
determining an overall drowsiness prediction based, at least, on the first prediction and the second prediction (paragraphs [0056]-[0058], [0069], [0074] and [0083]-[0090], Alpert discloses that the drowsiness determination is based on both the first and second neural network models. The overall drowsiness determination is accordingly interpreted to be based on the first and second neural network models);
determining the overall drowsiness prediction exceeds a threshold (paragraphs [0053]-[0062], Thresholds are used in order to determine different elements of the drowsiness determination. See also paragraph [0084] and Fig. 7, step 810, a determination is made if the driver is drowsy or not implying a threshold is used to make the determination); and
activating a driver assistance protocol for the vehicle (paragraphs [0042], [0048], when a determination is made the driver is drowsy, assistance can be provided in the form of alerting the driver with an alarm or voice message).
With regard to claim 3, Alpert discloses the computer-implemented method of claim 2, wherein the one or more environmental conditions relate to at least one of a road configuration, brightness, weather, time of day, location, speed, or number of surrounding objects (paragraphs [0042], [0076] and [0079]-[0080], Conditions monitored for the vehicle include the speed of the vehicle and driving operating conditions such as whether the vehicle is turning as well as GPS or location information and data from images of the area or environment surrounding the vehicle).
With regard to claim 4, Alpert discloses the computer-implemented method of claim 2, further comprising: determining the one or more environmental conditions using data from one or more cameras, sensors, global positioning system (GPS) signals, or network data sources (paragraphs [0039], [0042], [0046], [0076] and [0079]-[0080], Cameras capture the environment outside the vehicle. Sensors obtain information regarding speed and direction/turning of the vehicle. GPS location data is also captured. The system also communicates wirelessly with a network).
With regard to claim 5, Alpert discloses the computer-implemented method of claim 2, further comprising:
determining a blink scenario based at least in part upon the one or more environmental conditions (paragraph [0069]: “In some embodiments, the first model 212 may be a first neural network model trained to classify pose(s) based on context. For example, if the driver's head is facing down, then the neural network model may determine that the driver is not looking straight even if the eyes of the driver cannot be detected (e.g., because the eyes may be blocked by a hat/cap). Also, in some embodiments, the second model 214 may be a second neural network model trained to determine whether the driver is drowsy or not based on context. For example, if the blink rate exceeds a certain threshold, and/or if the head-down pose and head-straight pose repeats in a period pattern, then the neural network model may determine that the driver is drowsy.” The blink scenario includes eye closure information over a period of time as well as if the driver’s head pose meets certain criteria which are interpreted broadly as environmental conditions); and
determining the first drowsiness prediction using one or more blink thresholds corresponding to the blink scenario (paragraph [0069], The blink rate exceeding a threshold indicates possible drowsiness).
With regard to claim 7, Alpert discloses the computer-implemented method of claim 2, further comprising:
identifying a set of facial landmarks in image data including a representation of a face of the operator (paragraph [0045], Features are extracted such as the driver’s head, face, nose, eye, etc.);
determining, from the image data, eye state information indicating whether the eyes of the operator are fully or partially open or closed (paragraphs [0051] and [0058]-[0060], Determination is made if the driver’s eyes are open or closed);
determining, from the image data, head pose information for the operator (paragraphs [0051]-[0055], Pose classifications are determined for the driver);
determining, based at least in part upon the set of facial landmarks, the head pose information, and the eye state information, a set of blink parameters (paragraphs [0056]-[0061], Alpert discloses determining pose classification based on determined head feature extraction and further determines eye closure values over a period of time to determine a blink rate and/or PERCLOS metric as a measure of a certain amount of time. See also Figs. 5 and 6).
With regard to claim 8, Alpert discloses the computer-implemented method of claim 7, wherein the frequency of eye activity is a blink frequency, further comprising:
determining, from the eye state information, the blink frequency for the period of time (paragraphs [0056]-[0061], Alpert discloses determining pose classification based on determined head feature extraction and further determines eye closure values over a period of time to determine a blink rate and/or PERCLOS metric as a measure of a certain amount of time. See also Figs. 5 and 6).
With regard to claim 11, Alpert discloses a computer-implemented method, comprising:
determining a set of blink parameters for a person from image data including a representation of a face of the person over a period of time (paragraph [0045], Features are extracted such as the driver’s head, face, nose, eye, etc. See also paragraphs [0056]-[0061], Alpert discloses determining pose classification based on determined head feature extraction and further determines eye closure values over a period of time to determine a blink rate and/or PERCLOS metric as a measure of a certain amount of time. See also Figs. 5 and 6);
determining an action context, based on context data associated with one or more environmental conditions and to one or more physical characteristics of a vehicle operated by the person at a time when the image data was generated (paragraphs [0042], [0048], [0069], [0076] and [0079]-[0080], Conditions monitored for the vehicle include the speed of the vehicle and driving operating conditions such as whether the vehicle is turning as well as GPS or location information and data from images of the area or environment surrounding the vehicle. Environment context is gathered related to the area outside the vehicle, as well as driver conditions within the vehicle such as the driver’s pose and/or head position);
passing the set of blink parameters and the action context to at least a first neural network to generate at least a first drowsiness prediction for the person relative to a baseline selected for the action conditions (paragraph [0069]: “In some embodiments, the first model 212 may be a first neural network model trained to classify pose(s) based on context. For example, if the driver's head is facing down, then the neural network model may determine that the driver is not looking straight even if the eyes of the driver cannot be detected (e.g., because the eyes may be blocked by a hat/cap). Also, in some embodiments, the second model 214 may be a second neural network model trained to determine whether the driver is drowsy or not based on context. For example, if the blink rate exceeds a certain threshold, and/or if the head-down pose and head-straight pose repeats in a period pattern, then the neural network model may determine that the driver is drowsy.” The blink scenario includes eye closure information over a period of time as well as if the driver’s head pose meets certain criteria which are interpreted broadly as environmental conditions. The baseline for the action context refers to what is considered normal, i.e. head poses or driving conditions within a baseline range compared to conditions such as head pose or driving conditions that exceed a threshold);
passing a blink frequency determination to at least a second neural network to generate at least a second drowsiness prediction (Alpert further discloses a second neural network model that uses information from the first neural network model to make an additional drowsiness determination (paragraphs [0056]-[0058], [0069], [0074] and [0083]-[0090]). Alpert further discloses that the second model uses the blink rate or blink frequency to determine drowsiness among other eye closure variables (paragraph [0091]); and
determining an overall drowsiness prediction based, at least, on the first drowsiness prediction and the second drowsiness prediction (paragraphs [0056]-[0058], [0069], [0074] and [0083]-[0090], Alpert repeatedly discloses that the drowsiness determination is based on both the first and second neural network models. The overall drowsiness determination is accordingly interpreted to be based on the first and second neural network models).
With regard to claim 12, Alpert discloses the computer-implemented method of claim 11, wherein the one or more environmental conditions relate to at least one of a road configuration, brightness, weather, time of day, location, speed, or number of surrounding objects (paragraphs [0042], [0076] and [0079]-[0080], Environment context is also gathered from a camera of the area surrounding the vehicle including lanes and surrounding objects as well as sensor data such as accelerometer signals, speed signals, location signals, etc. Conditions monitored for the vehicle include the speed of the vehicle and driving operating conditions such as whether the vehicle is turning as well as GPS or location information and data from images of the area or environment surrounding the vehicle).
With regard to claim 13, Alpert discloses the computer-implemented method of claim 11, further comprising: determining the one or more environmental conditions using data from one or more cameras, sensors, global positioning system (GPS) signals, or network data sources (paragraphs [0039], [0042], [0046], [0076] and [0079]-[0080], Cameras capture the environment outside the vehicle. Sensors obtain information regarding speed and direction/turning of the vehicle. GPS location data is also captured. The system also communicates wirelessly with a network).
With regard to claim 14, Alpert discloses the computer-implemented method of claim 11, further comprising:
determining a blink scenario based at least in part upon the one or more environmental conditions (paragraph [0069]: “In some embodiments, the first model 212 may be a first neural network model trained to classify pose(s) based on context. For example, if the driver's head is facing down, then the neural network model may determine that the driver is not looking straight even if the eyes of the driver cannot be detected (e.g., because the eyes may be blocked by a hat/cap). Also, in some embodiments, the second model 214 may be a second neural network model trained to determine whether the driver is drowsy or not based on context. For example, if the blink rate exceeds a certain threshold, and/or if the head-down pose and head-straight pose repeats in a period pattern, then the neural network model may determine that the driver is drowsy.” The blink scenario includes eye closure information over a period of time as well as if the driver’s head pose meets certain criteria which are interpreted broadly as environmental conditions); and
determining the first drowsiness prediction using one or more blink thresholds corresponding to the blink scenario (paragraph [0069], The blink rate exceeding a threshold indicates possible drowsiness).
With regard to claim 16, Alpert discloses the computer-implemented method of claim 11, further comprising:
identifying a set of facial landmarks in the image data (paragraph [0045], Features are extracted such as the driver’s head, face, nose, eye, etc.);
determining, from the image data, eye state information indicating whether the eyes of the person are fully or partially open or closed (paragraphs [0051] and [0058]-[0060], Determination is made if the driver’s eyes are open or closed);
determining, from the image data, head pose information for the person (paragraphs [0051]-[0055], Pose classifications are determined for the driver);
determining, based at least in part upon the set of facial landmarks, the head pose information, and the eye state information, the set of blink parameters (paragraphs [0056]-[0061], Alpert discloses determining pose classification based on determined head feature extraction and further determines eye closure values over a period of time to determine a blink rate and/or PERCLOS metric as a measure of a certain amount of time. See also Figs. 5 and 6); and
determining, from the eye state information, the blink frequency for the recent period of time (paragraphs [0056]-[0061], Alpert discloses determining pose classification based on determined head feature extraction and further determines eye closure values over a period of time to determine a blink rate and/or PERCLOS metric as a measure of a certain amount of time. See also Figs. 5 and 6).
With regard to claim 17, Alpert discloses the computer-implemented method of claim 11, further comprising:
determining the overall drowsiness prediction exceeds a threshold (paragraphs [0053]-[0062], Thresholds are used in order to determine different elements of the drowsiness determination. See also paragraph [0084] and Fig. 7, step 810, a determination is made if the driver is drowsy or not implying a threshold is used to make the determination); and
activating a driver assistance protocol for the vehicle (paragraphs [0042], [0048], when a determination is made the driver is drowsy, assistance can be provided in the form of alerting the driver with an alarm or voice message).
With regard to claim 18, Alpert discloses a system, comprising:
a camera to capture image data including a representation of a face of a person over a period of time (paragraph [0074], cameras capture videos of face images);
one or more processors (paragraph [0043], processing unit 210 in Fig. 2); and
memory including instructions that, when executed by the one or more processors (paragraph [0045], program instructions perform the processing steps disclosed), cause the system to:
determine, from at least a portion of the image data, a set of blink parameters for the person (paragraph [0045], Features are extracted such as the driver’s head, face, nose, eye, etc. See also paragraphs [0056]-[0061], Alpert discloses determining pose classification based on determined head feature extraction and further determines eye closure values over a period of time to determine a blink rate and/or PERCLOS metric as a measure of a certain amount of time. See also Figs. 5 and 6);
determine context data for a time at which the image data was captured, the context data relating to one or more environmental conditions associated with a vehicle operated by the person and to one or more physical characteristics of the vehicle (paragraphs [0042], [0048], [0069], [0076] and [0079]-[0080], Conditions monitored for the vehicle include the speed of the vehicle and driving operating conditions such as whether the vehicle is turning as well as GPS or location information and data from images of the area or environment surrounding the vehicle. Environment context is gathered related to the area outside the vehicle, as well as driver conditions within the vehicle such as the driver’s pose and/or head position);
select a baseline for an action context associated with the context data (paragraph [0069]: “In some embodiments, the first model 212 may be a first neural network model trained to classify pose(s) based on context. For example, if the driver's head is facing down, then the neural network model may determine that the driver is not looking straight even if the eyes of the driver cannot be detected (e.g., because the eyes may be blocked by a hat/cap). Also, in some embodiments, the second model 214 may be a second neural network model trained to determine whether the driver is drowsy or not based on context. For example, if the blink rate exceeds a certain threshold, and/or if the head-down pose and head-straight pose repeats in a period pattern, then the neural network model may determine that the driver is drowsy.” The blink scenario includes eye closure information over a period of time as well as if the driver’s head pose meets certain criteria which are interpreted broadly as environmental conditions. The baseline for the action context refers to what is considered normal, i.e. head poses or driving conditions within a baseline range compared to conditions such as head pose or driving conditions that exceed a threshold);
generate, using the set of blink parameters and the action context with at least a first neural network, at least a first drowsiness prediction for the person relative to the baseline paragraph [0069]: “In some embodiments, the first model 212 may be a first neural network model trained to classify pose(s) based on context. For example, if the driver's head is facing down, then the neural network model may determine that the driver is not looking straight even if the eyes of the driver cannot be detected (e.g., because the eyes may be blocked by a hat/cap). Also, in some embodiments, the second model 214 may be a second neural network model trained to determine whether the driver is drowsy or not based on context. For example, if the blink rate exceeds a certain threshold, and/or if the head-down pose and head-straight pose repeats in a period pattern, then the neural network model may determine that the driver is drowsy.” The blink scenario includes eye closure information over a period of time as well as if the driver’s head pose meets certain criteria which are interpreted broadly as environmental conditions.);
generate, using blink frequency information with at least a second neural network, a second drowsiness prediction for the person Alpert further discloses a second neural network model that uses information from the first neural network model to make an additional drowsiness determination (paragraphs [0056]-[0058], [0069], [0074] and [0083]-[0090]). Alpert further discloses that the second model uses the blink rate or blink frequency to determine drowsiness among other eye closure variables (paragraph [0091]); and
determine an overall drowsiness prediction based, at least, on the first drowsiness prediction and the second drowsiness prediction (paragraphs [0056]-[0058], [0069], [0074] and [0083]-[0090], Alpert repeatedly discloses that the drowsiness determination is based on both the first and second neural network models. The overall drowsiness determination is accordingly interpreted to be based on the first and second neural network models).
With regard to claim 19, Alpert discloses the system of claim 18, wherein the instructions if performed by the one or more processors further cause the system to:
determine the one or more environmental conditions using data from one or more cameras, sensors, global positioning system (GPS) signals, or network data sources, wherein the one or more environmental conditions relate to at least one of a road configuration, brightness, weather, time of day, location, speed, or number of surrounding objects (paragraphs [0039], [0042], [0046], [0076] and [0079]-[0080], Cameras capture the environment outside the vehicle. Sensors obtain information regarding speed and direction/turning of the vehicle. GPS location data is also captured. The system also communicates wirelessly with a network).
With regard to claim 20, Alpert discloses the system of claim 18, wherein the instructions if performed by the one or more processors further cause the system to:
determine a blink scenario based at least in part upon the one or more environmental conditions (paragraph [0069]: “In some embodiments, the first model 212 may be a first neural network model trained to classify pose(s) based on context. For example, if the driver's head is facing down, then the neural network model may determine that the driver is not looking straight even if the eyes of the driver cannot be detected (e.g., because the eyes may be blocked by a hat/cap). Also, in some embodiments, the second model 214 may be a second neural network model trained to determine whether the driver is drowsy or not based on context. For example, if the blink rate exceeds a certain threshold, and/or if the head-down pose and head-straight pose repeats in a period pattern, then the neural network model may determine that the driver is drowsy.” The blink scenario includes eye closure information over a period of time as well as if the driver’s head pose meets certain criteria which are interpreted broadly as environmental conditions); and
determine the first drowsiness prediction using one or more blink thresholds corresponding to the blink scenario (paragraph [0069], The blink rate exceeding a threshold indicates possible drowsiness).
With regard to claim 21, Alpert discloses the system of claim 18, wherein the system comprises at least one of:
a system for performing simulation operations;
a system for performing simulation operations to test or validate autonomous machine applications;
a system for rendering graphical output (paragraph [0119], displaying information to a user);
a system for performing deep learning operations (paragraph [0070], deep learning or artificial intelligence used to develop models);
a system implemented using an edge device (paragraph [0123], processing carried out on a remote device);
a system incorporating one or more Virtual Machines (VMs);
a system implemented at least partially in a data center (paragraphs [0046]-[0047] and [0101]-[0107], servers); or
a system implemented at least partially using cloud computing resources (paragraphs [0046], [0101] and [0110], Configured to communicate via network with cloud, internet, etc.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over combination of USPNs 2022/0215200 to Alpert et al. and 2019/0300034 to Molne et al.
With regard to claim 6, Alpert discloses the computer-implemented method of claim 2, but does not disclose determine the driver’s profile for recording blink behaviors. Molne disclose a similar eye tracking for determining the state of a driver as that of Alpert. Molne further discloses determining, an identity of the operator from image data including a representation of a face of the operator (paragraph [0044]); identifying a blink profile for the operator including one or more blink behaviors specific to the operator; and generating at least the first drowsiness prediction based further upon data for the one or more blink behaviors (paragraph [0059], Molne discloses that an operator’s profile is updated over time and optimized by updating the parameters used in the analytical decision engine for setting the individuals drowsiness level based on the combined PVT and PERCLOS drowsiness level metrics).
Therefore, it would have been obvious to one of ordinary skill in the art before time of filing to use the operator profiles as taught by Molne in order to give a more accurate and personalized measuring and assessment of the operator’s drowsiness in combination with the drowsiness monitoring of Alpert.
With regard to claim 15, the discussion of claim 6 applies.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPNs 2022/0215200 to Alpert et al, and 2008/0101659 to Hammoud et al.
With regard to claim 9, Alpert discloses the computer-implemented method of claim 2, but do not explicitly disclose wherein at least a subset of the set of blink parameters are determined using aspect ratio information calculated from the set of facial landmarks. Hammoud discloses a system for determining if a person’s eyes are opened or closed and discloses that aspect ratio of the identified eye area of the image is sued to make the determination (paragraphs [0010] and [0048]). Therefore, it would have been obvious to one of ordinary skill in the art before time of filing to use aspect ratio to determine if an eye is open or closed as taught by Hammoud in combination with the blink determination of Alpert in order to identify the open or closed eye state.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPNs 2022/0215200 to Alpert et al, and 2020/0151474 to Zandi et al.
With regard to claim 10, Alpert discloses the computer-implemented method of claim 2, wherein a first neural network to generate the first prediction and a second neural network to generate the second prediction are long short term memory (LSTM) networks (paragraph [0108], Neural network models described may be a long/short term memory neural network), but does not explicitly disclose that wherein the first prediction and the second prediction generated by the LSTM networks correspond to Karolinska Sleepiness Scale (KSS) values.
The Karolinska Sleepiness Scale was developed in 1990 as a measure of state sleepiness. Zandi teaches another driver monitoring system for that measures the driver’s state using eye closure or blinking measurements (paragraph [0061] and further teaches using the Karolinska Sleepiness Scale (paragraph [0028]). Therefore, it would have been obvious to one of ordinary skill in the art before time of filing to use the Karolinska Sleepiness scale as taught by Zandi in combination with the drowsiness determination of Alpert in order to apply a standard well known measure of drowsiness.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WESLEY J TUCKER whose telephone number is (571)272-7427. The examiner can normally be reached 9AM-5PM Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WESLEY J TUCKER/Primary Examiner, Art Unit 2661
