DETAILED ACTION
Response to Amendment
Claims 1-8 and 21-32 are pending. Claims 1-8 and 21-32 are amended directly or by dependency on an amended claim.
Response to Arguments
Applicant's arguments filed 29 July, 2026 have been fully considered but they are not persuasive. Applicant argues on page 8 Koukiou et al. does not use eye movement to determine intoxication and argues Krishnakumar et al. detects eye movement but does not use this to determine intoxication. Examiner uses Koukiou et al. to disclose the concept of using weighted vectors to determine intoxication, and relies on Krishnakumar et al. to teach the concepts of eye movements which are condensed into vectors. Modifying the vectors used by Koukiou et al. with the eye movement vector taught by Krishnakumar et al. would arrive at the claimed invention.
Applicant argues on page 9 there would not be reason to combine as Krishnakumar et al. is in the field of virtual reality and the combination would require hindsight. Examiner reasserts Koukiou et al. and Krishnakumar et al. are in the same art of monitoring human physiological characteristics in general (Koukiou et al., [0178], [0179]; Krishnakumar et al., [0067], [0096]). The combination of Krishnakumar et al. with Koukiou et al. will enable taking data of a driver's face over a time interval. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the face data of Krishnakumar et al. with the invention of Koukiou et al.as this was known at the time of filing, the combination would have predictable results, and as Krishnakumar et al. state “An IHS generally processes, compiles, stores, and/or communicates information or data for business, personal, or other purposes thereby allowing users to take advantage of the value of the information. Because technology and information handling needs and requirements vary between different users or applications, IHSs may also vary regarding what information is handled, how the information is handled, how much information is processed, stored, or communicated, and how quickly and efficiently the information may be processed, stored, or communicated. The variations in IHSs allow for IHSs to be general or configured for a specific user or specific use such as financial transaction processing, airline reservations, enterprise data storage, or global communications” ([0002]) “Generally speaking, xR application 401 may include any type of application now existing or yet to be developed, including, but not limited to: entertainment, video games, robotics, healthcare, education and training, military uses, occupational safety, engineering, industrial or product design, collaboration applications, virtual meetings, etc.” ([0051]) indicating that the system can be customized for applications in a variety of settings which suggests a commercial advantage to combining inventions. Further this long listing of the xR application to any type of application now existing or yet to be developed, including, but not limited to: entertainment, video games, robotics, healthcare, education and training, military uses, occupational safety, engineering, industrial or product design, collaboration applications, virtual meetings, etc. means using the eye movements to determine inebriation for the purposes of a driving application would have been an obvious application for the technology in general.
Applicant’s arguments with respect to “at least one of the one or more parameters comprises a nystagmus-related parameter determined based on the eve movements” have been considered but are moot because the new ground of rejection does not rely on the combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. It is noted in addition to the additional reference being used in the rejection, the conclusion also cites relevant prior art.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 22, and 29 (and by dependency claims 3 and 23) are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claims 2, 22, and 29 indicate “the one or more parameters comprise common characteristics, head pose, relative gaze, and patch appearance” however claims 1, 21, and 28 claim “the one or more parameters, wherein at least one of the one or more parameters comprises a nystagmus-related parameter determined based on the eye movements”. The head pose for instance is not a nystagmus-related parameter determined based on the eve movements. Therefore examiner believes claims 2, 22, and 29 should probably be phrased as “one or more parameters further includes separate parameters that comprise common characteristics, head pose, relative gaze, and patch appearance” or some other language to clarify the common characteristics, head pose, relative gaze, and patch appearance are not modifiers of the nystagmus-related parameter but instead separate additional parameters that are also included along with the nystagmus-related parameter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 4, 21, 22, 28, and 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Koukiou et al. (US 20240273724 A1) [priority based on provisional application 63/216916 filed 11 June, 2021, and WO2022/256943A1, published in English, designating the United States, international filing date 10 June 2022] in view of Krishnakumar et al. (US 20200225742 A1) in view of Migdal (US 8899748 B1).
Regarding claims 1, 21, and 28, Koukiou et al. disclose a method comprising, vehicle comprising (integrated into personal or company fleet vehicles or machinery, [0219]): one or more processors; a memory encoded with instructions, which, when executed by the one or more processors, causes the one or more processors to ([0192]-[0197]) and non-transitory machine-readable medium having instructions stored therein, which when executed by one or more processors, causes the one or more processors to ([0195]-[0197]): receiving data of a driver's face over a time interval (receiving a thermographic image including a face or facial features of the individual, [0009], [0176], [0187], screening for drivers, [0219]); for each frame of the data, determining one or more parameters associated with eye movements of the driver (taking the pixel values of 20 different points on the face of each person, [0178], temperature, [0179]); based on the one or more parameters for each frame, featurizing the frames into one or more vectors, wherein each of the one or more vectors corresponds to a parameter of the one or more parameters (The first simple feature vector for identification of an intoxicated state was obtained by simply taking the pixel values of 20 different points on the face of each person, [0178], Temperature distribution on the eyes of non-intoxicated and intoxicated persons was the second feature vector used in the tested feature fusion procedure. In most cases the sclera is brighter than the iris for intoxicated persons, while histogram modification algorithms can display the grey level difference between them for intoxicated persons, [0179]); applying a weight to each of the one or more vectors (The fusion procedure refers to manipulating the correlation between the above features xa and xb, as well as the importance of each of the features by weighting them with proper coefficients, final feature vector to be transferred to the Neural Networks, [0181]); based on the weight of each of the one or more vectors, predicting whether the driver surpassed an intoxication threshold (One can assess and identify intoxication if the cumulative values above a specific threshold constitute the majority of the values. If the majority of the values are below this threshold, then the person is not intoxicated, [0175], “For providing a reliability scoring method, the algorithm of the methods disclosed herein may be capable of enumerating the probability that a person is intoxicated (e.g. drunk). The enumeration may be displayable by some means as described herein enabling deployment in settings where some tolerance can be built in. For example, law enforcement making a traffic stop may have a policy to err on the side of caution, while a device deployed at a sports stadium serving alcohol will be more permissive. The notifications thresholds can be tunable in each case, for example via onboard configuration, a connected system or in embedded implementations using DIP switches or potentiometers”, [0217]); and if the driver surpassed the intoxication threshold, altering an operating characteristic of a vehicle of the driver (integrated into personal or company fleet vehicles or machinery to prevent an intoxicated person from operating the vehicle or machinery (e.g. heavy machinery); or (iv) used to screen employees as they enter job sites that require a low or zero threshold for intoxication (e.g. sobriety), [0219]).
Koukiou et al. do not disclose the data of a driver's face is over a time interval or determining one or more parameters associated with eye movements. Koukiou et al. do not disclose at least one of the one or more parameters comprises a nystagmus-related parameter determined based on the eye movements.
Krishnakumar et al. teach receiving data of a driver's face over a time interval (inward-facing camera configured to take high-frame-rate images of the eyes and their reflection patterns, [0061]); for each frame of the data, determining one or more parameters associated with eye movements of the driver (As such, in various embodiments, gaze tracking module 405 may produce, collect, measure, and/or produce information about a user's intent, cognitive processes, and behavior, based upon the user's eye movements, that may include, but is not limited to: (A) Fixations: phases when the eyes are stationary between movements and visual input occurs (e.g., total fixation duration, mean fixation duration, fixation spatial density, number of areas fixated, fixation sequences and fixation rate); (B) Saccades: rapid and involuntary eye movements that occur between fixations (e.g., saccade number, amplitude and fixation-saccade ratio); (C) Scanpath: a series of short fixations and saccades alternating before the eyes reach a target location on the screen (e.g., scanpath direction, duration, length and area covered); (D) Gaze duration: a sum of all fixations made in an area of interest before the eyes leave that area and also the proportion of time spent in each area; and/or (E) Pupil size and blink: measures used to study cognitive workload, [0067], “At block 808, method 800 gathers system telemetry data from a number of sensors, such as user 101's head position and distance 809 (e.g., from SLAM subsystem), head orientation 810 (e.g., from IMU 206), ambient light telemetry (e.g., from ALS sensor 206), and/or device capabilities and/or accuracy 812 (e.g., pixels/frame, frames/second, detection errors, false positives etc. obtained via calibration)”, [0096]); based on the one or more parameters for each frame, featurizing the frames into one or more vectors, wherein each of the one or more vectors corresponds to a parameter of the one or more parameters (As used herein, the term “gaze vector” is defined as a group of variables calculated using reflections from pupil center and another set of glints generated in the user's eye. In some cases, a gaze vector may include one or more of the following variables: 3D coordinates (x, y, z) of an initial point (e.g., the midpoint between the user's eyes), and pitch and yaw direction angles, [0062], processing of selected EGT outputs or data (e.g., gaze vector, speed, duration, etc.) from different EGT sources or systems being provided by gaze tracking module 405, [0069], first-fourth EGT vectors, [0095], [0096]); applying a weight to each of the one or more vectors (Block 813 calculates a first weight for first EGT vector 803, a second weight for second EGT vector 804, a third weight for third EGT vector 805, and/or a fourth weight for fourth EGT vector 806, [0096], For example, each weight may be calculated as a value between 0 and 1 or between −1 and 1. The assignment of weights for optimal EGT results (e.g., least errors, most accuracy, etc.) may be based on a machine learning algorithm or the like. Then, block 814 stores a composite or weighted EGT vector, for example, with each weight stored in the vector's header, [0097]).
Koukiou et al. and Krishnakumar et al. are in the same art of monitoring human characteristics (Koukiou et al., [0178], [0179]; Krishnakumar et al., [0067], [0096]). The combination of Krishnakumar et al. with Koukiou et al. will enable taking data of a driver's face over a time interval. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the face data of Krishnakumar et al. with the invention of Koukiou et al.as this was known at the time of filing, the combination would have predictable results, and as Krishnakumar et al. state “An IHS generally processes, compiles, stores, and/or communicates information or data for business, personal, or other purposes thereby allowing users to take advantage of the value of the information. Because technology and information handling needs and requirements vary between different users or applications, IHSs may also vary regarding what information is handled, how the information is handled, how much information is processed, stored, or communicated, and how quickly and efficiently the information may be processed, stored, or communicated. The variations in IHSs allow for IHSs to be general or configured for a specific user or specific use such as financial transaction processing, airline reservations, enterprise data storage, or global communications” ([0002]) “Generally speaking, xR application 401 may include any type of application now existing or yet to be developed, including, but not limited to: entertainment, video games, robotics, healthcare, education and training, military uses, occupational safety, engineering, industrial or product design, collaboration applications, virtual meetings, etc.” ([0051]) indicating that the system can be customized for applications in a variety of settings which suggests a commercial advantage to combining inventions.
Koukiou et al. and Krishnakumar et al. do not disclose at least one of the one or more parameters comprises a nystagmus-related parameter determined based on the eye movements.
Migdal teaches receiving data of a driver's face over a time interval (In an example embodiment, the eye movement detection software 56 can identify and locate the iris of one or each of the eyes of the test subject in each captured image frame in which the test subject's face is present, and the eye movements can be tracked based upon a changing location and positioning of the iris for one or both eyes in relation to an initial/starting position as determined at an initial/starting frame and/or in relation to a position determined for any other preceding frame for a series of sequential captured image frames (e.g., a series of sequential image frames that are captured during a nystagmus testing session utilizing the camera 20), col. 6, lines 20-35); for each frame of the data, determining one or more parameters associated with eye movements of the driver (Based upon such eye movement detection, the software 56 can be configured to develop an eye tracking or eye movement trajectory plot or map that indicates whether there is any sudden lateral/horizontal and/or vertical jerking movements of the subject's eyes during a test session. For example, trajectory vectors for each eye can be determined on a frame-by-frame basis in the series of sequential captured image frames being analyzed utilizing the software 56, and such trajectory vectors can be combined to generate an eye movement trajectory plot or map, col. 6, lines 44-53); based on the one or more parameters for each frame, featurizing the frames into one or more vectors, wherein each of the one or more vectors corresponds to a parameter of the one or more parameters wherein at least one of the one or more parameters comprises a nystagmus-related parameter determined based on the eye movements (Nystagmus is a rapid involuntary oscillation of the eyeballs. Horizontal Gaze Nystagmus (HGN) refers to the lateral jerking of the eyes as they gaze to the side (e.g., a sudden jerking movement of the eyes in an opposing direction in relation to the direction at which the eyes are gazing). Vertical Gaze Nystagmus (VGN) refers to the vertical jerking of the eyes as they gaze vertically. HGN has been shown to be an effective indicator of alcohol impairment when administered by a trained law enforcement professional during field sobriety testing. HGN and/or VGN testing is administered because, when intoxicated, the brain experiences difficulty correctly controlling the eye muscles which presents as the nystagmus, col. 1, lines 5-20, “a series of sequential image frames that are captured during a nystagmus testing session utilizing the camera 20). Based upon such eye movement detection, the software 56 can be configured to develop an eye tracking or eye movement trajectory plot or map that indicates whether there is any sudden lateral/horizontal and/or vertical jerking movements of the subject's eyes during a test session. For example, trajectory vectors for each eye can be determined on a frame-by-frame basis in the series of sequential captured image frames being analyzed utilizing the software 56, and such trajectory vectors can be combined to generate an eye movement trajectory plot or map. The eye movement/trajectory plot can be displayed, for example, utilizing the display 24 of the device 4”, col. 6, lines 40-55); and predicting whether the driver surpassed an intoxication threshold (The nystagmus detection algorithm(s) software 59 comprises one or more algorithms which determine a degree of lateral/horizontal and/or vertical jerking movement that occurs, based upon the image frames that have been captured by the camera 20 and stored in the DVR 58 during a test session and based upon the eye movement detections of such captured image frames as determined by software 56. The algorithms of software 59 can apply any one or more suitable and recognized standards for an indication of nystagmus based upon the degree of horizontal and/or vertical jerking of the eyes during eye movements by a test subject to the left and the right (e.g., to a location of about 45.degree. to the left or the right from a central location between the test subject's eyes). For example, the algorithm(s) may apply one or more standards for a particular legal jurisdiction which define a particular nystagmus test and measured degree of jerking eye movement that adequately establish intoxication or other impairment of a test subject during a field sobriety test utilizing the system 2, col. 6, line 55 – col. 7, line 10, “The processor 52 utilizes the nystagmus detection algorithm software 59 to make a determination regarding whether the eye movement detection and tracking indicates a presentation of nystagmus or a presentation of a degree of nystagmus approaching or exceeding a threshold which correlates with impairment of the test subject (e.g., impairment due to the effects of alcohol consumed by the test subject, such as a blood alcohol concentration or BAC level of the subject that exceeds a threshold value, and/or any other form of impairment that can be detected by the presentation of nystagmus by the test subject during the test session)”, col. 12, lines 20-30).
Koukiou et al. and Arora et al. and Migdal are in the same art of monitoring human characteristics/driver characteristics (Koukiou et al., [0178], [0179], [0219]; Arora et al., [0021]; Migdal, abstract). The combination of Migdal with Koukiou et al. and Krishnakumar et al. will enable measuring nystagmus. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the nystagmus of Migdal with the invention of Koukiou et al. and Krishnakumar et al. as this was known at the time of filing, the combination would have predictable results, and as Migdal states “A system suitable for use in field testing and automatic determination of a degree of eye nystagmus is designed to be portable and hand-held to facilitate ease of operability in a number of different environments. The system is non-intrusive, requiring little or no contact with the test subject during operation. The automated determination of HGN/VGN responses by test subjects during operation of the device provides a number of benefits, including providing an objective standard as well as recorded data in determining field sobriety in contrast to current tests administered by law enforcement agencies (e.g., a subjective determination by a law enforcement officer, where sobriety is based upon an officer's observation of eye movement by the subject during the test)”, col. 2, lines 54-67) demonstrating the benefits to the combination of inventions when applied in a law enforcement or driver safety setting.
Regarding claims 2 and 22 and 29, Koukiou et al. and Krishnakumar et al. and Migdal disclose the method, vehicle, and CRM of claims 1, 21, and 28. Krishnakumar et al. further indicate the one or more parameters comprise common characteristics, head pose, relative gaze, and patch appearance (In various implementations, gaze tracking module 405 may use, among camera(s) and/or sensor(s), NIR light sources to produce glints on the surface of the cornea of the user's eye, and then it may capture images of the eye region using an inward-facing camera. Gaze tracking module 405 may estimate the user's gaze from the relative movement between the pupil center and glint positions. Particularly, an eye and/or gaze tracking algorithm may perform corneal reflection-based calculations that use MR illumination to estimate the gaze direction or the point of gaze using polynomial functions, or a geometrical model of the human eye, [0063], In some cases, gaze tracking module 405 may be configured to compensate for head movement. For example, a mathematical formula indicating the correspondence of eye gaze with head positions may represent a relationship between user reference gaze directions, head pose direction, and actual gaze direction; which is a result of both head and eye rotation, [0065], As such, in various embodiments, gaze tracking module 405 may produce, collect, measure, and/or produce information about a user's intent, cognitive processes, and behavior, based upon the user's eye movements, that may include, but is not limited to: (A) Fixations: phases when the eyes are stationary between movements and visual input occurs (e.g., total fixation duration, mean fixation duration, fixation spatial density, number of areas fixated, fixation sequences and fixation rate); (B) Saccades: rapid and involuntary eye movements that occur between fixations (e.g., saccade number, amplitude and fixation-saccade ratio); (C) Scanpath: a series of short fixations and saccades alternating before the eyes reach a target location on the screen (e.g., scanpath direction, duration, length and area covered); (D) Gaze duration: a sum of all fixations made in an area of interest before the eyes leave that area and also the proportion of time spent in each area; and/or (E) Pupil size and blink: measures used to study cognitive workload, [0067]).
Regarding claim 4, Koukiou et al. and Krishnakumar et al. and Migdal disclose the method of claim 1. Koukiou et al. and Krishnakumar et al. further indicate the data comprises video data from a plurality of cameras (Mirrors 201R and 201L (e.g., “hot mirrors”) may be positioned to direct NIR light reflected from eyes 101R and 101L into EGT cameras 202R and 202L located on each side of the user's face. In other implementations, instead of EGT cameras 202R and 202L, a single EGT camera, or a combination of a wide-angle camera with and a narrower-angle camera, may be used, [0038], Gesture detection begins when video frame data (e.g., a video or depth-video stream) is received at host IHS 103 from gesture or RGB camera 108 of HMD 102, [0057], Gaze tracking module 405 may use an inward-facing projector, configured to create a pattern of infrared or (near-infrared) light on the user's eyes, and an inward-facing camera configured to take high-frame-rate images of the eyes and their reflection patterns; which are then used to calculate the user's eye's position and gaze focus or point, [0061], “Projector/camera assembly 502 may be configured to take high-frame-rate images of the user's eyes and their reflection patterns”, [0079]) [high frame rate camera interpreted as video equivalent].
Claim(s) 3, 7, 23, 26, and 32 is/are rejected under 35 U.S.C. 103 as being unpatentable over Koukiou et al. (US 20240273724 A1) and Krishnakumar et al. (US 20200225742 A1) and Migdal (US 8899748 B1) as applied to claims 1, 21, and 28 above, further in view of Chen et al. (“Support Vector Machine Classification of Drunk Driving Behaviour”).
Regarding claims 3 and 23, Koukiou et al. and Krishnakumar et al. and Migdal disclose the method and vehicle of claims 1 and 21. Koukiou et al. and Krishnakumar et al. and Migdal do not disclose the relative gaze parameter has a highest weight.
Chen et al. teach relative gaze parameter has a highest weight (The PCA results show that the weights of the original features have different impacts on the overall measurements. A relatively larger weight means the sensitivity of the feature is relatively larger when judging whether a driver is exhibiting drunk driving behaviour, and those features are more effective in accurately identifying drunk driving. SDNN, RMSSD, LF, HF, LF/HF, and average blink duration were the highest-weighted features in the study. SDNN index, which reflects the slow change of heart
rate, is a sensitive index for evaluating the function of the sympathetic nerve, p10, “In addition, principal component analysis was conducted to rank the weights of the features. The
standard deviation of R–R intervals (SDNN), the root mean square value of the difference of the
adjacent R–R interval series (RMSSD), low frequency (LF), high frequency (HF), the ratio of the low and high frequencies (LF/HF), and average duration time of blink were the highest weighted features in the study. SDNN, RMSSD, LF, HF, and LF/HF were all extracted from the PPG signal, indicating the importance of collecting HRV data in the detection of drunk driving. Average blink duration explains the importance of eye tracking in the study of drunk driving prevention”, p11).
Koukiou et al. and Chen et al. are in the same art of monitoring human characteristics/drunk driving characteristics (Koukiou et al., [0178], [0179], [0219]; Chen et al., abstract). The combination of Chen et al. with Koukiou et al. and Krishnakumar et al. and Migdal will enable giving the relative gaze parameter a highest weight. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the weight of Chen et al. with the invention of Koukiou et al. and Krishnakumar et al. and Migdal as this was known at the time of filing, the combination would have predictable results, and as Chen et al. state “The results show that SVM classification can successfully distinguish drunk driving from normal driving with an accuracy of 70%. The driving performance data and the physiological measurements reported by this paper combined with air-alcohol concentration could be integrated using the support vector regression classification method to establish a better early warning model, thereby improving vehicle safety” (abstract) demonstrating a safety improvement when the inventions are combined.
Regarding claims 7, 26, and 32, Koukiou et al. and Krishnakumar et al. and Migdal disclose the method, vehicle, and CRM of claims 1, 21, and 28. Koukiou et al. and Krishnakumar et al. imply the one or more parameters evaluate how the frames change over the time interval (Koukiou et al., gaze tracking module 405 may be configured to identify a direction, extent, and/or speed of movement of the user's eyes in real-time, during execution of an xR application (e.g., a gaze vector), [0061]; Krishnakumar et al., In some embodiments, the methods herein involve a step of comparing data obtained from the one or more thermographic images to a previously collected data set. The previously collected data set may for example be the known data set. In some embodiments, the previously collected data set comprises thermographic imaging data for one or more states or levels of intoxication or for absolute non-intoxication (e.g. sobriety). In other embodiments, the previously collected data set may be thermographic images from the tested individual at an earlier point in time. For example, the earlier point in time may be a matter of seconds, minutes or hours prior to the current thermographic images. In some embodiments, the previously collected data set is 5 seconds, 10 seconds, 30 seconds, 1 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, 3 hours, 4 hours, 5 hours, 6 hours, 7 hours, 8 hours, 9 hours, 10 hours, 11 hours, 12 hours, 18 hours, 24 hours, 2 days, 3 days, 4 days, 5 days, 6 days, 7 days, or more prior to the current thermographic images. Comparing the previously collected data set to the current data set may provide information on thermographic trends useful in assessing intoxication status or level, [0126]), however another reference is added to make the limitation more explicit.
Chen et al. teach the one or more parameters evaluate how the frames change over the time interval (Several eye activity parameters have been shown to be sensitive to time on task, and research has shown that the disappearance of blinks, mini-blinks in eye movement, are the earliest reliable signs of drowsiness [27]. The eye activity monitoring could be used as an operator’s alertness level. The participants’ eye blink duration times were also analyzed in this study, p5, The driving performance data were collected from the driving simulator, and the physiological behaviour data were collected from the wireless sensors. In addition, eye blink data were collected from the Tobii eye tracker. For the driving performance and physiological measurements, 20 original features were selected that represent the time domain or the frequency domain characteristics of the various signals, p7) [the commercially available Tobii eye tracker is known to monitor blinks etc in frames over time].
Koukiou et al. and Chen et al. are in the same art of monitoring human characteristics/drunk driving characteristics (Koukiou et al., [0178], [0179], [0219]; Chen et al., abstract). The combination of Chen et al. with Koukiou et al. and Krishnakumar et al. and Migdal will enable evaluating how the frames change over the time interval. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the evaluating of Chen et al. with the invention of Koukiou et al. and Krishnakumar et al. and Migdal as this was known at the time of filing, the combination would have predictable results, and as Chen et al. state “The results show that SVM classification can successfully distinguish drunk driving from normal driving with an accuracy of 70%. The driving performance data and the physiological measurements reported by this paper combined with air-alcohol concentration could be integrated using the support vector regression classification method to establish a better early warning model, thereby improving vehicle safety” (abstract) demonstrating a safety improvement when the inventions are combined.
Claim(s) 5, 24, and 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Koukiou et al. (US 20240273724 A1) and Krishnakumar et al. (US 20200225742 A1) and Migdal (US 8899748 B1) as applied to claim 1, 21, and 28 above, further in view of Arora et al. (US 20210397859 A1).
Regarding claims 5, 24, and 30, Koukiou et al. and Krishnakumar et al. and Migdal disclose the method, vehicle, and CRM of claims 1, 21, and 28. Koukiou et al. and Krishnakumar et al. and Migdal do not disclose attributing a confidence value to each of the one or more vectors and weighting the one or more vectors based on the confidence values.
Arora et al. teach attributing a confidence value to each of the one or more vectors and weighting the one or more vectors based on the confidence values (Vehicle computing system 104 may determine weights to apply to the first and second initial 3D eye gaze vectors based on a confidence that the first or second initial 3D eye gaze vector was accurately determined. For example, if vehicle computing system 104 detected only a portion of the facial landmarks (e.g., just the eyes and nose, but no mouth or jaw), there may be an increased error range for the pitch, roll, and yaw angles of the facial plane. As such, vehicle computing system 104 may apply a reduced weighting value to the second initial 3D eye gaze vector and an increased weighting value to the first initial 3D eye gaze vector determined using eye tracking, [0021]).
Koukiou et al. and Arora et al. are in the same art of monitoring human characteristics/driver characteristics (Koukiou et al., [0178], [0179], [0219]; Arora et al., [0021]). The combination of Arora et al. with Koukiou et al. and Krishnakumar et al. and Migdal will enable attributing a confidence value to each of the one or more vectors and weighting the one or more vectors based on the confidence values. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the attributing of Arora et al. with the invention of Koukiou et al. and Krishnakumar et al. and Migdal as this was known at the time of filing, the combination would have predictable results, and as Arora et al. state “In this and other ways, vehicle computing system 104 may improve the driving or riding experience for one or more occupants of the vehicle” ([0014]) “Further, vehicle computing system 104 may use the more accurate determination of what the occupant is looking at to automatically take various safety-related actions or any number of other actions. Accordingly, techniques of this disclosure may enable improved vehicle safety systems” ([0029]) and “However, in various instances, the occupant's eyes may be occluded and, thus, the images captured by the cameras may not include a clear image of the occupant's eyes. Thus, rather than just relying upon tracking the position and movement of the occupant's eyes, eye gaze module 522 may further analyze the captured images to determine a facial plane of the occupant” ([0056]) thereby the combination can take cues such as occlusion to better use the correct gaze information, which will result in improved driver and passenger safety.
Claim(s) 6, 25, and 31 is/are rejected under 35 U.S.C. 103 as being unpatentable over Koukiou et al. (US 20240273724 A1) and Krishnakumar et al. (US 20200225742 A1) and Migdal (US 8899748 B1) and Arora et al. (US 20210397859 A1) as applied to claims 5, 24, and 30 above, further in view of Gallagher et al. (US 20190332902 A1).
Regarding claims 6, 25, and 31, Koukiou et al. and Krishnakumar et al. and Migdal and Arora et al. disclose the method, vehicle, and CRM of claims 5, 24, and 30. Krishnakumar et al. and Arora et al. partly indicate determining an overall confidence value based on the confidence values and the weights (Krishnakumar et al., calculate a composite EGT vector using a first weight associated with the first EGT system, a second weight associated with the second EGT system, and a third weight associated with the third EGT system, [0012], [0074], Arora et al., Once the first and second initial 3D eye gaze vectors are determined, vehicle computing system 104 may determine, based on both the first and second initial 3D eye gaze vectors, 3D eye gaze vector 112 of the occupant. In various instances, vehicle computing system 104 may determine 3D eye gaze vector 112 using an average of the first and second initial 3D eye gaze vectors. Vehicle computing system 104 may, in other examples, apply a weighting to one or more of the first and second 3D eye gaze vectors and use the weighted values to determine 3D eye gaze vector 112, [0021]) however another reference is added to make this explicit.
Gallagher et al. teach determining an overall confidence value based on the confidence values and the weights (“This system can also include cameras strategically positioned to look at the driver. Inward cameras can be used in conjunction with the seat sensors to achieve sensor fusion and increase specificity and accuracy of the distraction level detection. The camera generates multiple images of the occupant, which can be analyzed to determine additional occupant metrics. The metrics can include head position, a blink rate, pupil dilation, eye position, fixation, gaze patterns, eyelid closure, head movement facial expression, overall skeletal position, breathing rate, heart rate, and the like. The camera system takes an image and image processing circuitry analyzes the image to determine the image metric”, [0037], camera data is used to detect occupant physical characteristics, e.g., muscle movement, eye movement, breathing patterns, and the like, [0064], Examples of state can include angry, sad, happy, drunk, distracted drowsy, and the like, images from the camera may also be analyzed and determine that the occupant may be intoxicated, [0065], receives the emotional valence and arousal level 320, e.g., a vector as described herein, based on the CNS sensed data, “The combined arousal calculation level 372 combines the confidence levels 370, 370C to output an overall confidence level to the output stages for further processing or storage, which can each individually adjust the final emotional valence and arousal level 320C associated with the occupant of the vehicle. The processing at 300C computes a combined vector 320C based on a final valence/arousal level(s) from each type of sensed signal. The combined vector 320C represents the emotional state of the occupant. The combined vector can be fed into a further artificial intelligence logic tree, e.g., within the neural network in the vehicle, that uses long term information to take the occupant individuality into account for stabilization and environmental impacts to further improve weighting in varying surrounding trends. The final vector 320C or other outputs based on the calculations can be used in other vehicle systems to trigger or suggest preemptive countermeasures”, [0072]).
Koukiou et al. and Gallagher et al. are in the same art of monitoring human characteristics/driver characteristics (Koukiou et al., [0178], [0179], [0219]; Gallagher et al., [0037], [0065]). The combination of Gallagher et al. with Koukiou et al. and Krishnakumar et al. and Migdal and Arora et al. will enable determining an overall confidence value based on the confidence values and the weights. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the determining of Gallagher et al. with the invention of Koukiou et al. and Krishnakumar et al. and Migdal and Arora et al. as this was known at the time of filing, the combination would have predictable results, and as Gallagher et al. state “The combined vector can be fed into a further artificial intelligence logic tree, e.g., within the neural network in the vehicle, that uses long term information to take the occupant individuality into account for stabilization and environmental impacts to further improve weighting in varying surrounding trends. The final vector 320C or other outputs based on the calculations can be used in other vehicle systems to trigger or suggest preemptive countermeasures” ([0072]) suggesting an accuracy improvement that will result from combining inventions.
Claim(s) 8 and 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Koukiou et al. (US 20240273724 A1) and Krishnakumar et al. (US 20200225742 A1) and Migdal (US 8899748 B1) as applied to claims 1 and 21 above, further in view of Taylor et al. (US 20240142777 A1).
Regarding claims 8 and 27, Koukiou et al. and Krishnakumar et al. and Migdal disclose the method and vehicle of claims 1 and 21. Koukiou et al. and Krishnakumar et al. and Migdal do not disclose predicting whether the driver surpassed the intoxication threshold is accomplished with a temporal network.
Taylor et al. teach predicting whether the driver surpassed the intoxication threshold is accomplished with a temporal network (“In yet another aspect, the processor 142 may actuate the vehicle user inebriation test when the processor 142 determines that the vehicle user may be fatigued. In some aspects, the processor 142 may determine that the vehicle user may be fatigued by measuring vehicle user eye closure percentage, when the vehicle user is driving the vehicle 102 or otherwise. For example, a vehicle camera (e.g., a vehicle driver tracking camera or a vehicle occupant camera) may capture vehicle user eyes and may send the captured eye information to the processor 142. Responsive to receiving the eye information, the processor 142 may determine (e.g., by using deep or convolutional neural network architecture and/or training set) whether the vehicle user eye closure percentage is greater than a predetermined threshold. Responsive to determining that the closure percentage is greater than the predetermined threshold, the processor 142 may determine that the vehicle user may be fatigued and may actuate the inebriation test (or may actuate the general vehicle user focus test before actuating the inebriation test)”, [0045], “For instance, the algorithms may include long short-term memory (LSTM) neural network based algorithms, recurrent neural network (RNN) based algorithms, spatio-temporo neural network based algorithms and/or the like”, [0056]) [LSTMs are temporal networks].
Koukiou et al. and Taylor et al. are in the same art of monitoring human characteristics/driver characteristics (Koukiou et al., [0178], [0179], [0219]; Taylor et al., [0045]). The combination of Taylor et al. with Koukiou et al. and Krishnakumar et al. and Migdal will enable predicting whether the driver surpassed the intoxication threshold is accomplished with a temporal network. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the network of Taylor et al. with the invention of Koukiou et al. and Krishnakumar et al. and Migdal as this was known at the time of filing, the combination would have predictable results, and as Taylor et al. state, “The present disclosure provides various advantages over the conventional approaches to mitigate or prevent intoxicated operation of a motor vehicle configured with the disclosed system. The vehicle may provide coaching to the vehicle user to take the inebriation test without involvement of any third party (such as enforcement officers or police officers). Further, the system does not require addition of a breath analyzer to detect a vehicle user inebriation state” ([0020]) providing a safety benefit likely to result from combining inventions.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: US 20260205298 A1 [provisional predates but does not include nystagmus portion of the reference]: This temporal tracking may be used for detecting physiological and behavioral indicators such as smooth pursuit, saccades, and nystagmus, which are associated with intoxication or impairment. For example, after normalization and region-of-interest (ROI) preparation, optionally a given frame may be converted into compact feature vectors, such as learned embeddings from a CNN backbone focused on the eye crops, CV features (e.g., Eye Aspect Ratio, pupil diameter in pixels, eyelid geometry), and/or motion features from optical flow. These per-frame feature vectors form a temporal sequence that may be fed into an LSTM stack or a Transformer encoder; wherein both architectures model time-dependent behavior across windows of frames so that the system can reason about temporal phenomena such as blinks, smooth pursuit, saccades (rapid movement of the eye between fixation points), and/or nystagmus. As noted above, a transformer encoder may optionally be utilized to perform such feature detection. The transformer encoder operates on the same per-frame feature sequence augmented with positional encodings (e.g., vectors of the same dimension as the feature vector) to preserve temporal order. Multi-head self-attention enables the model to discover long-range dependencies (e.g., periodic jerk patterns that characterize nystagmus, or sustained intervals of smooth pursuit) and to weigh frames that are most predictive for stability indices (where attention can focus near maximum deviation or early onset angles). This is advantageous for gaze stability and eye-movement patterns, where meaningful signatures may span hundreds of milliseconds to seconds; US 20140152792 A1: The term nystagmus is derived from the Greek word, "nmstagmos", which was used to describe the wobbly head movements of a sleepy or inebriated individual. Nystagmus has also been described as "dancing eyes" or "jerking eyes". Doctors and researchers classify nystagmus by the characteristics of the eye movements like do they move back and forth like a pendulum or do they move slowly in one direction and then rapidly in another. Sometimes a horizontal and vertical nystagmus will occur together. They are usually of the same frequency, but their relative phase may different. Depending on the mutual phase, the eyes may take on an oblique direction (form an ellipse). This seems possible however, as in a nystagmus called "windmill nystagmus", the vector of the horizontal nystagmus changes from right-left to up-down and back again, constantly; US 20190191995 A1: Alcohol is present, but not the primary toxin, in 67% of ED admissions that require toxicological analysis in the U.S. Nystagmus is a reliable indicator of alcohol intoxication. Nystagmus is characterized by the gaze drifting away slowly from its point of focus (slow phase) and then snapping back quickly (fast phase). These quick phases of nystagmus are rapid, with maximum velocities as high as 500 degrees per second. These rapid eye movements are evolutionary forerunners of voluntary saccades. It is preferable for the backend server to have a separate vector for alcohol, to complement the pupillary measurements and help discriminate between substances. Therefore, in addition to pupillary diameter, some embodiments of the disclosed system and device also track and measure eye movements including, but not limited to, smooth pursuits, saccades, visual fixation movements, vergences and the various types of nystagmus; US 12756571 B2 [provisional predates but does not include nystagmus portion of the reference]: The nystagmus analysis service 212B may optionally also be utilized by the consent verification service 202B to determine if the user is intellectually capable of providing consent. For example, as described elsewhere herein, nystagmus refers to an inability to adequately control eye movements, which may be evidence of intoxication. The nystagmus analysis service 212B may be utilized to determine if a person is suffering from nystagmus, and hence may be under the influence of an intoxicating substance and is intellectually incapable of providing consent. Likelihood subject is intoxicated (LSI)=1/N(W.sub.1D.sub.1+W.sub.2D.sub.2+W.sub.3D.sub.3+W.sub.4D.sub.4 . . . W.sub.nD.sub.n) where: N=Normalization factor W=Weight D.sub.1=intoxication determination based on voice analysis D.sub.2=intoxication determination based on eye motion analysis (e.g., horizontal nystagmus, smooth tracking) D.sub.3=intoxication determination based on pupil size/diameter analysis D.sub.4=intoxication determination based on eye blink rate analysis D.sub.n=intoxication determination based on other determinations (e.g., smoothness of movement of the subject's device during testing, ability to focus gaze on the subject's device during testing, and/or other determinations) Wherein Subject is unable to provide consent when LSI>Threshold For example, after normalization and region-of-interest (ROI) preparation, optionally a given frame may be converted into compact feature vectors, such as learned embeddings from a CNN backbone focused on the eye crops, CV features (e.g., Eye Aspect Ratio, pupil diameter in pixels, eyelid geometry), and/or motion features from optical flow. These per-frame feature vectors form a temporal sequence that may be fed into an LSTM stack or a Transformer encoder; wherein both architectures model time-dependent behavior across windows of frames so that the system can reason about temporal phenomena such as blinks, smooth pursuit, saccades (rapid movement of the eye between fixation points), and/or nystagmus. As noted above, a transformer encoder may optionally be utilized to perform such feature detection. The transformer encoder operates on the same per-frame feature sequence augmented with positional encodings (e.g., vectors of the same dimension as the feature vector) to preserve temporal order. Multi-head self-attention enables the model to discover long-range dependencies (e.g., periodic jerk patterns that characterize nystagmus, or sustained intervals of smooth pursuit) and to weigh frames that are most predictive for stability indices (where attention can focus near maximum deviation or early onset angles). This is advantageous for gaze stability and eye-movement patterns, where meaningful signatures may span hundreds of milliseconds to seconds.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE ENTEZARI whose telephone number is (571)270-5084. The examiner can normally be reached 10-7 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent M Rudolph can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE M ENTEZARI HAUSMANN/Primary Examiner, Art Unit 2671
Prosecution Insights