DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 9, 18 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-11, 13-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2023/0112797 by Sicconi et al. in view of US 12,380,190 by Lyle et al.
Regarding claim 1, Sicconi et al. discloses a method comprising:
receiving two or more image frames of an operator of a machine (paragraph 0049 teaches “Detecting operator face contours may further include comparing a first frame of the video feed to a second from of the video feed and determining that a number of pixels exceeding a threshold amount has changed with respect to at least a parameter from the first frame to the second frame. “Parameters” as used herein for at least an operator-facing camera 140 may include a multitude of parameters such as those things referred to in detecting face contours. A parameter change is a change in a parameter of one or more pixels that exceeds some threshold of number of pixels experiencing the change per unit of time and/or framerate. As a non-limiting example, detecting a parameter change may include comparing a first frame of a video feed to a second frame of the video feed, and determining that a threshold number of pixels has changed with respect to at least a parameter from the first frame to the second frame.”);
determining, based on the two or more image frames, one or more gaze features of the operator (in addition to discussion above, paragraph 0049 teaches “Detecting operator face contours may include identifying eyes, nose, and/or mouth to evaluate yaw, pitch, roll of the face, orientation of operator eyes and eye gaze direction, orientation of operator face, operator eyelids closing patterns, or the like.”, paragraph 0051 teaches “The direction of operator focus, as used herein, refers to the orientation of the operator's eyes (eye gaze), for example straight ahead at the road, down at something internal to the vehicle such as a mobile phone, to the left outside the driver side window where a distracting even may be occurring, or the like.”);
providing, as input to an artificial intelligence (AI) model, the sequence of features of the operator, wherein the Al model is trained to analyze the sequence of features to provide an indication of an abnormal gaze pattern of the operator during the time window;
receiving, as output from the Al model, the indication of the abnormal gaze pattern (in addition to discussion above, paragraph 0050 teaches “A processing engine 120 may perform one or more artificial intelligence processes to detect face contours at any given time using a machine-learning model or deep learning model; for instance, and without limitation, those models mentioned above for external object detection by at least a road-facing camera 116. In an embodiment, operator face contours detected by at least an operator-facing camera 140 using any such method as described above may be used in generating at least an operator state datum.”, paragraph 0075 teaches “Referring now to FIG. 4, a method of using artificial intelligence to present geographically relevant user-specific recommendations based on vehicle operator attentiveness is illustrated.”); and
determining, based on at least the indication of the abnormal gaze pattern, whether the operator is distracted (in addition to discussion above, paragraph 0037 teaches “As a further non-limiting example, training data may associate any data from a sensor, camera, vehicle, or operator, as mentioned in this disclosure herein, or any combination of the like, which detect external objects, operator face contours, operator speech and the like, to assess driving conditions, operator distractibility, operator alertness, or the like.”, paragraph 0044 teaches “Monitoring with the mentioned variables may be correlated with the road ahead of vehicle, left mirror, right mirror, central rearview mirror, instrument cluster, center dash, passenger seat, mobile phone in hand, or the like; such detection may, without limitation, be performed using an operator-facing camera as described in further detail below. Monitoring information extracted by the least a driving condition sensor 108 may be compiled into at least an operator state datum to give the overall state of a current operator, which may include state of alertness and/or distractibility.”, paragraph 0051 teaches “With continued reference to FIG. 1, at least an operator sensor 124 may be further configured to generate at least an operator state datum by determining a direction of operator focus. The direction of operator focus, as used herein, refers to the orientation of the operator's eyes (eye gaze), for example straight ahead at the road, down at something internal to the vehicle such as a mobile phone, to the left outside the driver side window where a distracting even may be occurring, or the like. Detection and tracking of operator's eye may be performed by any combination of sensor types and/or camera types and/or computer devices, as described above.”, paragraph 0053-0054).
Sicconi et al. fails to disclose
wherein the one or more gaze features comprise a sequence of gaze features corresponding to a time window spanning a period of time; providing, as input to an artificial intelligence (AI) model, the sequence of gaze features of the operator, wherein the Al model is trained to analyze the sequence of gaze features.
Lyle et al. discloses wherein the one or more gaze features comprise a sequence of gaze features corresponding to a time window spanning a period of time (fig. 1, 6); providing, as input to an artificial intelligence (AI) model, the sequence of gaze features of the operator, wherein the Al model is trained to analyze the sequence of gaze features (col. 4 lines 55-60, col. 9 lines 50-56 teaches “Based on the received image data 140, the system will determine whether to authenticate the user for access to the secured resource. For example, verification module 150 can process and analyze image data 140 and determine whether the image data 140 includes a change in expression of the user that corresponds to the expected reference pattern that should be elicited when the pattern 154 was displayed.”, col. 14 lines 43+, col. 18 lines 38-65 teaches “Referring next to FIG. 5, a depiction of video 500 as received by the second device 314 is shown to the agent 312. In response to the second control signal, first image data 510 from video 500 is captured at a first time, in this case beginning just prior to the presentation of the visual output. The first image data 510 may be provided to authorization system 250 for processing, as indicated by a first status message 550 (“Receiving Video . . . Processing Video Data”). During the duration of the presentation of the visual output, the camera continues to capture video. The image data is processed in order to extract relevant features, as shown in the example of FIG. 6, where a set of features from a user input 610 has been extracted, comprising a first gaze direction 612, a second gaze direction 614, a third gaze direction 616, and a fourth gaze direction 618. It can be understood that each change in gaze, from first to second to third to fourth, is substantially in sync with the flash of icons on the screen (see FIG. 4). This response is identified, and will be compared to a reference reflexive response 650 to verify that the second user 340 responded to the visual output as expected.”).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the ability to include the one or more gaze features comprise a sequence of gaze features corresponding to a time window spanning a period of time, as taught by Lyle et al. into the system of Sicconi et al., because such incorporation would allow for the benefit of tracking the gaze feature to find change/abnormality, thus increase user flexibility to the system.
Regarding claim 2, the method wherein the one or more gaze features comprise at least one of a raw gaze vector, a gaze pattern, a gaze fixation, or a regional gaze prediction (in addition to discussion above, Sicconi et al., paragraph 0051 teaches “With continued reference to FIG. 1, at least an operator sensor 124 may be further configured to generate at least an operator state datum by determining a direction of operator focus. The direction of operator focus, as used herein, refers to the orientation of the operator's eyes (eye gaze), for example straight ahead at the road, down at something internal to the vehicle such as a mobile phone, to the left outside the driver side window where a distracting even may be occurring, or the like. Detection and tracking of operator's eye may be performed by any combination of sensor types and/or camera types and/or computer devices, as described above.”, paragraph 0052 teaches “generating at least an operator state datum as a function of the spatial difference. Spatial difference may include, without limitation, an angle between a vector from the operator to spatial location of an object/event/distraction and a vector in a direction of current operator eye gaze, as measured using cosine similarity or the like.”).
Regarding claim 3, the method wherein determining the one or more gaze features comprises:
providing, as second input to a second Al model, each of the two or more image frames (in addition to discussion above, Sicconi et al., paragraph 0024 teaches “Connectivity data records may be aggregated at remote device, cloud service, and/or at least a transport communication device 104 by location, time, and/or condition set, producing a set of data records indicative thereof, which may be periodically updated.”); and
receiving, as second output from the second Al model, the one or more gaze features (in addition to discussion above, Sicconi et al., fig. 4, paragraph 0075 teaches “Referring now to FIG. 4, a method of using artificial intelligence to present geographically relevant user-specific recommendations based on vehicle operator attentiveness is illustrated. At step 405, at least a transport communication device 104 communicates with a vehicle. This may be implemented, for instance, as described above in reference to FIGS. 1-3. At step 410, at least a transport communication device 104 uses at least a driving condition sensor 108 to monitor driving conditions, wherein at least a driving condition datum is generated. This may be implemented, for instance, as described above in reference to FIGS. 1-3. Monitoring driving conditions may further include capturing a video feed of conditions external to the vehicle. This may be implemented, for instance, as described above in reference to FIGS. 1-3. Capturing a video feed of conditions external to the vehicle may further include detecting external objects and generating at least a driving condition datum as a function of detected external objects. This may be implemented, for instance, as described above in reference to FIGS. 1-3 . . . above in reference to FIGS. 1-3.”).
Regarding claim 5, the method further comprising: determining, based on the two or more image frames, one or more additional features corresponding to the machine; and providing, as additional input to the Al model, the one or more additional features corresponding to the machine (in addition to discussion above, Sicconi et al., paragraph 0049 teaches “Detecting operator face contours may further include comparing a first frame of the video feed to a second from of the video feed and determining that a number of pixels exceeding a threshold amount has changed with respect to at least a parameter from the first frame to the second frame….for instance, a sample rate may be set to sample frames frequently enough to detect parameter changes consistent with motion of operator head, eyes, or mouth, motion of passengers, motion of operator hands, motion of steering wheel, or the like. Selection of frame rate may be determined using a machine-learning process; for instance, where object analysis and/or classification has been performed, as described above, to identify objects in similar video feeds, motion of such objects and rates of pixel parameter changes in video feeds may be correlated in training data derived from such video feeds, and used in any machine-learning, deep learning, and/or neural network process as described below to identify rates of pixel parameter change consistent with motion of classified objects….”).
Regarding claim 6, the method wherein the one or more additional features comprise at least one of a two-dimensional mobile device detection bounding box, a three-dimensional mobile device detection location, a standard deviation of lane position of the machine, a steering angle, first historic data on mobile device usage detection, second historic data on fixation information, or a hands on wheel signal (in addition to discussion above, Sicconi et al., paragraph 0029 teaches “At least a driving condition sensor 108 may monitor the external environment of a vehicle in a continuous fashion for detection of, as non-limiting examples, other vehicles, weather conditions, pedestrians, bikers, buildings, sidewalks, driving lanes, guardrails, stoplights, traffic conditions, and the like.”, paragraph 0032 teaches “As another non-limiting example, at least a road-facing camera 116 may detect lane lines, distance from vehicles in front, distance from objects other than vehicles, and perform scene analysis and recording.”, paragraph 0044 teaches “At least an operator sensor 124 may monitor the internal environment of a vehicle in a continuous fashion for detection of, as non-limiting examples, operator identification, steering wheel motion, orientation of operator face, orientation of operator eyes, operator eyelids closing patterns, hands position and gestures, spoken utterances and their transcription, acoustic noises in the cabin, operator head movements such as yaw, pitch, and roll, and the like…. Monitoring with the mentioned variables may be correlated with the road ahead of vehicle, left mirror, right mirror, central rearview mirror, instrument cluster, center dash, passenger seat, mobile phone in hand, or the like;”, paragraph 0046 teaches “a biometric sensor 132 may be a transducer, semiconductor, wireless device, or the like coupled with a sensor, camera, or any combination of sensors and/or cameras as described in this disclosure, which allow for operator biometric extraction including, without limitation, fingerprints, facial recognition, iris recognition, speech recognition, hand geometry, heartbeat, heart rate, breathing patterns, blood pressure, temperature, sweat level, and the like.”, paragraph 0049 teaches “..for instance, a sample rate may be set to sample frames frequently enough to detect parameter changes consistent with motion of operator head, eyes, or mouth, motion of passengers, motion of operator hands, motion of steering wheel, or the like”, paragraph 0055 teaches “For example and without limitation, training data may correlate a combination of operator historical data and historical driving condition data to outcome data including crashes, near-crashes, deviations from lanes, and/or other data indicative of negative outcomes; a supervised machine-learning process may take a combination of driving condition data and operator state data as inputs and generate as output a probability of one or more negative driving outcomes, which probability may indicate a risk level.”)
Regarding claim 7, the method wherein determining whether the operator is distracted comprises: comparing the indication of the abnormal gaze pattern to a predetermined threshold (in addition to discussion above, Sicconi et al., paragraph 0056 teaches “Each of these steps may be performed using any suitable component and/or process as described in this disclosure. As a non-limiting example, attention level may be calculated by calculation of an initial attention level, comparison to a risk level determined as described above, and determination of a degree of attention needed to operate vehicle at risk level. This may be performed, without limitation, by receiving training data correlating risk levels and/or sets of historical driving condition data and/or historical operator data as described above, and generating a machine-learning model that outputs attentiveness level, risk level, and outcome probability as described above; attentiveness level may be compared to attentiveness levels associated with a low probability of negative outcomes. For example, attentiveness level may be computed as a percentage of or portion of an attentiveness level associated with a low probability of negative outcome at a risk level matching conditions detected using at least a driving condition sensor and/or at least an operator sensor; percentage and/or proportion may be compared to a threshold level, which may be, as a non-limiting example, 85% of the attentiveness level associated with low probability of negative outcome, where failure to exceed the threshold level prevents generation of an output message as described in further detail below.”).
Regarding claim 8, the method wherein determining whether the operator is distracted comprises: identifying additional indications of the abnormal gaze pattern, wherein the additional indications of the abnormal gaze pattern represent a gaze pattern of the operator over time; and comparing the indication of the abnormal gaze pattern and the additional indications of the abnormal gaze pattern to a criterion (in addition to discussion above, Sicconi et al., paragraph 0054 teaches “With continued reference to FIG. 1, attention state module 144 may monitor attentiveness level of an operator against a personalized behavior model; personalized behavior model may be generated using machine-learning and/or neural net processes as described above, for instance utilizing operator data collected by attention state module 144 as training data. Attention state module 144 may alternatively or additionally compare attentiveness level to permissible thresholds, which may include thresholds corresponding to duration, frequency, and/or other patterns, compatible with operator attentiveness computed from a driving context. In an embodiment, if a driver is not found to be distracted but shows signs of drowsiness, system may start evaluation of driver attentiveness against user behavioral models and safety margins following the same flow used for distracted driving monitoring. Where driver is not found to be distracted nor drowsy the system may continue to observe the driver's face and hands, and iteratively performing the above steps.”, paragraph 0056).
Claim 9 is rejected for the same reason as discussed in the corresponding claim 1 above (in addition to discussion above, paragraph 0017 teaches processing units).
Claim 10 is rejected for the same reason as discussed in the corresponding claim 2 above.
Claim 11 is rejected for the same reason as discussed in the corresponding claim 3 above.
Claim 13 is rejected for the same reason as discussed in the corresponding claim 5 above.
Claim 14 is rejected for the same reason as discussed in the corresponding claim 6 above.
Claim 15 is rejected for the same reason as discussed in the corresponding claim 7 above.
Claim 16 is rejected for the same reason as discussed in the corresponding claim 8 above.
Regarding claim 17, the system wherein the system is comprised in at least one of: an in-vehicle system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system implementing one or more large language models (LLMs);a system implementing one or more language models; a system for performing one or more generative Al operations; a system using or deploying one or more inference microservices; a system that incorporates one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package (e.g., a container);a system incorporating one or more virtual machines (VMs);or a system implemented at least partially using cloud computing resources (in addition to discussion above, Sicconi et al., Abstract teaches “A system for using artificial intelligence to present geographically relevant user-specific recommendations based on vehicle operator attentiveness includes at least a transport communication device installed in a vehicle…”, paragraph 0034 teaches “used in any machine-learning, deep learning, and/or neural network process as described below to identify rates of pixel parameter change consistent with motion of classified objects.”).
Claim 18 is rejected for the same reason as discussed in the corresponding claim 1 above (in addition to discussion above, fig. 1, 116, 140 shows cameras).
Regarding claim 19, the machine wherein the body comprises a windshield, and the camera is pointing away from the windshield toward the operator of the machine (in addition to discussion above, Sicconi et al., paragraph 0032 teaches “As a non-limiting example, at least a road-facing camera 116 may be attached to the inside windshield of a vehicle oriented in the opposite direction of the vehicle operator.”, paragraph 0048 teaches “At least an operator-facing camera 116 may be mounted and/or attached to any suitable structure and/or portion of a vehicle; as a non-limiting example, at least an operator-facing camera 140 may be mounted next to a rearview mirror (attached to windshield or to body of rearview mirror)”, paragraph 0074 teaches “as a non-limiting example, road-facing camera 116 may be attached to a windshield, next to a rearview mirror's mount. Wireless connectivity may provide data transfer between camera module unit 308, operator-facing camera 140, road-facing camera 116, and/or processor 312 and a processing unit such as without limitation a smartphone. More specifically, camera module unit 308 may be mounted next to the rearview mirror (attached to windshield or to body of rearview mirror) to provide best view of an operator's face while minimizing interference with road view.”).
Regarding claim 20, the machine wherein at least one of the two or more image frames depicts the operator using a mobile device in a region overlapping with the windshield, and wherein the indication of the abnormal gaze pattern indicates that the operator is distracted (in addition to discussion above, Sicconi et al., paragraph 0074 teaches “Road-facing camera 116 may be mounted and/or attached to any suitable structure and/or portion of a vehicle; as a non-limiting example, road-facing camera 116 may be attached to a windshield, next to a rearview mirror's mount. Wireless connectivity may provide data transfer between camera module unit 308, operator-facing camera 140, road-facing camera 116, and/or processor 312 and a processing unit such as without limitation a smartphone. More specifically, camera module unit 308 may be mounted next to the rearview mirror (attached to windshield or to body of rearview mirror) to provide best view of an operator's face while minimizing interference with road view. Camera module unit 308 may contains a road-facing camera 116, an operator-facing camera 140 and a processing unit 312 to analyze and process video streams from the two cameras, and to communicate 316 (wirelessly or via USB connection) with a mobile application on a phone 230 or other processing device as described above.”, paragraph 0054 teaches “With continued reference to FIG. 1, attention state module 144 may monitor attentiveness level of an operator against a personalized behavior model; personalized behavior model may be generated using machine-learning and/or neural net processes as described above, for instance utilizing operator data collected by attention state module 144 as training data. Attention state module 144 may alternatively or additionally compare attentiveness level to permissible thresholds, which may include thresholds corresponding to duration, frequency, and/or other patterns, compatible with operator attentiveness computed from a driving context. In an embodiment, if a driver is not found to be distracted but shows signs of drowsiness, system may start evaluation of driver attentiveness against user behavioral models and safety margins following the same flow used for distracted driving monitoring. Where driver is not found to be distracted nor drowsy the system may continue to observe the driver's face and hands, and iteratively performing the above steps.”).
Regarding claim 21, the method wherein each gaze feature of the sequence of gaze features represents a gaze characteristic of the operator extracted from at least one of the two or more image frames (in addition to discussion above, Lyle et al., fig. 5-6, col. 18 lines 38-65).
The motivation for combining references has been discussed in independent claim above.
Claim 22 is rejected for the same reason as discussed in the corresponding claim 21 above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIGAR CHOWDHURY whose telephone number is (571)272-8890. The examiner can normally be reached Monday-Friday 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thai Tran can be reached at 571-272-7382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NIGAR CHOWDHURY/Primary Examiner, Art Unit 2484
Prosecution Insights