DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 were previously pending. Claims 1, 4-11, 13-14, and 20 have been amended. No claims have been newly added or cancelled. Thus, claims 1-20 remain pending and have been examined in this application.
Examiner's Note
Examiner has cited particular paragraphs/columns and line numbers or figures in the
references as applied to the claims below for the convenience of the applicant. Although the
specified citations are representative of the teachings in the art and are applied to the specific
limitations within the individual claim, other passages and figures may apply as well. It is
respectfully requested from the applicant, in preparing the responses, to fully consider the
references in their entirety as potentially teaching all or part of the claimed invention, as well as
the context of the passage as taught by the prior art or disclosed by the examiner. Applicant is
reminded that the Examiner is entitled to give the broadest reasonable interpretation to the
language of the claims. Furthermore, the Examiner is not limited to Applicant's definition which is not specifically set forth in the disclosure.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1, 10, and 20 recite “a neural network computer processor… an eye tracker computer processor… a gaze distribution computer processor… a distance distribution computer processor” and this aspect appears to be new matter. Paragraph [0039] of the instant application reads “Vehicles have become computationally advanced and equipped with multiple microcontrollers, sensors, processors, and control systems, including for example, autonomous vehicle and advanced driver assistance systems (AV/ADAS) such as adaptive cruise control, automated parking, automatic brake hold, automatic braking, evasive steering assist, lane keeping assist, adaptive headlights, backup assist, blind spot detection, cross traffic alert, local hazard alert, and rear automatic braking may depend on information obtained from cameras and sensors on a vehicle.” Although paragraph [0039] broadly states that vehicles are equipped with multiple microcontrollers, processors, and control systems, nothing in the instant application specifically indicates that the neural network, eye tracker, gaze distribution, and distance distribution are performed on separate computer processors. The steps could be performed on a single processor with different modules.
Claims 1, 10, and 20 recite “wherein the inward looking camera is selected from the group consisting of an optical camera, an infrared imaging device, a light-emitting-diode (LED) device, and an ultrasound sensor, and combinations thereof” and this aspect appears to be new matter. According to paragraph [0051] of the instant application, the group consisting of an optical camera, an infrared imaging device, a light-emitting-diode (LED) device, and an ultrasound sensor are considered as other types of imaging devices (alternatives). However, nothing in the instant application points to a combination of these devices or cameras.
Claims 2-9 and 11-19 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as being dependent on rejected claims 1 and 10 and for failing to cure the deficiencies listed above.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 10, and 20 recite “a neural network computer processor… an eye tracker computer processor… a gaze distribution computer processor… a distance distribution computer processor” and this aspect appears to be new matter. Paragraph [0039] of the instant application reads “Vehicles have become computationally advanced and equipped with multiple microcontrollers, sensors, processors, and control systems, including for example, autonomous vehicle and advanced driver assistance systems (AV/ADAS) such as adaptive cruise control, automated parking, automatic brake hold, automatic braking, evasive steering assist, lane keeping assist, adaptive headlights, backup assist, blind spot detection, cross traffic alert, local hazard alert, and rear automatic braking may depend on information obtained from cameras and sensors on a vehicle.” Although paragraph [0039] broadly states that vehicles are equipped with multiple microcontrollers, processors, and control systems, nothing in the instant application specifically indicates that the neural network, eye tracker, gaze distribution, and distance distribution are performed on separate computer processors. The steps could be performed on a single processor with different modules. The metes and bounds of the claim limitation are vague and ill-defined, rendering the claim indefinite. As best understood, the claims will be interpreted broadly such that one or more processors perform the steps for the neural network, eye tracker, gaze distribution, and distance distribution.
Claims 1, 10, and 20 recite “wherein generating the predicted gaze distribution by the neural network computer processor does not require explicit detection and identification of objects in the surrounding environment” and it is unclear what is being conveyed by this limitation. Specifically, what is meant by “explicit detection and identification” in the context of the claim? How can one differentiate between what is explicit and what is not in the context of the claim? Paragraphs [0041-0042] mention explicit detection but do not explain what is meant by this. How are elements in a scene be detected and identified in a non-explicit way? The metes and bounds of the claim limitation are vague and ill-defined, rendering the claim indefinite. As best understood (see Figs. 5B and 6 in the instant application), the claims will be interpreted broadly such that predicted gaze distribution is based on detection and identification of objects in the surrounding environment, the detection and identification does not necessarily require detecting and identifying an exact boundary/periphery of each element.
Claims 1, 10, and 20 recite “wherein the inward looking camera is selected from the group consisting of an optical camera, an infrared imaging device, a light-emitting-diode (LED) device, and an ultrasound sensor, and combinations thereof.” It is unclear how a camera can be an LED device or an ultrasound sensor. According to paragraph [0051] of the instant application, the group consisting of an optical camera, an infrared imaging device, a light-emitting-diode (LED) device, and an ultrasound sensor are considered as other types of imaging devices (alternatives). Furthermore, nothing in the instant application points to a combination of these devices or cameras. The metes and bounds of the claim limitation are vague and ill-defined, rendering the claim indefinite. As best understood, the claims will be interpreted broadly such that an inward looking imaging device includes an optical camera, an infrared imaging device, a light-emitting-diode (LED) device, or an ultrasound sensor.
Claims 8 and 17 recite “a warning indication” and it is unclear if this is referring to the same warning indication as introduced in claims 1 and 10, respectively or a different one. The metes and bounds of the claim limitation are vague and ill-defined, rendering the claim indefinite. As best understood, the claims will be interpreted broadly such that claims 8 and 17 are referring to the same warning indication as in claims 1 and 10, respectively.
Claim 18 recites “a vehicle action” and it is unclear if this is referring to the same vehicle action as introduced in claim 10 or a different one. The metes and bounds of the claim limitation are vague and ill-defined, rendering the claim indefinite. As best understood, the claims will be interpreted broadly such that claim 18 is referring to the same vehicle action as in claim 10.
Claims 2-9 and 11-19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being dependent on rejected claims 1 and 10 and for failing to cure the deficiencies listed above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hutchings (US 2020/0198644 A1) in view of Xia (“Predicting Driver Attention in Critical Situations”) and Kanzawa (US 2020/0160487 A1).
Regarding claim 1, Hutchings discloses a system of estimating of a driver state based on eye gaze comprising: an outward looking camera, situated in a vehicle, configured to capture and send a first video stream of a surrounding environment to a computer processor ([0010, 0030-0032, 0044, 0083] – An expected gaze direction or pattern of the driver based on a planned route of the vehicle and objects in an external environment of the vehicle… Different models may be used to determine and/or allocate where a driver should be looking given the visual stimuli in the field of view… The perception system 172 also includes one or more components for detecting objects external to the vehicle such as other vehicles, obstacles in the roadway, traffic signals, signs, trees, etc. For example, the perception system 172 may include one or more LIDAR sensors, sonar or other acoustical devices, radar units, cameras (e.g., video or still, visible light or infrared) and/or any other detection devices that record data which may be processed by computing devices 110.); the computer processor, situated in the vehicle, configured to generate a predicted gaze based on the first video stream ([0010, 0030-0032, 0044, 0083] – An expected gaze direction or pattern of the driver based on a planned route of the vehicle and objects in an external environment of the vehicle… Different models may be used to determine and/or allocate where a driver should be looking given the visual stimuli in the field of view. Such normative models of visual scanning may be based on recorded attention allocation behavior of human drivers (using fatigue data as well as attentive driving data) but may also be derived from driving environment statistics. The models may use machine learning, or other statistical techniques, to learn the statistical properties of such data in order to determine the appropriate gaze direction or pattern of a driver, e.g., an assist driver, in a given driving scenario. A normative scanning model based on human data may be established by training the model to reproduce the visual scanning patterns associated with attentive drivers in similar scenarios.); an inward looking camera, situated in the vehicle, configured to capture and send a second video stream of a face of a driver to an eye tracker computer processor, wherein the eye tracker computer processor, based on the second video stream, is configured to extract a plurality of actual gaze directions and generate an actual gaze ([0010, 0030-0032, 0082] – The actual gaze direction (or pattern of gaze direction) may be detected through use of cameras within the compartment of the vehicle… the actual gaze direction or pattern as measured by video in the vehicle.); and a distance computer processor, situated in the vehicle, configured to generate a distance measure based on a difference between the predicted gaze and the actual gaze, wherein if the distance measure exceeds a threshold, then a vehicle action is generated ([0025, 0030-0032, 0082-0084] – Additional action… The expected gaze direction or pattern may then be used as a benchmark and compared to the actual gaze direction or pattern… deviations between the expected gaze direction or pattern of a driver based on the normative scanning model may be compared to the driver’s actual gaze direction or pattern as measured by a video to determine whether the driver is properly allocating his attention or gaze direction.); wherein the vehicle action comprises: (a) generating a warning indication and/or (b) generating an alert to the driver to take control of the vehicle ([0025] - sounding a distinctive alarm intended to awake even a sleeping driver); wherein the vehicle action further comprises: (a) providing a haptic feedback signal to the driver ([0025] – vibrating the driver’s seat) and/or (b) providing an evasive steering assist; and wherein the inward looking camera is selected from the group consistent of an optical camera ([0030,0048, 0082] – video or still camera capturing digital images), an infrared imaging device, a light-emitting-diode (LED) device, and an ultrasound sensor.
It would have been obvious to one of ordinary skill in the art before the effective filing date to have incorporated both of Hutchings’ vehicle actions into a single embodiment with a reasonable expectation of success. For example, providing both an alarm and vibrating the driver’s seat is more likely than just providing one of those actions to alert the driver to improve safety. Combining these teachings into a single embodiment would have yielded predictable results.
Hutchings does not appear to explicitly disclose a neural network computer processor; an expected gaze distribution; an actual gaze distribution; a distance distribution computer processor; wherein generating the predicted gaze distribution by the neural network computer processor does not require explicit detection and identification of objects in the surrounding environment; wherein the neural network computer processor is configured to identify one or more areas of obstructions; wherein each of the one or more areas of obstructions has a continuous shape.
Xia, in the same field of endeavor, teaches the following limitations: a neural network computer processor (page 661 and 667 – neural model); an expected gaze distribution (pages 660, 665, and 667-668 –driver attention map prediction); an actual gaze distribution (pages 660, 665, and 667-668 – human driver attention map); a distance distribution computer processor (pages 660, 665, and 667-668 – KL divergence); wherein generating the predicted gaze distribution by the neural network computer processor does not require explicit detection and identification of objects in the surrounding environment (pages 659-660 – We introduce a new protocol that uses crowd-sourced driving videos containing interesting events and makes multi-focus driver attention maps by averaging gazes collected from multiple human observers in lab with great accuracy (Fig. 1). We will refer to this protocol as the in-lab driver attention collection protocol. We show that data collected with our protocol reliably reveal where an experienced driver should look and can serve as a substitute for data collected with the in-car protocol.); wherein the neural network computer processor is configured to identify one or more areas of obstructions (pages 660 and 663 – attention maps show the multiple regions that demand the driver’s attention).
Since Hutchings discloses using that different models may be used to determine and/or allocate where a driver should be looking given the visual stimuli in the field of view and that the models may use machine learning, or other statistical techniques, to learn the statistical properties of such data in order to determine the appropriate gaze direction or pattern of a driver (Hutchings – [0031]), it would have been obvious to one of ordinary skill in the art before the effective filing date to have incorporated the teachings of Xia into the invention of Hutchings with a reasonable expectation of success. The motivation of doing so is to train the model to more accurately predict driver attention data (Xia – pages 658-659). A neural network is a specific type of machine learning that can handle more complex problems than traditional machine learning models and can also learn and adapt to new information to improve performance over time, making them more robust for real-world applications where patterns might change over time. Xia also teaches that the Kullback-Leibler divergence (KL divergence) is a commonly used metric for attention map prediction (Xia - page 667).
Kanzawa, in the same field of endeavor, teaches the following limitations: wherein the neural network computer processor is configured to identify one or more areas of obstructions; wherein each of the one or more areas of obstructions has a continuous shape (Figs. 7-9B, [0045-0046, 0049-0050, 0095] – 3D boundary information 710/810/910 having a continuous shape as shown corresponding to pedestrian 310… neural network).
It would have been obvious to one of ordinary skill in the art before the effective filing date to have incorporated the teachings of Kanzawa into the invention of Hutchings with a reasonable expectation of success. The motivation of doing so is that by forming the boundary and projecting the position based on speed and heading would improve safety for avoiding the pedestrian or other obstacle by taking into account future positions of the obstacle.
Regarding claim 2, Hutchings discloses wherein the surrounding environment comprises a forward view (Fig. 2, [0044-0046]).
Regarding claim 3, Hutchings discloses wherein the outward looking camera is forward looking (Fig. 2, [0044-0046]).
Regarding claim 4, Hutchings discloses wherein the computer processor is configured to detect a road junction, a target vehicle, and/or a pedestrian from the first video stream ([0044] – The perception system 172 also includes one or more components for detecting objects external to the vehicle such as other vehicles, obstacles in the roadway, traffic signals, signs, trees, etc… the sensors of the perception system may detect objects and their characteristics such as location, orientation, size, shape, type (for instance, vehicle, pedestrian, bicyclist, etc.)).
Hutchings does not appear to explicitly disclose the neural network computer processor.
Xia, in the same field of endeavor, teaches the following limitations: the neural network computer processor (page 661 and 667 – neural model).
The motivation to combine Hutchings and Xia is the same as in the rejection of claim 1.
Regarding claim 5, Hutchings discloses wherein the outward looking camera is forward facing (Fig. 2, [0044-0046]).
Regarding claim 6, Hutchings discloses wherein if the distance measure exceeds the threshold, then the driver is deemed to be inattentive ([0030-0032, 0082-0084] – driver attentiveness and fatigue… The expected gaze direction or pattern may then be used as a benchmark and compared to the actual gaze direction or pattern… deviations between the expected gaze direction or pattern of a driver based on the normative scanning model may be compared to the driver’s actual gaze direction or pattern as measured by a video to determine whether the driver is properly allocating his attention or gaze direction.).
Regarding claim 7, Hutchings discloses wherein if the distance measure is less than the threshold, then the driver is deemed to be attentive ([0030-0032, 0082-0084] – driver attentiveness and fatigue… The expected gaze direction or pattern may then be used as a benchmark and compared to the actual gaze direction or pattern… deviations between the expected gaze direction or pattern of a driver based on the normative scanning model may be compared to the driver’s actual gaze direction or pattern as measured by a video to determine whether the driver is properly allocating his attention or gaze direction).
Regarding claim 8, Hutchings discloses wherein if the distance measure exceeds the threshold, then a warning indication is generated ([0025, 0030-0032, 0082-0084] – additional action may include alerting a dispatcher or sounding a distinctive alarm… The expected gaze direction or pattern may then be used as a benchmark and compared to the actual gaze direction or pattern… deviations between the expected gaze direction or pattern of a driver based on the normative scanning model may be compared to the driver’s actual gaze direction or pattern as measured by a video to determine whether the driver is properly allocating his attention or gaze direction).
Regarding claim 9, Hutchings does not appear to explicitly disclose wherein the distance distribution computer processor is configured to generate the distance measure using a Kullback-Leibler divergence or a Jensen-Shannon divergence.
Xia, in the same field of endeavor, teaches the following limitations: wherein the distance distribution computer processor is configured to generate the distance measure using a Kullback-Leibler divergence or a Jensen-Shannon divergence (pages 660, 665, and 667-668 – KL divergence).
The motivation to combine Hutchings and Xia is the same as in the rejection of claim 1.
Regarding claim 10, all the limitations have been analyzed in view of claim 1, respectively, and it has been determined that claim 10 does not teach or define any new limitations beyond those previously recited in claim 1; therefore, claim 10 is also rejected over the same rationale as claim 1.
Regarding claim 11, all the limitations have been analyzed in view of claim 2, respectively, and it has been determined that claim 11 does not teach or define any new limitations beyond those previously recited in claim 2; therefore, claim 11 is also rejected over the same rationale as claim 2.
Regarding claim 12, all the limitations have been analyzed in view of claim 3, respectively, and it has been determined that claim 12 does not teach or define any new limitations beyond those previously recited in claim 3; therefore, claim 12 is also rejected over the same rationale as claim 3.
Regarding claim 13, all the limitations have been analyzed in view of claim 4, respectively, and it has been determined that claim 13 does not teach or define any new limitations beyond those previously recited in claim 4; therefore, claim 13 is also rejected over the same rationale as claim 4.
Regarding claim 14, all the limitations have been analyzed in view of claim 5, respectively, and it has been determined that claim 14 does not teach or define any new limitations beyond those previously recited in claim 5; therefore, claim 14 is also rejected over the same rationale as claim 5.
Regarding claim 15, all the limitations have been analyzed in view of claim 6, respectively, and it has been determined that claim 15 does not teach or define any new limitations beyond those previously recited in claim 6; therefore, claim 15 is also rejected over the same rationale as claim 6.
Regarding claim 16, all the limitations have been analyzed in view of claim 7, respectively, and it has been determined that claim 16 does not teach or define any new limitations beyond those previously recited in claim 7; therefore, claim 16 is also rejected over the same rationale as claim 7.
Regarding claim 17, all the limitations have been analyzed in view of claim 8, respectively, and it has been determined that claim 17 does not teach or define any new limitations beyond those previously recited in claim 8; therefore, claim 17 is also rejected over the same rationale as claim 8.
Regarding claim 18, Hutchings discloses further comprising generating a vehicle action if the distance measure exceeds the threshold ([0025, 0030-0032, 0082-0084] – additional action may include playing loud music, vibrating the driver’s seat, or pulling the vehicle over… The expected gaze direction or pattern may then be used as a benchmark and compared to the actual gaze direction or pattern… deviations between the expected gaze direction or pattern of a driver based on the normative scanning model may be compared to the driver’s actual gaze direction or pattern as measured by a video to determine whether the driver is properly allocating his attention or gaze direction).
Regarding claim 19, all the limitations have been analyzed in view of claim 9, respectively, and it has been determined that claim 19 does not teach or define any new limitations beyond those previously recited in claim 9; therefore, claim 19 is also rejected over the same rationale as claim 9.
Regarding claim 20, all the limitations have been analyzed in view of claims 10-19, and it has been determined that claim 20 does not teach or define any new limitations beyond those previously recited in claims 10-19; therefore, claim 20 is also rejected over the same rationale as claims 10-19. Although Hutchings does not appear to explicitly disclose generating a warning and generating a vehicle action (i.e., Hutchings appears to teach these as possible alternatives) it would have been obvious to one of ordinary skill in the art before the effective filing date to have incorporated both actions because it would be important to warn the driver that the vehicle is initiating pulling over the vehicle, because if the driver wakes up during the countermeasure of pulling over the vehicle they may be frightened and unaware of what is happening. Alerting the driver before pulling over the vehicle improves safety and comfort for the driver, and also would yield predictable results.
Response to Arguments
Applicant’s arguments, see pages 11-14 filed 10/24/2025, with respect to the previous 35 U.S.C. 101 rejections have been fully considered and are persuasive. The previous 35 U.S.C. 101 rejections have been withdrawn.
Applicant's arguments filed 10/24/2025 with respect to the previous prior art rejections have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAITLIN MCCLEARY whose telephone number is (703)756-1674. The examiner can normally be reached Monday - Friday 10:00 am - 7:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Navid Z Mehdizadeh can be reached at (571) 272-7691. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.R.M./Examiner, Art Unit 3669
/NAVID Z. MEHDIZADEH/Supervisory Patent Examiner, Art Unit 3669
Prosecution Insights