Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/22/2026 has been entered.
Detailed Action
Applicant amended claims 1, 16 and 20, canceled claims 17 and 21, added claim 21 and presented claims 1-16, 18-20 and 22 for reconsideration on 04/22/2026.
Claim Objections
Claim 1 is objected to for the following informalities.
Claim 1 is an amended claim but it is given “Previously Presented” status which is incorrect. Please correct the informalities in the claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-11, 16-19 and 22 are rejected under 35 U.S.C. 103(a) as being unpatentable over Doshi et al., "Investigating the relationships between gaze patterns, dynamic vehicle surround analysis, and driver intentions." (Doshi) in view of Satzoda et al., Pub. No.: US 2019/0265712 A1 (Satzoda).
Claim 1. Doshi teaches:
A system, comprising: a processor; and a memory storing:
an image reception module including instructions that, when executed by the processor, cause the processor to receive an image that includes a representation of an object, but lacks a representation of a human-provided indication associated with the object; (p. 888, sec. III, images from outside a car shows important objects outside a car: “Section III-B describes how the surround images are analyzed using optical flow and background motion is subtracted to produce a saliency map”; p. 889, sec. B, “Visual saliency maps are produced by extracting useful and pertinent features of the surrounding environment that may attract the driver’s attention”)
a recording reception module including instructions that, when executed by the processor, cause the processor to receive a recording that includes the representation of the human-provided indication, but lacks the representation of the object; (p. 888, sec. III, a driver’s gaze direction lacks the representation of the object: “The driver view images are processed to produce an estimate of the gaze location, as described in Section III-A”)
a relationship determination module including instructions that, when executed by the processor, cause the processor to determine, based on: information about a location of the object at a first time, the first time being when the image was produced by a camera, and (p. 888, sec. III, location of the objects captured by outside camera are shown in a saliency map: “Section III-B describes how the surround images are analyzed using optical flow and background motion is subtracted to produce a saliency map”; p. 889, sec. B, “Visual saliency maps are produced by extracting useful and pertinent features of the surrounding environment that may attract the driver’s attention”) information about a relative motion between the camera and the object between the first time and a second time, the second time being when the recording was produced, (pp. 889-890, images from outside camera showing an object at a first time, images from inside camera showing driver’s gaze direction produced in a second time, gaze direction in a second time is calibrated with images taken in a first time for identifying distraction reason: “Once the gaze direction is determined, it becomes necessary to calibrate the gaze direction with a section of the environment in the outside camera. The approximate regions corresponding to each gaze direction can be seen in the bottom of Figure 2. … Given the location of the gaze direction, along with the saliency map, it is straightforward to determine whether there was a salient object in the driver’s view. The right column of Figures 5 and 6 show this fusion step”)
an existence of a relationship between the object and the human-provided indication; and (p. 889, sec. A, images from driver-facing camera and omni-directional camera are related based on the direction of gaze and location of the objects in the surrounding images: “Once the gaze direction is determined, it becomes necessary to calibrate the gaze direction with a section of the environment in the outside camera”; p.890, sec. C, “Given a particular gaze, if there is no salient object in the region, then it is much more likely that the driver’s attention is motivated by a goal the driver had in mind”)
a database relationship establishment module including instructions that, when executed by the processor, cause the processor to cause, without a need to use textual information, information about the existence of the relationship to be stored in a database; and (see above, wherein “Once the gaze direction is determined, it becomes necessary to calibrate the gaze direction with a section of the environment in the outside camera. The approximate regions corresponding to each gaze direction can be seen in the bottom of Figure 2” where “Approximate location of glance directions superimposed on the surround map” stores a relationship between an object and a direction of gaze without a need to use textual information)
a database query module including instructions that, when executed by the processor, cause the processor to cause, based on the existence of the relationship, a database to produce, in response to a query about a subject of the human-provided indication, information about the object. (p. 888, sec. B, p. 890, sec. C, images have to be queried for identifying objects in the driver’s view: “Given the location of the gaze direction, along with the saliency map, it is straightforward to determine whether there was a salient object in the driver’s view… Given a particular gaze, if there is no salient object in the region, then it is much more likely that the driver’s attention is motivated by a goal the driver had in mind. For certain glances, there are indeed salient objects in the region”)
Doshi did not specifically disclose but Satzoda discloses a time at which the object is substantially outside a field of view of the camera by using a camera that is directed outward from the vehicle and can “be directed toward the vehicle side(s), top, bottom, rear, or any other suitable region exterior the vehicle and/or including the vehicle surroundings” as in ¶¶ 27, 77, 88: “the driver's eye gaze and/or head pose (e.g., the direction of a driver's eye gaze, the angular orientation of a driver's head, etc.) is extracted from a series of interior images (e.g., over time) (sensed by an interior facing camera, e.g., 535) that are synchronized with the exterior sensor data to infer scanning patterns from the driver. The range of the external scene can be determined from the extremities of the scanning pattern. The scanning pattern can be used to infer the regions (regions of interest) in the external scene that the user is looking at, based on the gaze direction (determined from the interior-facing camera, e.g., 535) and the camera calibration parameters (e.g., relating the interior camera 535 and exterior camera 536, such as the extrinsic matrix). These regions can optionally be used to generate a driving dataset for a vehicle (as described herein), which can include: interior images annotated for gaze, and exterior images annotated for region of interest (e.g., the region that the driver was looking at)… determining a region in a scene represented by the exterior image data that the driver ( of the driving data set) is looking at based an eye gaze of the driver extracted from the interior image data includes: determining whether the driver is looking at locations of high saliency based on detecting the driver's gaze in the interior image data”.
It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing a time at which the object is substantially outside a field of view of the camera by using a camera that can be directed to “a region in front of the vehicle” and “can alternatively be directed toward the vehicle side(s), top, bottom, rear, or any other suitable region exterior the vehicle and/or including the vehicle surroundings” as an alternative for achieving the same predictable result as Doshi.
Claims 16 and 20 are rejected under the same rationale as above.
Claim 2. The system of claim 1, wherein at least one of:
the recording is a recording of sound, or the image is a first image and the recording is a second image. (Doshi, p. 889, wherein images are related to inside and outside of a car: “The driver-facing monocular camera was mounted above the radio controls, looking at an angle toward the driver… Once the gaze direction is determined, it becomes necessary to calibrate the gaze direction with a section of the environment in the outside camera”)
Claim 3. The system of claim 2, wherein:
the first camera is a first camera, and the second image was produced by a second camera. (Doshi, pp. 888-889, wherein images are taken using inside and outside camaras: “we use separate cameras for looking in and looking out of the vehicle…The driver-facing monocular camera was mounted above the radio controls, looking at an angle toward the driver… Once the gaze direction is determined, it becomes necessary to calibrate the gaze direction with a section of the environment in the outside camera”)
Claim 4. The system of claim 3, wherein:
the first camera is at least one of:
a forward-facing camera disposed on a vehicle, or a rearward-facing camera disposed on the vehicle, and the second camera is a cabin view camera disposed on the vehicle. (Doshi, pp. 888-889, wherein images are taken using inside and outside camaras: “we use separate cameras for looking in and looking out of the vehicle…The driver-facing monocular camera was mounted above the radio controls, looking at an angle toward the driver… Once the gaze direction is determined, it becomes necessary to calibrate the gaze direction with a section of the environment in the outside camera”)
Claim 5. The system of claim 1, wherein the human-provided indication comprises at least one of:
a hand gesture, a gaze, or an audible comment. (Doshi, p. 889, sec. A., “Once the gaze direction is determined, it becomes necessary to calibrate the gaze direction with a section of the environment in the outside camera”)
Claim 6. The system of claim 5, wherein at least one of:
the hand gesture is a gesture to point in a specific direction, the gaze is in the specific direction, or the audible comment includes information that signifies the specific direction. (Doshi, p. 889, sec. A., “Once the gaze direction is determined, it becomes necessary to calibrate the gaze direction with a section of the environment in the outside camera”)
Claim 7. The system of claim 5, wherein at least one of:
the hand gesture signifies an opinion of a human that produced the hand gesture, the gaze signifies an opinion of a human that produced the gaze, or the audible comment signifies an opinion of a human that produced the audible comment. (Doshi, p. 890, a gaze signifies an opinion/goal of a driver: “Given a particular gaze, if there is no salient object in the region, then it is much more likely that the driver’s attention is motivated by a goal the driver had in mind”)
Claim 8. The system of claim 1, wherein:
the human-provided indication signifies a specific direction, a location of the object at the second time is in the specific direction from a human that produced the human-provided indication, and (Doshi, pp. 889-890, secs. A-C: the gaze is in a specific direction and an object is located in the gaze direction: “To detect motion, we use the Lucas-Kanade dense optical flow algorithm, comparing the previous frame with the current frame…. The “saliency maps” on the bottom of Figure 4 are generated by taking the magnitude of the foreground motion map…Given the location of the gaze direction, along with the saliency map, it is straightforward to determine whether there was a salient object in the driver’s view”)
the instructions to determine the existence of the relationship include instructions to determine that the location of the object at the first time corresponds to the representation of the object included in the image. (p. 889-890, secs. A-C: the detected object is in motion and images are corresponding to a first time, a second time, etc., and a gaze is captured in a direction and location of the object in motion: “To detect motion, we use the Lucas-Kanade dense optical flow algorithm, comparing the previous frame with the current frame…. The “saliency maps” on the bottom of Figure 4 are generated by taking the magnitude of the foreground motion map…Given the location of the gaze direction, along with the saliency map, it is straightforward to determine whether there was a salient object in the driver’s view”)
Claim 9. The system of claim 8, wherein the memory further stores at least one of:
a hand gesture module including instructions that, when executed by the processor, cause the processor to cause, in response to the human-provided indication including a hand gesture, an operation of a hand gesture technique, the hand gesture technique including:
operating gesture recognition technology to determine that the hand gesture is a gesture to point in the specific direction, and producing a hand gesture vector in the specific direction, an origin of the hand gesture vector being a hand arranged to produce the hand gesture, or a gaze module including instructions that, when executed by the processor, cause the processor to cause, in response to the human-provided indication including a gaze, an operation of a gaze technique, the gaze technique including: operating eye point-of-gaze tracking technology to determine that a point of gaze of an eye is in the specific direction, and producing a gaze vector in the specific direction, an origin of the gaze vector being the eye. (Doshi, p. 888, sec. A, p. 889, sec. A, a technology is used for detecting the direction of a gaze: “More precise gaze estimates can be derived from eye gaze detectors [19]. NHTSA has most recently conducted studies of Driver Workload Metrics [14], including eye gaze as a proxy for driver workload… Ideally, a properly designed stereo or monocular eye gaze system could provide robust data. Head pose estimators [16] could also be used to determine gaze direction, though not as precisely…Nine different gaze locations were derived from the procedure described in [14] as relevant to the task at hand. Sample images from each of these cases can be seen in Figure 2”)
Claim 10. The system of claim 1, wherein the relationship information includes:
the recording that includes the representation of the human-provided indication, and (Doshi, p. 889, sec. A, gaze is the human provided indication captured by inside camera and object in the direction of the gaze is captured by the outside camera: “Nine different gaze locations were derived from the procedure described in [14] as relevant to the task at hand. Sample images from each of these cases can be seen in Figure 2. Once the gaze direction is determined, it becomes necessary to calibrate the gaze direction with a section of the environment in the outside camera. The approximate regions corresponding to each gaze direction can be seen in the bottom of Figure 2”)
at least one of the image that includes the representation of the object or another image that includes the representation of the object. (p. 890, sec. C, gaze direction and a salient object is obtained/stored for determining the driver’s distraction: “Given the location of the gaze direction, along with the saliency map, it is straightforward to determine whether there was a salient object in the driver’s view. The right column of Figures 5 and 6 show this fusion step”)
Claim 11. The system of claim 10, wherein the memory further stores an image annotation module including instructions that, when executed by the processor, cause the processor to cause the at least one of the image that includes the representation of the object or the other image that includes the representation of the object to be annotated with supplemental information, the supplemental information being based on at least one of information signified by the human provided indication or information produced concurrently with a production of the human provided indication. (Doshi, p. 890, sec. C, wherein an image including a salient object is further supplemented by the human provided indication, e.g., gaze direction in fusion step: “Given the location of the gaze direction, along with the saliency map, it is straightforward to determine whether there was a salient object in the driver’s view. The right column of Figures 5 and 6 show this fusion step”)
Claim 18. The method of claim 16 wherein the second time is after the first time. (Doshi, pp. 889-890, secs. A-C: the detected object is in motion and images are corresponding to a first time, a second time, etc., and a gaze is captured in a direction and location of the object in motion might be captured before or after the first or a second time: “To detect motion, we use the Lucas-Kanade dense optical flow algorithm, comparing the previous frame with the current frame…. The “saliency maps” on the bottom of Figure 4 are generated by taking the magnitude of the foreground motion map…Given the location of the gaze direction, along with the saliency map, it is straightforward to determine whether there was a salient object in the driver’s view”)
Claim 19. The method of claim 16 wherein the second time is before the first time. (Doshi, pp. 889-890, secs. A-C: the detected object is in motion and images are corresponding to a first time, a second time, etc., and a gaze is captured in a direction and location of the object in motion might be captured before or after the first or a second time: “To detect motion, we use the Lucas-Kanade dense optical flow algorithm, comparing the previous frame with the current frame…. The “saliency maps” on the bottom of Figure 4 are generated by taking the magnitude of the foreground motion map…Given the location of the gaze direction, along with the saliency map, it is straightforward to determine whether there was a salient object in the driver’s view”)
Claim 22. The system of claim 1, wherein the textual information comprises information input using at least one of a keyboard interface or speech-to-text technology. (Doshi, the feature has no impact on claim 1 implementation because claim 1 recites: “without a need to use textual information”)
Claims 12-15 are rejected under 35 U.S.C. 103(a) as being unpatentable over Doshi and Satzoda, as applied to claims 1 and 10 above in view of Dharia et al., Pub. No.: 2022/0188560 A1 (Dharia).
Claim 12. Doshi as modified taught the system of claim 10; Doshi as modified did not explicitly teach but Dharia teaches: wherein the memory further stores an object recognition and classification module including instructions that, when executed by the processor, cause the processor to cause the object to be recognized and classified. (Dharia, ¶¶ 24-25, 51, 129, wherein identified objects are classified: “the detecting of the at least one object of interest may include determining… a classification of the at least one object of interest”)
Doshi, p. 888, sec. B, implicitly discloses the feature by classifying an object based on its attractiveness: “The analysis of gaze patterns can be aided by developing a robust saliency detector which can determine the relative attractiveness of objects in a scene. The potential structure of these saliency maps varies based on the motivation and context of the scene”. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing wherein the memory further stores an object recognition and classification module including instructions that, when executed by the processor, cause the processor to cause the object to be recognized and classified because doing so would further provide for recognizing and classifying object of interest explicitly for achieving the same predictable result.
Claim 13. The system of claim 10, wherein the memory further stores an image transformation module including instructions that, when executed by the processor, cause the processor to produce, based on the image that includes the representation of the object, the other image that includes the representation of the object, the other image being a transformation of the image. (Doshi, pp. 889-890, secs. B-C, images are transformed for selecting images that includes salient objects: “The optical flow vectors for each pixel are accumulated over several hundred frames, sampled over the last ten seconds of data. The average of these detections becomes our “Background Motion Map”. Figure 3 shows an example of the background motion vectors in the scene, superimposed on the scene itself. Then in order to detect “foreground” motion that could be of interest to the driver, we can collect the most current motion vectors and subtract out the Background Motion Map. Examples of this foreground motion detection can be seen in Figures 5 and 6… to be more robust we can average the motion vectors over the previous second of data, and then subtract the background motion to obtain our foreground motion map”; Dharia. ¶ 21, “These 5 frames can be used to perform image processing tasks such as, for example, reducing noise, image enhancement, image sharpening, object detection, object tracking, and so forth”)
Claim 14. The system of claim 10, wherein:
the image that includes the representation of the object is a member of a set of images that include the representation of the object, and the memory further stores:
an image quality measurement module including instructions that, when executed by the processor, cause the processor to determine an image, of the set of images, in which a measurement of an image quality of the object is a greatest value; and an image designation module including instructions that, when executed by the processor, cause the processor to designate the image, of the set of images, in which the measurement of the image quality of the object is the greatest value, as the other image that includes the representation of the object. (Dharia. ¶¶ 100-104, wherein high-quality image data is produced: “The selected mode of operation may be based on the initial image data captured, and may be selected to improve the quality of future images captured in the environment represented by the initial image data… the first image sensor(s) 602 may capture additional image data, which may have higher quality (e.g., has a better exposure, magnification, white balance, etc.) than the initial image data captured”)
Claim 15. The system of claim 10, wherein the memory further stores a map augmentation module including instructions that, when executed by the processor, cause the processor to cause useful map information to be included in a map of a vicinity of a location of the object, the useful map information including at least one of: the at least one of the image that includes the representation of the object or the other image that includes the representation of the object, or supplemental information, the supplemental information being based on at least one of information signified by the human-provided indication or information produced concurrently with a production of the human-provided indication. (Doshi, wherein the generated saliency maps include captured objects in a gaze direction: pp. 889-890, secs. B-C, “Visual saliency maps are produced by extracting useful and pertinent features of the surrounding environment that may attract the driver’s attention…. The saliency map produced from the surround analysis shows that there are only salient objects on the other side of the scene (on the driver’s left). We may thereby conclude, that the gaze shift which occurred in that window of time was associated with a specific goal of the driver. In this case, a highway situation, that goal would most likely be a lane change, and so an intent prediction system could more confidently predict the upcoming lane change”)
Response to Amendment and Arguments
Applicant’s arguments with respect to applied references have been considered but are moot in view of the new ground of rejections as provided above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHSEN ALMANI whose telephone number is (571)270-7722. The examiner can normally be reached on M-F, 9:00 to 5:00.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann J. Lo can be reached on 571-272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHSEN ALMANI/Primary Examiner, Art Unit 2159
Prosecution Insights