DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of claims
Claims 5 and 8 are cancelled.
Claims 1,3 and 9 are amended.
Claims 1-5, 6-7, 9-11 are pending.
Response to arguments
With respect to Applicant’s remarks filed on 08/05/2026; Applicant's “Amendments and Remarks” have been fully considered. Applicant’s remarks will be addressed in sequential order as they were presented.
Arguments:
Argument should overcome 35 U.S.C. 101 rejection.
Regarding claim 1, Hampiholi’s wearable device continuously streams image data to the in-vehicle computing system. Hampiholi does not teach selectively acquiring or not acquiring data from the mobile device based on the driver’s state, Matsumura likewise does not teach this feature as well.
Hampholi does not teach -not acquiring sensor data from the mobile terminal when the driver is determined to be in driving state.
Neither Matsumura nor Hampholi teaches ease the judgmental criteria when judging the driver is in the awake state – regarding claim 3.
Office response:
Arguments overcome 35 U.S.C. 101 rejection.
Hampiholi teaches para[0003] that in vehicle computing system receive image data from the mobile device via the external device interface. In response to determining that the driver state indicates that the driver is distracted, then receive vehicle data from vehicle system via the inter-vehicle system. Though in response to the driver’s state, receive vehicle data from vehicle system via the inter-vehicle system for determining the destruction severality but one ordinary skilled consider the selective acquiring is similar to the claim 1. Matsumura teaches conditional acquiring such as in para[0071]-[0076] that when it is estimated that the reliability of the determination result of the wakefulness determination process based on the first image is low during the execution of the automatic driving control, It is determined whether or not the person DR is in an awakened state. Hereinafter, the information acquired by the mobile terminal 110 will be referred to as "terminal-side information". Furthermore, the process of determining whether or not the driver DR is in an awake state using the terminal-side information is referred to as "specific process" (it is similar to the re-examination).
Regarding claim 3, please see the new mapping.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4, 6-7,9 and 11 are rejected under 35 U.S.C. 103 as being obvious over JP2022145990 A to Matsumura (herein after “Matsumura”).
Regarding claim 1, Matsuma discloses A driver monitoring device of a vehicle comprising; (see Matsumura automatic driving device)
a processor programmed to (see Matsumura ECU 10), during automatic driving of the vehicle at a first automatic driving level:
acquire an image of a driver of the vehicle from an imaging device of the vehicle, (see Matsumura first imaging unit 76; para [0008] An arousal state determination system in one or more embodiments includes a first imaging unit (76) mounted in a vehicle (VA) for imaging a driver (DR) of the vehicle)
determine whether the driver of the vehicle is in a drive ready state in which the driver can perform driving operations based on the image acquired from the imaging device of the vehicle (see Matsumura para[0008] A control device (10) that executes an arousal state determination process for determining whether the driver is in an awake state based on a first image that is an image of the driver;) and
in response to determining that the driver is not in the drive ready state based on the acquired image (See Matsumura para [0071] (Overview of System Operation) As shown in FIG. 4, the driver DR may operate the portable terminal 110 while the ECU 10 is executing automatic driving control. The driver DR operates the mobile terminal 110 while looking down. Accordingly, the driver monitor camera 76 acquires the first image 500 shown in FIG. The driver DR's eyes in the first image 500 appear to be closed, para [0072] Therefore, when it is estimated that the reliability of the determination result of the wakefulness determination process based on the first image is low during the execution of the automatic driving control, the ECU 10 uses the information acquired by the mobile terminal 110 to perform driving. It is determined whether or not the person DR is in an awakened state. Hereinafter, the information acquired by the mobile terminal 110 will be referred to as "terminal-side information", figures 4 and 5 ), acquire sensor data from a mobile terminal of the driver (see Matsumura para[0072] he information acquired by the mobile terminal 110 will be referred to as "terminal-side information", terminal-side information (that is the second image).)
and re- determine whether the driver is in the drive ready state based on the acquired sensor data (Matsumura teaches conditional acquiring such as in para[0071]-[0076] , para[0074] Furthermore, the process of determining whether or not the driver DR is in an awake state using the terminal-side information is referred to as "specific process" (it is similar to the re-examination).), wherein the sensor data includes at least one of image data from a camera of the mobile terminal (See Matsumura terminal side information, para[0082] The ECU 10 receives the second image 700 shown in FIG. 7 from the mobile terminal 110 as terminal-side information), input data from a touch sensor of the mobile terminal (See Matsumura [0109] the ECU 10 may perform specific processing in step 906 of FIG. 9 as follows. For example, when the driver DR is asleep, the finger of the driver DR may accidentally touch the display 201), and acceleration data from an acceleration sensor of the mobile terminal, (See Matsumura para [0067] Further, the mobile terminal 110 has an acceleration sensor 203 inside)
in response to determining that the driver is in the drive ready state based on the acquired image, not acquire the sensor data from the mobile terminal,(See Matsumura para [0092] When the specific processing condition is not satisfied, the ECU 10 makes a "No" determination in step 902, proceeds to step 903, and executes the awakening state determination processing based on the first image. Next, at step 904, the ECU 10 determines whether or not the driver DR is in an awakened state. If the driver DR is awake, the ECU 10 makes a “Yes” determination in step 904 and proceeds to step 995) and
issue a warning to the driver if it is judged that the driver is not in the drive ready state. (see Matsumura para [0063] When the ECU 10 determines that the driver is not in an awake state, the ECU 10 causes the notification device 51 to perform predetermined attention calling processing. Specifically, the ECU 10 uses the meter ECU 50 to sound a buzzer and display a warning lamp on the display device.).
Matsumura does not explicitly mentioned “doing re-examination using mobile terminal picture”, ”display a warning notification to the driver when the driver is in sleep(non-driving state)”,
However, Matsumura teaches when the reliability of the determination result of the wakefulness determination process based on the first image is low, the ECU 10 uses the information acquired by the mobile terminal 110 to perform driving, which one can consider the selective acquiring of the mobile terminal image when the driver determined as non-driving state. Matsumua teaches in figure 9, In response to the determination the driver is in awake state (see below figure with red boxed part), not acquire mobile terminal image. Additionally, Matsumura’s display unit can display a notification when the driver is not in driving state.
PNG
media_image1.png
576
494
media_image1.png
Greyscale
Figure ( Matsumura (JP2022145990 A) -US2022301323 A1 US publication of the prior art)
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine Matsumura’s awakening state determination system considering selective acquiring information by acquiring the mobile terminal image when it is determined that the reliability of the determination result based on the first image from the vehicle terminal is low(meaning driver is non-driving state) and displaying the notification when the driver falls asleep in order to allow to provide a technique capable of accurately determining whether the driver is in an awake state even when the driver is not facing forward (See para[0007]).
Regarding claim 2, Matsumura discloses wherein the driver information includes an image generated by a camera (see Matsumura driver monitor camera ; para[0024] example of an image (first image) acquired by the driver monitor camera in the situation of FIG. 4; ) mounted at the terminal or information input by the driver to the terminal (see Matsumura para[[0008] An arousal state determination system in one or more embodiments includes a first imaging unit (76) mounted in a vehicle (VA) for imaging a driver (DR) of the vehicle) .
Regarding claim 4, Matsumura discloses wherein the processor is configured to judge whether the driver is in the drive ready state based on the image by a predetermined judgment criteria and, judge whether the driver is in an awake state based on the driver information when judging that the driver is not in the drive ready state based on the image (see Matsumura para[0008] The control device uses terminal-side information, which is information acquired by the mobile terminal, when a predetermined condition that is satisfied when the reliability of the determination result of the arousal state determination process is estimated to be low is satisfied. , and is configured to determine whether the driver is in the awake state. ), and, invalidate a result of judgment based on the image when judging that the driver is in the awake state (see Matsumura para[0062] On the other hand, when the eye closure rate is greater than the predetermined threshold, the ECU 10 determines that the degree of wakefulness of the driver is less than the predetermined degree (that is, the driver is not in an alert state).
Regarding claim 6, Matsumura discloses wherein the processor is configured to limit or prohibit operation of the terminal by the driver when level 2 or lower automated driving is being performed at the vehicle (see Matsumura para [0085] After that, the driver DR drives the vehicle VA. A vehicle VA is traveling on an expressway at a vehicle speed SPD equal to or lower than the speed threshold SPDth. In this situation, the driver DR presses the automatic operation switch 81 . The ECU 10 determines that the conditions for executing automatic driving are satisfied, and starts automatic driving control (level 3) (step 802). The ECU 10 transmits an activation instruction signal to the mobile terminal 110 (step 803). The activation instruction signal is a signal that instructs activation of the monitoring application 330).
Regarding claim 7, Matsumura discloses wherein the terminal is a smart phone (see Matsumura para [0006] For example, assume that the driver operates a mobile terminal (for example, a smart phone) during execution of automatic driving control; see Matsumura para [0064] (Structure of Portable Terminal) The portable terminal 110 is a device carried by the driver, and in this example, is the smart phone shown in FIG) .
Regarding claim 9, Matsumura discloses A non-transitory recording medium having (see Matsumura automatic driving device) recorded thereon a computer program, the computer program causing a computer to: (see Matsumura ECU 10), acquire an image of a driver from an imaging device of a vehicle; (see Matsumura first imaging unit 76; para [0008] An arousal state determination system in one or more embodiments includes a first imaging unit (76) mounted in a vehicle (VA) for imaging a driver (DR) of the vehicle)
determine whether the driver of the vehicle is in a drive ready state in which the driver can perform driving operations based on the image acquired from the imaging device of the vehicle (see Matsumura para[0008] A control device (10) that executes an arousal state determination process for determining whether the driver is in an awake state based on a first image that is an image of the driver;)
in response to determining that the driver is not in the drive ready state based on the acquired image (See Matsumura para [0071] (Overview of System Operation) As shown in FIG. 4, the driver DR may operate the portable terminal 110 while the ECU 10 is executing automatic driving control. The driver DR operates the mobile terminal 110 while looking down. Accordingly, the driver monitor camera 76 acquires the first image 500 shown in FIG. The driver DR's eyes in the first image 500 appear to be closed, para [0072] Therefore, when it is estimated that the reliability of the determination result of the wakefulness determination process based on the first image is low during the execution of the automatic driving control, the ECU 10 uses the information acquired by the mobile terminal 110 to perform driving. It is determined whether or not the person DR is in an awakened state. Hereinafter, the information acquired by the mobile terminal 110 will be referred to as "terminal-side information", figures 4 and 5 ), acquire sensor data from a mobile terminal of the driver (see Matsumura para[0072] he information acquired by the mobile terminal 110 will be referred to as "terminal-side information", terminal-side information (that is the second image) and
re-determine whether the driver is in the drive ready state based on the acquired sensor data (Matsumura teaches conditional acquiring such as in para[0071]-[0076] , para[0074] Furthermore, the process of determining whether or not the driver DR is in an awake state using the terminal-side information is referred to as "specific process" (it is similar to the re-examination) wherein the sensor data includes at least one of image data from a camera of the mobile terminal (See Matsumura terminal side information, para[0082] The ECU 10 receives the second image 700 shown in FIG. 7 from the mobile terminal 110 as terminal-side information), input data from a touch sensor of the mobile terminal (See Matsumura [0109] the ECU 10 may perform specific processing in step 906 of FIG. 9 as follows. For example, when the driver DR is asleep, the finger of the driver DR may accidentally touch the display 201) and acceleration data from an acceleration sensor of the mobile terminal(See Matsumura para [0067] Further, the mobile terminal 110 has an acceleration sensor 203 inside) and
in response to determining that the driver is in the drive ready state based on the acquired image, not acquire the sensor data from the mobile terminal ,(See Matsumura para [0092] When the specific processing condition is not satisfied, the ECU 10 makes a "No" determination in step 902, proceeds to step 903, and executes the awakening state determination processing based on the first image. Next, at step 904, the ECU 10 determines whether or not the driver DR is in an awakened state. If the driver DR is awake, the ECU 10 makes a “Yes” determination in step 904 and proceeds to step 995) and
issue a warning to the driver if it is judged that the driver is not in the drive ready state. (see Matsumura para [0063] When the ECU 10 determines that the driver is not in an awake state, the ECU 10 causes the notification device 51 to perform predetermined attention calling processing. Specifically, the ECU 10 uses the meter ECU 50 to sound a buzzer and display a warning lamp on the display device.).
However, Matsumura teaches when the reliability of the determination result of the wakefulness determination process based on the first image is low, the ECU 10 uses the information acquired by the mobile terminal 110 to perform driving, which one can consider the selective acquiring of the mobile terminal image when the driver determined as non-driving state. Matsumua teaches in figure 9, In response to the determination the driver is in awake state (see below figure with red boxed part), not acquire mobile terminal image. Matsumura’s display unit can display a notification when the driver is not in driving state.
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine Matsumura’s awakening state determination system considering selective acquiring information by acquiring the mobile terminal image when it is determined that the reliability of the determination result based on the first image from the vehicle terminal is low(meaning driver is non-driving state) and displaying the notification when the driver falls asleep in order to allow to provide a technique capable of accurately determining whether the driver is in an awake state even when the driver is not facing forward (See para[0007]).
Regarding claim 10, Matsumura teaches wherein the processor is further programmed to send a control signal to the mobile terminal to prohibit operation of the mobile terminal by the driver during automatic driving at a second automatic driving level. (See Matsumura para [0119] In another example, the mobile terminal 110 may prohibit activation of "an application in which only the first operation is used". Since an application that uses only relatively simple operations is not activated, it is possible to prevent the driver DR from falling asleep.).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine Matsumura’s awakening state determination system considering selective acquiring information by acquiring the mobile terminal image when it is determined that the reliability of the determination result based on the first image from the vehicle terminal is low(meaning driver is non-driving state) and displaying the notification when the driver falls asleep in order to allow to provide a technique capable of accurately determining whether the driver is in an awake state even when the driver is not facing forward (See para[0007]).
Regarding claim 11, Matsumura discloses A system comprising:
a vehicle including an imaging device configured to capture an image of a driver (See Matsumura para[0008] a vehicle (VA) for imaging a driver (DR) of the vehicle) and a first processor programmed to control the vehicle to execute automatic driving at a plurality of automatic driving levels including a first automatic driving level; (See Matsumura para[0004] The level of automatic driving control (so-called automatic driving level) is divided into a plurality of levels 0 to 5 by SAE (Society of Automotive Engineers). In recent years, level 3 automatic driving control has begun to be put into practical use. Level 3 is a level at which the automatic driving device executes all driving operations under a specific environment. This is the level at which the driver is required to perform the driving operation. Therefore, automatic driving control of level 3 may be called "conditional automatic driving control.")
a mobile terminal of the driver (See Matsumura para[0006] For example, assume that the driver operates a mobile terminal (for example, a smart phone) during execution of automatic driving control. The driver operates the mobile terminal while looking down.), the mobile terminal including a plurality of sensors including a camera configured to generate image data (See Matsumura para [0020] In one or more embodiments, the wakefulness determination system further includes a first acceleration sensor (72) mounted on the vehicle and acquiring information on acceleration occurring in the vehicle as first acceleration information. The mobile terminal includes a second acceleration sensor (203) that acquires information on acceleration occurring in the mobile terminal as second acceleration information.), a touch sensor configured to generate input data (See Matsumura para [0065] The mobile terminal 110 has a display 201 . The display 201 is a touch panel display, and can detect a touch operation on the display surface (operation surface) of the display 201) , and an acceleration sensor configured to generate acceleration data (See Matsumura para[0020] a first acceleration sensor (72) mounted on the vehicle and acquiring information on acceleration occurring in the vehicle as first acceleration information. The mobile terminal includes a second acceleration sensor (203) that acquires information on acceleration occurring in the mobile terminal as second acceleration information.); and
a driver monitoring device including a second processor programmed to, during automatic driving of the vehicle at the first automatic driving level (See Matsumura para[0004] The level of automatic driving control (so-called automatic driving level) is divided into a plurality of levels 0 to 5 by SAE (Society of Automotive Engineers).)
acquire the image of the driver from the imaging device, (see Matsumura first imaging unit 76; para [0008] An arousal state determination system in one or more embodiments includes a first imaging unit (76) mounted in a vehicle (VA) for imaging a driver (DR) of the vehicle)
determine whether the driver is in a drive ready state in which the driver can perform driving operations based on the acquired image (see Matsumura para[0008] A control device (10) that executes an arousal state determination process for determining whether the driver is in an awake state based on a first image that is an image of the driver;)
in response to determining that the driver is not in the drive ready state based on the acquired image (See Matsumura para [0071] (Overview of System Operation) As shown in FIG. 4, the driver DR may operate the portable terminal 110 while the ECU 10 is executing automatic driving control. The driver DR operates the mobile terminal 110 while looking down. Accordingly, the driver monitor camera 76 acquires the first image 500 shown in FIG. The driver DR's eyes in the first image 500 appear to be closed, para [0072] Therefore, when it is estimated that the reliability of the determination result of the wakefulness determination process based on the first image is low during the execution of the automatic driving control, the ECU 10 uses the information acquired by the mobile terminal 110 to perform driving. It is determined whether or not the person DR is in an awakened state. Hereinafter, the information acquired by the mobile terminal 110 will be referred to as "terminal-side information", figures 4 and 5 ), acquire sensor data from the mobile terminal (see Matsumura para[0072] he information acquired by the mobile terminal 110 will be referred to as "terminal-side information", terminal-side information (that is the second image) and
re-determine whether the driver is in the drive ready state based on the acquired sensor data (Matsumura teaches conditional acquiring such as in para[0071]-[0076] , para[0074] Furthermore, the process of determining whether or not the driver DR is in an awake state using the terminal-side information is referred to as "specific process" (it is similar to the re-examination), wherein the sensor data includes at least one of the image data from the camera of mobile terminal (See Matsumura terminal side information, para[0082] The ECU 10 receives the second image 700 shown in FIG. 7 from the mobile terminal 110 as terminal-side information), the input data from the touch sensor (See Matsumura [0109] the ECU 10 may perform specific processing in step 906 of FIG. 9 as follows. For example, when the driver DR is asleep, the finger of the driver DR may accidentally touch the display 201), and acceleration data from an acceleration sensor of the mobile terminal, (See Matsumura para [0067] Further, the mobile terminal 110 has an acceleration sensor 203 inside)
and the acceleration data from the acceleration sensor(See Matsumura para [0067] Further, the mobile terminal 110 has an acceleration sensor 203 inside)
in response to determining that the driver is in the drive ready state based on the acquired image, not acquire the sensor data from the mobile terminal ,(See Matsumura para [0092] When the specific processing condition is not satisfied, the ECU 10 makes a "No" determination in step 902, proceeds to step 903, and executes the awakening state determination processing based on the first image. Next, at step 904, the ECU 10 determines whether or not the driver DR is in an awakened state. If the driver DR is awake, the ECU 10 makes a “Yes” determination in step 904 and proceeds to step 995) and
issue a warning to the driver if it is judged that the driver is not in the drive ready state (see Matsumura para [0063] When the ECU 10 determines that the driver is not in an awake state, the ECU 10 causes the notification device 51 to perform predetermined attention calling processing. Specifically, the ECU 10 uses the meter ECU 50 to sound a buzzer and display a warning lamp on the display device.).
However, Matsumura teaches when the reliability of the determination result of the wakefulness determination process based on the first image is low, the ECU 10 uses the information acquired by the mobile terminal 110 to perform driving, which one can consider the selective acquiring of the mobile terminal image when the driver determined as non-driving state. Matsumua teaches in figure 9, In response to the determination the driver is in awake state (see below figure with red boxed part), not acquire mobile terminal image. Matsumura’s display unit can display a notification when the driver is not in driving state.
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine Matsumura’s awakening state determination system considering selective acquiring information by acquiring the mobile terminal image when it is determined that the reliability of the determination result based on the first image from the vehicle terminal is low(meaning driver is non-driving state) and displaying the notification when the driver falls asleep in order to allow to provide a technique capable of accurately determining whether the driver is in an awake state even when the driver is not facing forward (See para[0007]).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatented over Matsumura in view of JP 2023082251 A to Sakurai (herein after “Sakurai”).
However, Matsumura does not teach explicitly T programmed to judge whether the driver is in the drive ready state based on the image acquired from the imaging device of the vehicle by a predetermined judgment criteria, judge whether the driver is in an awake state based on the sensor data, and ease the judgment criteria when judging the driver is in the awake state. Nevertheless, Sakurai same field of endeavor teaches The driver monitoring device according to wherein the processor is further programmed to judge whether the driver is in the drive ready state based on the image acquired from the imaging device of the vehicle by a predetermined judgment criteria, judge whether the driver is in an awake state based on the sensor data, and ease the judgment criteria when judging the driver is in the awake state (See Sakurai Furthermore, when it is determined that the driver is not awake (S20), the conditions for determining whether the driver has let go are relaxed (S40), making it easier to determine that the driver has let go.).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine Matsumura’s awakening state determination system considering selective acquiring information by acquiring the mobile terminal image when it is determined that the reliability of the determination result based on the first image from the vehicle terminal is low(meaning driver is non-driving state) and displaying the notification when the driver falls asleep with Sakurai’s ease the predetermined condition in order to allow to provide a technique capable of accurately determining whether the driver is in an awake state even when the driver is not facing forward (See para[0007]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAZIA AFRIN whose telephone number is (703)756-1175. The examiner can normally be reached Monday-Friday 7:30-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott A Browne can be reached at 5712700151. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NAZIA AFRIN/ Examiner, Art Unit 3666
/SCOTT A BROWNE/ Supervisory Patent Examiner, Art Unit 3666
