DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/08/2026 has been entered.
Foreign Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application JP2020-008647, filed on 01/22/2020.
Status of Claims
Claims 1-14 of U.S. Application No. 17/791,345 filed on 06/08/2026 have been examined. Claims 1, 4-5, and 14 have been amended. Claims 15 and 16 is are cancelled.
Response to Arguments
Regarding 35 U.S.C. 112, Applicants arguments are unpersuasive. Claim 16’s rejection is withdrawn due to the claim being cancelled. However, claims 1, 4, 5, 14 remain rejected under 112(a) for failing to meet the written description requirement. Applicant’s specification does not describe or provide support for choosing between different alert types based on different vehicle driving modes. See more details in the rejection below.
Regarding 35 U.S.C. 103 Rejection, Applicant’s argument regarding the limitation “wherein the first mode has a first alert, and the second mode has a second alert…” is moot since the amendment changes the scope of the invention. New reference Andres and previously used reference Sobhany teach these limitations:
Strubbe fails to disclose wherein the first mode has a first alert, and the second mode has a second alert using one of a warning beep and flashing light.
However, Andres teaches wherein the first mode has a first alert ([0058] “The takeover alert may include an audible description of the upcoming adverse event”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Andres’ teaching of outputting a verbal alert during autonomous driving. One would be motivated with a reasonable expectation of success to warn a driver during autonomous driving in order keep the driver attentive in case they need to take over into manual mode ([0058] “allowing the driver to further prepare for encountering the adverse event.”).
Further, Sobhany teaches the second mode has a second alert using one of a warning beep and flashing light ([abstract] “driver is not attentive to driving the vehicle, the processor causes the vehicle to output an alert to the driver or to automatically control a driving function of the vehicle.” One type of alert occurs during manual driving when driver is not paying attention: [0036] “if the driver is determined to be not attentive … the vehicle output can be an alert that notifies the driver to re-engage … the alert can include … a beep … a flashing light”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of outputting at least an alert over a speaker in the car when it is determined that the driver is inattentive during manual driving. One would be motivated with a reasonable expectation of success to warn a driver in order prevent fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Applicant argues the prior art deos not teach the speaker outputs first conversation data corresponding to the first alert when the vehicle is under the first mode.
However, Andres teaches wherein the speaker outputs first conversation data corresponding to the first alert when the vehicle is under the first mode ([0058] “The takeover alert may include an audible description of the upcoming adverse event”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Andres’ teaching of outputting a verbal alert during autonomous driving. One would be motivated with a reasonable expectation of success to warn a driver during autonomous driving in order keep the driver attentive in case they need to take over into manual mode ([0058] “allowing the driver to further prepare for encountering the adverse event.”).
Applicant argues the prior art does not teach the first conversation data contains a self-counseling function that is configured to talk about a subject regarding user's biological information in order to alert the user that the user’s attention is decreased at the time.
However, Penilla teaches the first conversation data contains a self-counseling function that is configured to talk about a subject regarding user's biological information in order to alert the user that the user’s attention is decreased at the time. ([0281] “vehicle system can deduct that the user is getting tired, and may need to rest or find a hotel. The vehicle can say, for example: ‘Are you ok?’ ‘Are you tired?’”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Penilla’s teaching of detecting the state of the driver including sleepiness and engaging in verbal dialog with the user. One would be motivated with reasonable expectation of success to engage in dialog with a sleepy driver in order to rouse the driver or ask if they would like to go somewhere to remedy the sleepiness (Penilla [0147] “if threshold percentage of conditions exist … provide verbal feedback/output to the user … “Would you like me to open the windows for some fresh air?”, “Big Coffee down the road has coffee on sale,” “SLEEP-E hotel is in 10 miles, book hotel room?”. In some embodiments, if no passenger is present, and driver fails to respond, sound audio noise to wake driver up, turn AC on to stimulate driver, etc.”).
Applicant argues the prior art do not suggest nor reasonably suggest the amended claim limitations and Examiner respectfully disagrees. The examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 4, 5, and 14 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claims 1, 4, 5, and 14, there is no support in Applicant’s specification for the following limitations in the above claims:
“wherein the first mode has a first alert, and the second mode has a second alert using one of a warning beep and flashing light,” and “the speaker outputs first conversation data corresponding to the first alert when the vehicle is under the first mode,”
There is no support in Applicant’s specification for deciding between first alert and second alert types based on different drive modes of the vehicle. There is generic support for a beep or flashing light and for conversing with the driver, but they are general alerts, and not determined based on the drive mode of the vehicle.
Applicant cites specification paragraphs [0008] and [0044] as support for the amendments, however, these paragraphs still only provide generic support for these alert types and do not provide support for deciding which to use based on different drive modes. There are no other paragraphs in Applicant’s specification that support choosing between different alert types based on different drive modes of the vehicle.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Strubbe et al. (US 6,731,307 B1), in view of Sicconi et al. (US 20190213429 A1), Sobhany (US 20200239007 A1), Andres (US 20200319635 A1), and Penilla et al. (US 20200152197 A1) hereafter Strubbe, Sicconi, Sobhany, Andres, and Penilla, respectively.
Regarding claim 1, Strubbe discloses A data processing system comprising:
a biosensor ([column 21, lines 53, 58-59] “The other UI classifiers 235 may include … galvanic skin response sensor, heart rate sensor,”);
a conversation data generation unit ([FIG. 2] response data generator 445);
an operation unit ([FIG. 1] Controller 100);
a speaker ([FIG. 1] Speaker 114);
and a microphone ([FIG. 1] Microphone 112),
wherein the conversation data generation unit comprises a classifier ([FIG. 3] Mood/personality classifier 290) that has learned first information of a user ([column 22 lines 44-47] “The mood/personality classifier 290 receives signals from the various classifiers and processes these to generate a mood/personality state signal. The mood/personality classifier 290 may be a trained neural network” [column 22 lines 51-53] “The personality signal, preferably, is the result of many observations of behavior and would tend to persist over time.”),
wherein the biosensor detects second information of the user ([column 10, lines 63-65] “A user's gaze direction is determined by two factors: the orientation of the head, and the orientation of the eyes.” [column 21, lines 58-59] “galvanic skin response sensor, heart rate sensor”),
wherein the first conversation data is generated by the conversation data generation unit generates first conversation data on the basis of the first information and the second information ([column 24 lines 60-65] “Referring now also to FIG. 4, the response generator 415 receives the mood/personality state vector and parsed reply data from the mood/personality classifier 290 and input parser 410 respectively. The response generator 415 also receives the environment/user state signal and events signal from the event/class processor 207.” [column 24 line 67 – column 25 line 4] “The response generator 415 selects a response from the response data store 440 based on the parsed reply from the user, the mood/ personality state, the environment/user state,” [column 20, lines 32-35] “the response generator 415 takes state information from the classifiers 405, such as the mood of the user, the user's attention level, personality, interests, etc., and generates a response.”),
wherein the microphone obtains second conversation data from the user and outputs the second conversation data to the classifier ([column 21 lines 1-2] “The audio input 245, which may be received by a microphone” [column 31 lines 10-13] “The audio input 245 is applied again to the speech to text converter 215 and the audio classifier 210.” [FIGs 3 and 6] audio input 245),
wherein the classifier updates the first information with use of the second conversation data ([column 21 lines 7-10] “The audio classifier 210 then generates a current state information signal which it applies to both a mood/personality classifier 290” the mood/personality classifier is updated with the signal generated by the audio classifier using the microphone detecting speech from the user. [column 30 lines 29-31] “The mood/personality classifier 290 stores personality data regarding the user and has detected, from previous conversations, the user's mood.”).
Strubbe fails to disclose wherein the data processing system in configured to communicate with a vehicle connected with wireless communication or wire communication.
However, Sicconi teaches the data processing system in configured to communicate with a vehicle connected with wireless communication or wire communication ([0178] “The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.” [0055] “a processing unit 4033 to analyze and process video streams from the two cameras, and to communicate 405 (wirelessly or via USB connection) with a mobile application on a phone 409.” [0058] “A Bluetooth connected speakerphone 703 is used to position microphone and speaker on a visor, close to the driver. A laptop 707 provides computation to run context analysis, combine it with the features extracted by the smart camera to decide about driver's attention margin and provide audio feedback when necessary. An optional WiFi-connected smartphone 711 provides telemetry data, map/routing info, cloud services (weather, traffic), audio/video recording capabilities and speech recognition and synthesis for dialog interaction with the driver.” The sensors in the vehicle may communicate data to the processing system, which may wired or wirelessly connected to the vehicle’s sensors and processed via smartphone, laptop, or even remote server.).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sicconi’s teaching of wired and/or wireless connection between the user data and the processing system. One would be motivated with a reasonable expectation of success to connect user data to the wired or wireless processing system in order to allow for secure process and storage backup (Sicconi [0123] “Secure (encrypted) synchronization allows the smartphone to transfer EDR content (live or historic) to the cloud for further processing and storage backup”).
Strubbe fails to disclose where the vehicle has a first mode and a second mode.
However, Sobhany teaches the vehicle has a first mode and a second mode ([0018] “the vehicle control system 115 may engage or disengage the partially autonomous mode upon satisfaction of a triggering condition. For example, if the attentiveness of the driver falls below a specified threshold while the vehicle is operated in a fully manual mode, the vehicle control system 115 can engage the partially autonomous self-driving mode.”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of a first and second mode in the ego vehicle. One would be motivated with a reasonable expectation of success to use a first and second mode in a vehicle in order to either alert the driver to regain attention or take over control of the vehicle since lack of attention may result in potentially fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe fails to disclose wherein the first mode is a mode with a semi-automatic driving feature turned on that automatically control the vehicle driving by the vehicle itself, and the second mode is a mode with the semi-automatic driving feature turned off, wherein the semi-automatic driving feature includes automatic tracking drive.
However, Sobhany teaches wherein the first mode is a mode with a semi-automatic driving feature turned on that automatically control the vehicle driving by the vehicle itself, and the second mode is a mode with the semi-automatic driving feature turned off, wherein the semi-automatic driving feature includes automatic tracking drive ([0018] “the vehicle control system 115 may control a lane assist function, maintaining a lateral position of the vehicle 110 in a marked lane on a road, while the driver controls the speed of the vehicle … if the attentiveness of the driver falls below a specified threshold while the vehicle is operated in a fully manual mode, the vehicle control system 115 can engage the partially autonomous self-driving mode” both a manual mode where the tracking would be disabled, and a semi-autonomous mode where some tracking is enabled.).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of at least two driving modes that include manual control from the driver and at least partial autonomous control of the vehicle. One would be motivated with a reasonable expectation of success to use these two modes in a vehicle in order to either alert the driver to regain attention or take over control of the vehicle since lack of attention in a manual mode may result in potentially fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe fails to disclose wherein the first mode has a first alert, and the second mode has a second alert using one of a warning beep and flashing light.
However, Andres teaches wherein the first mode has a first alert ([0058] “The takeover alert may include an audible description of the upcoming adverse event”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Andres’ teaching of outputting a verbal alert during autonomous driving. One would be motivated with a reasonable expectation of success to warn a driver during autonomous driving in order keep the driver attentive in case they need to take over into manual mode ([0058] “allowing the driver to further prepare for encountering the adverse event.”).
Further, Sobhany teaches the second mode has a second alert using one of a warning beep and flashing light ([abstract] “driver is not attentive to driving the vehicle, the processor causes the vehicle to output an alert to the driver or to automatically control a driving function of the vehicle.” One type of alert occurs during manual driving when driver is not paying attention: [0036] “if the driver is determined to be not attentive … the vehicle output can be an alert that notifies the driver to re-engage … the alert can include … a beep … a flashing light”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of outputting at least an alert over a speaker in the car when it is determined that the driver is inattentive during manual driving. One would be motivated with a reasonable expectation of success to warn a driver in order prevent fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe discloses wherein the speaker outputs the first conversation data ([column 20 lines 39-43] “the response generator 415 selects appropriate data from the response data store 440 and signals the output user interface 425 to output synthetic speech corresponding to the response.”),
Strubbe fails to explicitly disclose discloses wherein the speaker outputs the first conversation data corresponding to the first alert when the vehicle is under the first mode, and the second information means that the user is distracted ([column 20, lines 32-35] “the response generator 415 takes state information from the classifiers 405, such as the mood of the user, the user's attention level, personality, interests, etc., and generates a response.”).
Similarly, Sicconi also fails to explicitly teach the speaker outputs the first conversation data corresponding to the first alert when the vehicle is under the first mode, and the second information means that the user is distracted ([0174] “This is particularly useful in a contained environment like a car cabin (especially in a self-driving or autonomous vehicle)” [0033] “The present invention … helps with transition to self-driving vehicles to assist in smooth transfer of driving control from vehicle to driver and vice versa.” [0034] “the benefits of the present … makes car-driver hand over in self driving vehicles tight and safe.).
However, Andres teaches wherein the speaker outputs first conversation data corresponding to the first alert when the vehicle is under the first mode ([0058] “The takeover alert may include an audible description of the upcoming adverse event”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Andres’ teaching of outputting a verbal alert during autonomous driving. One would be motivated with a reasonable expectation of success to warn a driver during autonomous driving in order keep the driver attentive in case they need to take over into manual mode ([0058] “allowing the driver to further prepare for encountering the adverse event.”).
Further, Sobhany teaches first information being user preferences ([0029] “The user account database 210 can store data related to user accounts, such as user identification information (e.g., name, age, sex, birthday, address, VIP status, key user status, preferences, preferred language, vehicle(s) owned, greeting name, channel, talking points (e.g., favorite sports team), or geographic region of residence), account numbers, authorized users associated with one or more accounts, login credentials, or known biometric data associated with the user.”) and second information being biological information indicating user attention ([0013] “The sensors 112 each measure a parameter associated with a driver of the vehicle. The measured parameter can be any parameter related to the driver's state of attentiveness, including parameters that describe a position of at least a portion of the driver's body in the vehicle, parameters that quantify or qualify an expression on the driver's face, or parameters that measure a direction or rate of change of direction of the driver's gaze.”), and the second information means that the user is distracted (Sobhany teaches a plurality of embodiments, one having semi-autonomous driving as default: [0018] “vehicle control system 115 controls one or more driving functions while the driver concurrently controls one or more other driving functions. For example, the vehicle control system 115 may control a lane assist function, maintaining a lateral position of the vehicle 110 in a marked lane on a road, while the driver controls the speed of the vehicle. This partially autonomous self-driving mode may be active whenever the vehicle is operated” [0027] “the attention determination module 215 determines that the driver is inattentive based on a set of parameters and causes the vehicle to output an alert in response.”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of verbally alerting the driver of a partially autonomous vehicle when their attention is low. One would be motivated with reasonable expectation of success to alert the driver in order to rectify inattentiveness ([0009] “A vehicle monitors whether a driver is attentive to driving the vehicle and causes outputs to rectify driver inattentiveness.”).
Strubbe fails to disclose the first conversation data contains a self-counseling function that is configured to talk about a subject regarding user's biological information in order to alert the user that the user’s attention is decreased at the time.
However, Penilla teaches the first conversation data contains a self-counseling function that is configured to talk about a subject regarding user's biological information in order to alert the user that the user’s attention is decreased at the time ([0281] “vehicle system can deduct that the user is getting tired, and may need to rest or find a hotel. The vehicle can say, for example: ‘Are you ok?’ ‘Are you tired?’”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Penilla’s teaching of detecting the state of the driver including sleepiness and engaging in verbal dialog with the user. One would be motivated with reasonable expectation of success to engage in dialog with a sleepy driver in order to rouse the driver or ask if they would like to go somewhere to remedy their sleepiness (Penilla [0147] “if threshold percentage of conditions exist … provide verbal feedback/output to the user … “Would you like me to open the windows for some fresh air?”, “Big Coffee down the road has coffee on sale,” “SLEEP-E hotel is in 10 miles, book hotel room?”. In some embodiments, if no passenger is present, and driver fails to respond, sound audio noise to wake driver up, turn AC on to stimulate driver, etc.”).
Regarding claim 2, Strubbe fails to disclose A driver assistance system comprising the data processing system according to claim 1, wherein the user is a driver.
However, Sicconi teaches a driver assistance system comprising a data processing system, wherein the user is a driver ([0015] “a driver assistant system” which includes similar components as the system in Strubbe, but in the embodiment of a driver assist system for a user who is a driver of the vehicle.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Strubbe with the teachings from Sicconi in order to utilize the system taught by Strubbe in a vehicle setting where the user is a driver. One of ordinary skill would, with a reasonable expectation of success, would be motivated to use conversational data and sensors measuring the driver state to ensure the driver is in an attentive state and can drive the car safely.
Regarding claim 3, Strubbe discloses A data processing device comprising the data processing system according to claim 1 ([column 19 lines 59-60] “a conversation simulator operates as a program running on a controller 100” where the conversation simulator is the system and it runs on the controller, a device.).
Regarding claim 14, Strubbe discloses A data processing device comprising:
a communication device ([FIG. 2] response data generator 445);
a biosensor ([column 21, lines 53, 58-59] “The other UI classifiers 235 may include … galvanic skin response sensor, heart rate sensor,”);
an operation unit ([FIG. 1] Controller 100);
a classifier to classify conversation data ([column 31 lines 10-13] “The audio input 245 is applied again to the speech to text converter 215 and the audio classifier 210.” [FIGs 3 and 6] audio input 245);
a speaker ([FIG. 1] Speaker 114); and
a microphone ([FIG. 1] Microphone 112),
wherein the classifier has learned first information of a user ([FIG. 3] Mood/personality classifier 290. [column 22 lines 44-47] “The mood/personality classifier 290 receives signals from the various classifiers and processes these to generate a mood/personality state signal. The mood/personality classifier 290 may be a trained neural network” [column 22 lines 51-53] “The personality signal, preferably, is the result of many observations of behavior and would tend to persist over time.”),
wherein the biosensor is configured to detect second information of the user ([column 24 lines 60-65] “Referring now also to FIG. 4, the response generator 415 receives the mood/personality state vector and parsed reply data from the mood/personality classifier 290 and input parser 410 respectively. The response generator 415 also receives the environment/user state signal and events signal from the event/class processor 207.” [column 24 line 67 – column 25 line 4] “The response generator 415 selects a response from the response data store 440 based on the parsed reply from the user, the mood/ personality state, the environment/user state,” where first information is the mood state and the environment state can include the heart rate monitor or skin sensor.),
wherein the microphone is configured to obtain second conversation data from the user and output the second conversation data to the classifier ([column 21 lines 1-2] “The audio input 245, which may be received by a microphone” [column 31 lines 10-13] “The audio input 245 is applied again to the speech to text converter 215 and the audio classifier 210.” [FIGs 3 and 6] audio input 245), and
wherein the classifier is configured to update the first information with use of the second conversation data ([column 21 lines 7-10] “The audio classifier 210 then generates a current state information signal which it applies to both a mood/personality classifier 290” the mood/personality classifier is updated with the signal generated by the audio classifier using the microphone detecting speech from the user. [column 30 lines 29-31] “The mood/personality classifier 290 stores personality data regarding the user and has detected, from previous conversations, the user's mood.”).
Strubbe fails to disclose wherein the communication device is configured to communicate a vehicle connected with wireless communication.
However, Sicconi teaches the data processing system in configured to communicate a vehicle connected with wireless communication or wire communication ([0178] “The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.” [0055] “a processing unit 4033 to analyze and process video streams from the two cameras, and to communicate 405 (wirelessly or via USB connection) with a mobile application on a phone 409.” [0058] “A Bluetooth connected speakerphone 703 is used to position microphone and speaker on a visor, close to the driver. A laptop 707 provides computation to run context analysis, combine it with the features extracted by the smart camera to decide about driver's attention margin and provide audio feedback when necessary. An optional WiFi-connected smartphone 711 provides telemetry data, map/routing info, cloud services (weather, traffic), audio/video recording capabilities and speech recognition and synthesis for dialog interaction with the driver.” The sensors in the vehicle may communicate data to the processing system, which may wired or wirelessly connected to the vehicle’s sensors and processed via smartphone, laptop, or even remote server.).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sicconi’s teaching of wired and/or wireless connection between the user data and the processing system. One would be motivated with a reasonable expectation of success to connect user data to the wired or wireless processing system in order to allow for secure process and storage backup ([0123] “Secure (encrypted) synchronization allows the smartphone to transfer EDR content (live or historic) to the cloud for further processing and storage backup”).
Strubbe fails to disclose where the vehicle has a first mode and a second mode.
However, Sobhany teaches the vehicle has a first mode and a second mode ([0018] “the vehicle control system 115 may engage or disengage the partially autonomous mode upon satisfaction of a triggering condition. For example, if the attentiveness of the driver falls below a specified threshold while the vehicle is operated in a fully manual mode, the vehicle control system 115 can engage the partially autonomous self-driving mode.”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of a first and second mode in the ego vehicle. One would be motivated with a reasonable expectation of success to use a first and second mode in a vehicle in order to either alert the driver to regain attention or take over control of the vehicle since lack of attention may result in potentially fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe fails to disclose wherein the first mode is a mode with a semi-automatic driving feature turned on that automatically control the vehicle driving by the vehicle itself, and the second mode is a mode with the semi-automatic driving feature turned off, wherein the semi-automatic driving feature includes automatic tracking drive.
However, Sobhany teaches wherein the first mode is a mode with a semi-automatic driving feature turned on that automatically control the vehicle driving by the vehicle itself, and the second mode is a mode with the semi-automatic driving feature turned off, wherein the semi-automatic driving feature includes automatic tracking drive ([0018] “the vehicle control system 115 may control a lane assist function, maintaining a lateral position of the vehicle 110 in a marked lane on a road, while the driver controls the speed of the vehicle … if the attentiveness of the driver falls below a specified threshold while the vehicle is operated in a fully manual mode, the vehicle control system 115 can engage the partially autonomous self-driving mode” both a manual mode where the tracking would be disabled, and a semi-autonomous mode where some tracking is enabled.).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of at least two driving modes that include manual control from the driver and at least partial autonomous control of the vehicle. One would be motivated with a reasonable expectation of success to use these two modes in a vehicle in order to either alert the driver to regain attention or take over control of the vehicle since lack of attention in a manual mode may result in potentially fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe fails to disclose wherein the first mode has a first alert, and the second mode has a second alert using one of a warning beep and flashing light.
However, Andres teaches wherein the first mode has a first alert ([0058] “The takeover alert may include an audible description of the upcoming adverse event”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Andres’ teaching of outputting a verbal alert during autonomous driving. One would be motivated with a reasonable expectation of success to warn a driver during autonomous driving in order keep the driver attentive in case they need to take over into manual mode ([0058] “allowing the driver to further prepare for encountering the adverse event.”).
Further, Sobhany teaches the second mode has a second alert using one of a warning beep and flashing light ([abstract] “driver is not attentive to driving the vehicle, the processor causes the vehicle to output an alert to the driver or to automatically control a driving function of the vehicle.” One type of alert occurs during manual driving when driver is not paying attention: [0036] “if the driver is determined to be not attentive … the vehicle output can be an alert that notifies the driver to re-engage … the alert can include … a beep … a flashing light”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of outputting at least an alert over a speaker in the car when it is determined that the driver is inattentive during manual driving. One would be motivated with a reasonable expectation of success to warn a driver in order prevent fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe discloses wherein the speaker outputs the first conversation data ([column 20 lines 39-43] “the response generator 415 selects appropriate data from the response data store 440 and signals the output user interface 425 to output synthetic speech corresponding to the response.”),
Strubbe fails to explicitly disclose discloses wherein the speaker outputs the first conversation data corresponding to the first alert when the vehicle is under the first mode, and the second information means that the user is distracted ([column 20, lines 32-35] “the response generator 415 takes state information from the classifiers 405, such as the mood of the user, the user's attention level, personality, interests, etc., and generates a response.”).
Similarly, Sicconi also fails to explicitly teach the speaker outputs the first conversation data corresponding to the first alert when the vehicle is under the first mode, and the second information means that the user is distracted ([0174] “This is particularly useful in a contained environment like a car cabin (especially in a self-driving or autonomous vehicle)” [0033] “The present invention … helps with transition to self-driving vehicles to assist in smooth transfer of driving control from vehicle to driver and vice versa.” [0034] “the benefits of the present … makes car-driver hand over in self driving vehicles tight and safe.).
However, Andres teaches wherein the speaker outputs first conversation data corresponding to the first alert when the vehicle is under the first mode ([0058] “The takeover alert may include an audible description of the upcoming adverse event”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Andres’ teaching of outputting a verbal alert during autonomous driving. One would be motivated with a reasonable expectation of success to warn a driver during autonomous driving in order keep the driver attentive in case they need to take over into manual mode ([0058] “allowing the driver to further prepare for encountering the adverse event.”).
Further, Sobhany teaches first information being user preferences ([0029] “The user account database 210 can store data related to user accounts, such as user identification information (e.g., name, age, sex, birthday, address, VIP status, key user status, preferences, preferred language, vehicle(s) owned, greeting name, channel, talking points (e.g., favorite sports team), or geographic region of residence), account numbers, authorized users associated with one or more accounts, login credentials, or known biometric data associated with the user.”) and second information being biological information indicating user attention ([0013] “The sensors 112 each measure a parameter associated with a driver of the vehicle. The measured parameter can be any parameter related to the driver's state of attentiveness, including parameters that describe a position of at least a portion of the driver's body in the vehicle, parameters that quantify or qualify an expression on the driver's face, or parameters that measure a direction or rate of change of direction of the driver's gaze.”), and the second information means that the user is distracted (Sobhany teaches a plurality of embodiments, one having semi-autonomous driving as default: [0018] “vehicle control system 115 controls one or more driving functions while the driver concurrently controls one or more other driving functions. For example, the vehicle control system 115 may control a lane assist function, maintaining a lateral position of the vehicle 110 in a marked lane on a road, while the driver controls the speed of the vehicle. This partially autonomous self-driving mode may be active whenever the vehicle is operated” [0027] “the attention determination module 215 determines that the driver is inattentive based on a set of parameters and causes the vehicle to output an alert in response.”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of verbally alerting the driver of a partially autonomous vehicle when their attention is low. One would be motivated with reasonable expectation of success to alert the driver in order to rectify inattentiveness ([0009] “A vehicle monitors whether a driver is attentive to driving the vehicle and causes outputs to rectify driver inattentiveness.”).
Strubbe fails to disclose the first conversation data contains a self-counseling function that is configured to talk about a subject regarding user's biological information in order to alert the user that the user’s attention is decreased at the time.
However, Penilla teaches the first conversation data contains a self-counseling function that is configured to talk about a subject regarding user's biological information in order to alert the user that the user’s attention is decreased at the time ([0281] “vehicle system can deduct that the user is getting tired, and may need to rest or find a hotel. The vehicle can say, for example: ‘Are you ok?’ ‘Are you tired?’”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Penilla’s teaching of detecting the state of the driver including sleepiness and engaging in verbal dialog with the user. One would be motivated with reasonable expectation of success to engage in dialog with a sleepy driver in order to rouse the driver or ask if they would like to go somewhere to remedy the sleepiness (Penilla [0147] “if threshold percentage of conditions exist … provide verbal feedback/output to the user … “Would you like me to open the windows for some fresh air?”, “Big Coffee down the road has coffee on sale,” “SLEEP-E hotel is in 10 miles, book hotel room?”. In some embodiments, if no passenger is present, and driver fails to respond, sound audio noise to wake driver up, turn AC on to stimulate driver, etc.”).
Claims 4-13 are rejected under 35 U.S.C. 103 as being unpatentable over Strubbe, in view of Sicconi, Sobhany, Andres, and Penilla, further in view of Joyce et al. (US 20190385371 A1), and Yasui et al (US 20210150772 A1), hereafter Joyce and Yasui, respectively.
Regarding claim 4, Strubbe discloses A data processing device comprising:
a conversation data generation unit ([FIG. 2] response data generator 445);
an operation unit ([FIG. 1] Controller 100);
an image processing unit ([column 19 line 62] “image processor 305”)
a display device ([column 19 lines 62-63] “The display device may be a monitor 175 as shown in FIG. 1.”)
an imaging device ([column 26 lines 1-3] “an image processor 305 connected to cameras 135 and 136”)
a biosensor ([column 21, lines 53, 58-59] “The other UI classifiers 235 may include … galvanic skin response sensor, heart rate sensor,”);
a speaker ([FIG. 1] Speaker 114);
and a microphone ([FIG. 1] Microphone 112),
wherein the conversation data generation unit comprises a classifier ([FIG. 3] Mood/personality classifier 290) that has learned first information of a user ([column 22 lines 44-47] “The mood/personality classifier 290 receives signals from the various classifiers and processes these to generate a mood/personality state signal. The mood/personality classifier 290 may be a trained neural network” [column 22 lines 51-53] “The personality signal, preferably, is the result of many observations of behavior and would tend to persist over time.”),
wherein the biosensor is configured to detect second information of the user who uses the data processing device ([column 24 lines 60-65] “Referring now also to FIG. 4, the response generator 415 receives the mood/personality state vector and parsed reply data from the mood/personality classifier 290 and input parser 410 respectively. The response generator 415 also receives the environment/user state signal and events signal from the event/class processor 207.” [column 24 line 67 – column 25 line 4] “The response generator 415 selects a response from the response data store 440 based on the parsed reply from the user, the mood/ personality state, the environment/user state,” [column 10, lines 63-65] “A user's gaze direction is determined by two factors: the orientation of the head, and the orientation of the eyes.” [column 21, lines 58-59] “galvanic skin response sensor, heart rate sensor”),
wherein the imaging device is configured to capture a first image ([column 21 lines 39-41] “A video image classifier 240 receives video input 255, classifies image data and generates state information signals which are applied to the mood/personality classifier 290”),
wherein the operation unit is configured to detect a designated first object in the first image ([column 32 lines 63-65] “The video image classifier 240 may contain a process that regularly attempts to 65 distinguish objects in the room”),
wherein the image processing unit is configured to generate a second image when the first object is detected ([column 31 lines 16-17] “The video image classifier 240 classifies the user as idle and seated” [column 31 lines 24-27] “It also generates a synchronized light effect through the television cabinet effects driver 265 and an animation synchronized to the joke followed by a laughing animation through the animation driver 260.” The image processor detects the user and generates a second image (animation) when the first object (user sitting) is detected by the camera.),
wherein the image processing unit is configured to display the second image on the display device ([column 26 lines 1-4 “The animation driver 260 in turn may output a particular corresponding animation on a display device (not shown). The display device may be a monitor 175”),
wherein the first conversation data is configured to generated by the conversation data generation unit generates first conversation data on the basis of the first information and the second information ([column 24 lines 60-65] “Referring now also to FIG. 4, the response generator 415 receives the mood/personality state vector and parsed reply data from the mood/personality classifier 290 and input parser 410 respectively. The response generator 415 also receives the environment/user state signal and events signal from the event/class processor 207.” [column 24 line 67 – column 25 line 4] “The response generator 415 selects a response from the response data store 440 based on the parsed reply from the user, the mood/ personality state, the environment/user state,” [column 20, lines 32-35] “the response generator 415 takes state information from the classifiers 405, such as the mood of the user, the user's attention level, personality, interests, etc., and generates a response.”),
wherein the microphone is configured to obtain second conversation data corresponding to a response from the user and output the second conversation data to the classifier ([column 21 lines 1-2] “The audio input 245, which may be received by a microphone” [column 31 lines 10-13] “The audio input 245 is applied again to the speech to text converter 215 and the audio classifier 210.” [FIGs 3 and 6] audio input 245), and
wherein the classifier is configured to update the first information with use of the second conversation data ([column 21 lines 7-10] “The audio classifier 210 then generates a current state information signal which it applies to both a mood/personality classifier 290” the mood/personality classifier is updated with the signal generated by the audio classifier using the microphone detecting speech from the user. [column 30 lines 29-31] “The mood/personality classifier 290 stores personality data regarding the user and has detected, from previous conversations, the user's mood.”).
Strubbe fails to disclose wherein the data processing system in configured to communicate a vehicle connected with wireless communication or wire communication.
However, Sicconi teaches the data processing system in configured to communicate a vehicle connected with wireless communication or wire communication ([0178] “The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.” [0055] “a processing unit 4033 to analyze and process video streams from the two cameras, and to communicate 405 (wirelessly or via USB connection) with a mobile application on a phone 409.” [0058] “A Bluetooth connected speakerphone 703 is used to position microphone and speaker on a visor, close to the driver. A laptop 707 provides computation to run context analysis, combine it with the features extracted by the smart camera to decide about driver's attention margin and provide audio feedback when necessary. An optional WiFi-connected smartphone 711 provides telemetry data, map/routing info, cloud services (weather, traffic), audio/video recording capabilities and speech recognition and synthesis for dialog interaction with the driver.” The sensors in the vehicle may communicate data to the processing system, which may wired or wirelessly connected to the vehicle’s sensors and processed via smartphone, laptop, or even remote server.).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sicconi’s teaching of wired and/or wireless connection between the user data and the processing system. One would be motivated with a reasonable expectation of success to connect user data to the wired or wireless processing system in order to allow for secure process and storage backup ([0123] “Secure (encrypted) synchronization allows the smartphone to transfer EDR content (live or historic) to the cloud for further processing and storage backup”).
Strubbe fails to disclose where the vehicle has a first mode and a second mode.
However, Sobhany teaches the vehicle has a first mode and a second mode ([0018] “the vehicle control system 115 may engage or disengage the partially autonomous mode upon satisfaction of a triggering condition. For example, if the attentiveness of the driver falls below a specified threshold while the vehicle is operated in a fully manual mode, the vehicle control system 115 can engage the partially autonomous self-driving mode.”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of a first and second mode in the ego vehicle. One would be motivated with a reasonable expectation of success to use a first and second mode in a vehicle in order to either alert the driver to regain attention or take over control of the vehicle since lack of attention may result in potentially fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe fails to disclose wherein the first mode is a mode with a semi-automatic driving feature turned on that automatically control the vehicle driving by the vehicle itself, and the second mode is a mode with the semi-automatic driving feature turned off, wherein the semi-automatic driving feature includes automatic tracking drive.
However, Sobhany teaches wherein the first mode is a mode with a semi-automatic driving feature turned on that automatically control the vehicle driving by the vehicle itself, and the second mode is a mode with the semi-automatic driving feature turned off, wherein the semi-automatic driving feature includes automatic tracking drive ([0018] “the vehicle control system 115 may control a lane assist function, maintaining a lateral position of the vehicle 110 in a marked lane on a road, while the driver controls the speed of the vehicle … if the attentiveness of the driver falls below a specified threshold while the vehicle is operated in a fully manual mode, the vehicle control system 115 can engage the partially autonomous self-driving mode” both a manual mode where the tracking would be disabled, and a semi-autonomous mode where some tracking is enabled.).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of at least two driving modes that include manual control from the driver and at least partial autonomous control of the vehicle. One would be motivated with a reasonable expectation of success to use these two modes in a vehicle in order to either alert the driver to regain attention or take over control of the vehicle since lack of attention in a manual mode may result in potentially fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe fails to disclose wherein the first mode has a first alert, and the second mode has a second alert using one of a warning beep and flashing light.
However, Andres teaches wherein the first mode has a first alert ([0058] “The takeover alert may include an audible description of the upcoming adverse event”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Andres’ teaching of outputting a verbal alert during autonomous driving. One would be motivated with a reasonable expectation of success to warn a driver during autonomous driving in order keep the driver attentive in case they need to take over into manual mode ([0058] “allowing the driver to further prepare for encountering the adverse event.”).
Further, Sobhany teaches the second mode has a second alert using one of a warning beep and flashing light ([abstract] “driver is not attentive to driving the vehicle, the processor causes the vehicle to output an alert to the driver or to automatically control a driving function of the vehicle.” One type of alert occurs during manual driving when driver is not paying attention: [0036] “if the driver is determined to be not attentive … the vehicle output can be an alert that notifies the driver to re-engage … the alert can include … a beep … a flashing light”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of outputting at least an alert over a speaker in the car when it is determined that the driver is inattentive during manual driving. One would be motivated with a reasonable expectation of success to warn a driver in order prevent fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe fails to explicitly disclose wherein the image processing unit is configured to generate a second image where a second object overlaps with part of the first object.
However, Joyce teaches wherein the image processing unit is configured to generate a second image where a second object overlaps with part of the first object ([0014] “a pass through image of the physical environment may displayed on a display portion of the head mounted device 18, with virtual objects placed by the user superimposed on the pass through image of the physical environment.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the conversation system disclosed by Strubbe with the teachings from Joyce to utilize the system comprises augmented reality glasses in order to display the second object over the first object. Overlaying the character generated by Strubbe onto the environment and having it interact with the environment like the windows and the user would achieve a similar result with a reasonable expectation of success.
Strubbe teaches wherein the speaker is configured to output the first conversation data corresponding to the first alert in conjunction with movement of the second object ([column 25 lines 66-67] “The response generator 415 may select the output template and transmit it to an animation driver 260” [column 26 lines 17-21] “The animation provided by the animation driver 260 may be synchronized with the speech channel 80 by the text to speech converter 275 so that a 20 character appearing on the animation may be given the appearance of speaking.”),
Strubbe fails to disclose wherein the speaker is configured to output the first conversation data in conjunction with movement of the second object when the vehicle is under the first mode and the second information means that the user is distracted ([column 10 lines 58-61] “Gaze information plays an important role in identifying a person's focus of attention. This information may be used to provide communication cues in the present system.”).
However, Yasui teaches the speaker is configured to output the first conversation data corresponding to the first alert in conjunction with movement of the second object when the vehicle is under the first mode ([see FIG. 6] [0090] “The original data generator 132 generates original data of content that is obtained by causing the sound (for example, singing, an utterance, and the like of the virtual image) output from the sound output section 244 of the vehicle 200 to be coordinated with actions of the virtual image generated by the virtual image generator 242 of the vehicle 200” [0091] “utterance of the virtual image (that is, speech sounds of the virtual image)” [0059] “the content player 246 causes the sound (for example, a music) output from the sound output section 244 to be synchronized with display (for example, actions of the virtual image) output from the virtual image generator 242.” [0003] “In the related art, automated driving for automatically performing at least one of speed control and steering control” this is interpreted as semi-autonomous driving, where at least one control is done by the vehicle.).
Further, However, Sobhany teaches first information being user preferences ([0029] “The user account database 210 can store data related to user accounts, such as user identification information (e.g., name, age, sex, birthday, address, VIP status, key user status, preferences, preferred language, vehicle(s) owned, greeting name, channel, talking points (e.g., favorite sports team), or geographic region of residence), account numbers, authorized users associated with one or more accounts, login credentials, or known biometric data associated with the user.”) and second information being biological information indicating user attention ([0013] “The sensors 112 each measure a parameter associated with a driver of the vehicle. The measured parameter can be any parameter related to the driver's state of attentiveness, including parameters that describe a position of at least a portion of the driver's body in the vehicle, parameters that quantify or qualify an expression on the driver's face, or parameters that measure a direction or rate of change of direction of the driver's gaze.”) a speaker that outputs conversation data ([0036] “the alert can include an audio recording of … a spoken message (such as, “Please stay alert”).”) the second information means that the user is distracted (Sobhany teaches a plurality of embodiments, one having semi-autonomous driving as default: [0018] “vehicle control system 115 controls one or more driving functions while the driver concurrently controls one or more other driving functions. For example, the vehicle control system 115 may control a lane assist function, maintaining a lateral position of the vehicle 110 in a marked lane on a road, while the driver controls the speed of the vehicle. This partially autonomous self-driving mode may be active whenever the vehicle is operated” [0027] “the attention determination module 215 determines that the driver is inattentive based on a set of parameters and causes the vehicle to output an alert in response.”).
It would be obvious to one of ordinary skill in the art to modify Strubbe’s virtual character that is synced with conversation data which takes into account user state including gaze tracking, skin and heart sensors, with Yasui’s teaching of a semi-autonomous vehicle using a virtual character synced with conversation data which speaks with the occupants of the vehicle and Sobhany’s teaching of a semi-autonomous vehicle detecting the distraction of the driver and generating conversation data with the driver. One would be motivated with a reasonable expectation of success to combine the virtual character animated in sync with the conversation data with the conversation data generated when the driver is detected as being distracted in order to bring back attention to the road and reduce distraction or drowsiness ([0028] “it is possible to entertain the passenger with utterance and actions of the virtual image dynamically changing in accordance with a situation of the passenger of the vehicle and a situation outside the vehicle.” Sobhany [0009] “A vehicle monitors whether a driver is attentive to driving the vehicle and causes outputs to rectify driver inattentiveness.”).
Strubbe fails to disclose the first conversation data contains a self-counseling function that is configured to talk about a subject regarding user's biological information in order to alert the user that the user’s attention is decreased at the time.
However, Penilla teaches the first conversation data contains a self-counseling function that is configured to talk about a subject regarding user's biological information in order to alert the user that the user’s attention is decreased at the time. ([0281] “vehicle system can deduct that the user is getting tired, and may need to rest or find a hotel. The vehicle can say, for example: ‘Are you ok?’ ‘Are you tired?’”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Penilla’s teaching of detecting the state of the driver including sleepiness and engaging in verbal dialog with the user. One would be motivated with reasonable expectation of success to engage in dialog with a sleepy driver in order to rouse the driver or ask if they would like to go somewhere to remedy the sleepiness (Penilla [0147] “if threshold percentage of conditions exist … provide verbal feedback/output to the user … “Would you like me to open the windows for some fresh air?”, “Big Coffee down the road has coffee on sale,” “SLEEP-E hotel is in 10 miles, book hotel room?”. In some embodiments, if no passenger is present, and driver fails to respond, sound audio noise to wake driver up, turn AC on to stimulate driver, etc.”).
Regarding claim 5, Strubbe discloses A data processing device comprising:
a conversation data generation unit ([FIG. 2] response data generator 445);
an operation unit ([FIG. 1] Controller 100);
an image processing unit ([column 19 line 62] “image processor 305”)
a display device ([column 19 lines 62-63] “The display device may be a monitor 175 as shown in FIG. 1.”)
an imaging device ([column 26 lines 1-3] “an image processor 305 connected to cameras 135 and 136”)
a biosensor ([column 21, lines 53, 58-59] “The other UI classifiers 235 may include … galvanic skin response sensor, heart rate sensor,”);
a speaker ([FIG. 1] Speaker 114);
and a microphone ([FIG. 1] Microphone 112),
wherein the conversation data generation unit comprises a classifier ([FIG. 3] Mood/personality classifier 290) that has learned first information of a user ([column 22 lines 44-47] “The mood/personality classifier 290 receives signals from the various classifiers and processes these to generate a mood/personality state signal. The mood/personality classifier 290 may be a trained neural network” [column 22 lines 51-53] “The personality signal, preferably, is the result of many observations of behavior and would tend to persist over time.”),
wherein the biosensor is configured to detect second information of the user who uses the data processing device ([column 24 lines 60-65] “Referring now also to FIG. 4, the response generator 415 receives the mood/personality state vector and parsed reply data from the mood/personality classifier 290 and input parser 410 respectively. The response generator 415 also receives the environment/user state signal and events signal from the event/class processor 207.” [column 24 line 67 – column 25 line 4] “The response generator 415 selects a response from the response data store 440 based on the parsed reply from the user, the mood/ personality state, the environment/user state,” where first information is the mood state and the environment state can include the heart rate monitor or skin sensor.),
wherein the imaging device is configured to capture a first image ([column 21 lines 39-41] “A video image classifier 240 receives video input 255, classifies image data and generates state information signals which are applied to the mood/personality classifier 290”),
wherein the operation unit is configured to detect a designated first object in the first image ([column 32 lines 63-65] “The video image classifier 240 may contain a process that regularly attempts to 65 distinguish objects in the room”),
wherein the image processing unit is configured to generate a second image when the first object is detected ([column 31 lines 16-17] “The video image classifier 240 classifies the user as idle and seated” [column 31 lines 24-27] “It also generates a synchronized light effect through the television cabinet effects driver 265 and an animation synchronized to the joke followed by a laughing animation through the animation driver 260.” The image processor detects the user and generates a second image (animation) when the first object (user sitting) is detected by the camera.),
wherein the first conversation data is configured to generated by the conversation data generation unit generates first conversation data on the basis of the first information and the second information ([column 24 lines 60-65] “Referring now also to FIG. 4, the response generator 415 receives the mood/personality state vector and parsed reply data from the mood/personality classifier 290 and input parser 410 respectively. The response generator 415 also receives the environment/user state signal and events signal from the event/class processor 207.” [column 24 line 67 – column 25 line 4] “The response generator 415 selects a response from the response data store 440 based on the parsed reply from the user, the mood/ personality state, the environment/user state,” [column 20, lines 32-35] “the response generator 415 takes state information from the classifiers 405, such as the mood of the user, the user's attention level, personality, interests, etc., and generates a response.”),
wherein the speaker is configured to output the first conversation data corresponding to the first alert in conjunction with movement of the second object ([column 25 lines 66-67] “The response generator 415 may select the output template and transmit it to an animation driver 260” [column 26 lines 17-21] “The animation provided by the animation driver 260 may be synchronized with the speech channel 80 by the text to speech converter 275 so that a 20 character appearing on the animation may be given the appearance of speaking.”), and the second information means that the user is distracted ([column 20, lines 32-35] “the response generator 415 takes state information from the classifiers 405, such as the mood of the user, the user's attention level, personality, interests, etc., and generates a response.”);
wherein the microphone is configured to obtain second conversation data corresponding to a response from the user and output the second conversation data to the classifier ([column 21 lines 1-2] “The audio input 245, which may be received by a microphone” [column 31 lines 10-13] “The audio input 245 is applied again to the speech to text converter 215 and the audio classifier 210.” [FIGs 3 and 6] audio input 245), and
wherein the classifier is configured to update the first information with use of the second conversation data ([column 21 lines 7-10] “The audio classifier 210 then generates a current state information signal which it applies to both a mood/personality classifier 290” the mood/personality classifier is updated with the signal generated by the audio classifier using the microphone detecting speech from the user. [column 30 lines 29-31] “The mood/personality classifier 290 stores personality data regarding the user and has detected, from previous conversations, the user's mood.”).
Strubbe fails to disclose wherein the data processing system in configured to communicate a vehicle connected with wireless communication or wire communication.
However, Sicconi teaches the data processing system in configured to communicate a vehicle connected with wireless communication or wire communication ([0178] “The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.” [0055] “a processing unit 4033 to analyze and process video streams from the two cameras, and to communicate 405 (wirelessly or via USB connection) with a mobile application on a phone 409.” [0058] “A Bluetooth connected speakerphone 703 is used to position microphone and speaker on a visor, close to the driver. A laptop 707 provides computation to run context analysis, combine it with the features extracted by the smart camera to decide about driver's attention margin and provide audio feedback when necessary. An optional WiFi-connected smartphone 711 provides telemetry data, map/routing info, cloud services (weather, traffic), audio/video recording capabilities and speech recognition and synthesis for dialog interaction with the driver.” The sensors in the vehicle may communicate data to the processing system, which may wired or wirelessly connected to the vehicle’s sensors and processed via smartphone, laptop, or even remote server.).
It would be obvious to one of ordinary skill in the art to modify Strubbe with Sicconi’s teaching of wired and/or wireless connection between the user data and the processing system. One would be motivated with a reasonable expectation of success to connect user data to the wired or wireless processing system in order to allow for secure process and storage backup ([0123] “Secure (encrypted) synchronization allows the smartphone to transfer EDR content (live or historic) to the cloud for further processing and storage backup”).
Strubbe fails to disclose where the vehicle has a first mode and a second mode.
However, Sobhany teaches the vehicle has a first mode and a second mode ([0018] “the vehicle control system 115 may engage or disengage the partially autonomous mode upon satisfaction of a triggering condition. For example, if the attentiveness of the driver falls below a specified threshold while the vehicle is operated in a fully manual mode, the vehicle control system 115 can engage the partially autonomous self-driving mode.”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of a first and second mode in the ego vehicle. One would be motivated with a reasonable expectation of success to use a first and second mode in a vehicle in order to either alert the driver to regain attention or take over control of the vehicle since lack of attention may result in potentially fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe fails to disclose wherein the first mode is a mode with a semi-automatic driving feature turned on that automatically control the vehicle driving by the vehicle itself, and the second mode is a mode with the semi-automatic driving feature turned off, wherein the semi-automatic driving feature includes automatic tracking drive.
However, Sobhany teaches wherein the first mode is a mode with a semi-automatic driving feature turned on that automatically control the vehicle driving by the vehicle itself, and the second mode is a mode with the semi-automatic driving feature turned off, wherein the semi-automatic driving feature includes automatic tracking drive ([0018] “the vehicle control system 115 may control a lane assist function, maintaining a lateral position of the vehicle 110 in a marked lane on a road, while the driver controls the speed of the vehicle … if the attentiveness of the driver falls below a specified threshold while the vehicle is operated in a fully manual mode, the vehicle control system 115 can engage the partially autonomous self-driving mode” both a manual mode where the tracking would be disabled, and a semi-autonomous mode where some tracking is enabled.).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Sobhany’s teaching of at least two driving modes that include manual control from the driver and at least partial autonomous control of the vehicle. One would be motivated with a reasonable expectation of success to use these two modes in a vehicle in order to either alert the driver to regain attention or take over control of the vehicle since lack of attention in a manual mode may result in potentially fatal safety hazards (Sobhany [0003] “lack of attention can lead to safety hazards ranging from trivial to catastrophic and potentially fatal.”).
Strubbe fails to explicitly disclose wherein the image processing unit is configured to generate a second image where a second object overlaps with part of the first object.
However, Joyce teaches wherein the image processing unit is configured to generate a second image where a second object overlaps with part of the first object ([0014] “a pass through image of the physical environment may displayed on a display portion of the head mounted device 18, with virtual objects placed by the user superimposed on the pass through image of the physical environment.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the conversation system disclosed by Strubbe with the teachings from Joyce to utilize the system comprises augmented reality glasses in order to display the second object over the first object. Overlaying the character generated by Strubbe onto the environment and having it interact with the environment like the windows and the user would achieve a similar result with a reasonable expectation of success.
Strubbe discloses wherein the image processing unit is configured to display the second image on the display device ([column 26 lines 1-4 “The animation driver 260 in turn may output a particular corresponding animation on a display device (not shown). The display device may be a monitor 175”).
Strubbe fails to disclose wherein the image processing unit is configured to display the second image on the display device when the vehicle is under the first mode.
However, Yasui teaches the image processing unit is configured to display the second image on the display device when the vehicle is under the first mode (([see FIG. 6] [0090] “The original data generator 132 generates original data of content that is obtained by causing the sound (for example, singing, an utterance, and the like of the virtual image) output from the sound output section 244 of the vehicle 200 to be coordinated with actions of the virtual image generated by the virtual image generator 242 of the vehicle 200” [0091] “utterance of the virtual image (that is, speech sounds of the virtual image)” [0059] “the content player 246 causes the sound (for example, a music) output from the sound output section 244 to be synchronized with display (for example, actions of the virtual image) output from the virtual image generator 242.” [0003] “In the related art, automated driving for automatically performing at least one of speed control and steering control” this is interpreted as semi-autonomous driving, where at least one control is done by the vehicle.).).
It would be obvious to one of ordinary skill in the art to modify Strubbe’s virtual character that is synced with conversation data which takes into account user state including gaze tracking, skin and heart sensors, with Yasui’s teaching of an autonomous vehicle using a virtual character synced with conversation data which speaks with the occupants of the vehicle. One would be motivated with a reasonable expectation of success to incorporate the character into an autonomous vehicle in order to entertain passengers of the vehicle (Yasui [0006] “program capable of providing ways to entertain passengers of vehicles”).
Strubbe fails to disclose the first conversation data contains a self-counseling function that is configured to talk about a subject regarding user's biological information in order to alert the user that the user’s attention is decreased at the time.
However, Penilla teaches the first conversation data contains a self-counseling function that is configured to talk about a subject regarding user's biological information in order to alert the user that the user’s attention is decreased at the time ([0281] “vehicle system can deduct that the user is getting tired, and may need to rest or find a hotel. The vehicle can say, for example: ‘Are you ok?’ ‘Are you tired?’”).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Strubbe with Penilla’s teaching of detecting the state of the driver including sleepiness and engaging in verbal dialog with the user. One would be motivated with reasonable expectation of success to engage in dialog with a sleepy driver in order to rouse the driver or ask if they would like to go somewhere to remedy the sleepiness (Penilla [0147] “if threshold percentage of conditions exist … provide verbal feedback/output to the user … “Would you like me to open the windows for some fresh air?”, “Big Coffee down the road has coffee on sale,” “SLEEP-E hotel is in 10 miles, book hotel room?”. In some embodiments, if no passenger is present, and driver fails to respond, sound audio noise to wake driver up, turn AC on to stimulate driver, etc.”).
Regarding claims 6 and 10, Strubbe discloses The data processing device according to claim 4, wherein the first information is preference information ([column 7 lines 41-44] “an electronic program guide preference database can be augmented by extracting likes and dislikes from dialogues and applying them to the database” [column 25 lines 14-15] “the database 430 is a user profile containing preferences”).
Regarding claims 7 and 11, Strubbe discloses The data processing device according claim 4, wherein the second information is biological information (the input classifiers 235 heart rate sensor and galvanic skin sensor would be considered biological information to one of ordinary skill in the art.).
Regarding claims 8 and 12, Strubbe fails to disclose A wearable device comprising the data processing device according to claim 4, wherein the data processing device can serve as glasses.
However, Joyce teaches A wearable device comprising the data processing device according to claim 4, wherein the data processing device can serve as glasses ([FIG. 1B] Glasses 16 [0014] “the user may view and experience the mixed reality environment via a display portion of a head mounted device 16. In this arrangement, the physical environment may be visible to the user through the head mounted device 16, and the virtual objects may be placed by the user and visible on a display portion of the head mounted device 16, superimposed on the user's view of the physical environment”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the conversation system disclosed by Strubbe with the teachings from Joyce to utilize the system comprises augmented reality glasses in order to provide a hands free and more immersive experience with the conversation with Strubbe’s animated character.
Regarding claims 9 and 13, Strubbe fails to disclose A wearable device comprising the data processing device according to claim 4, wherein the data processing device can serve as glasses, and wherein a location where the second object is displayed can be designated by the user.
However, Joyce teaches the data processing device can serves as glasses, and wherein a location where the second object is displayed can be designated by the user ([0016] “The user may place a first virtual object A in the mixed reality scene 100, as shown in FIG. 2C.” [0014] “the physical environment may be visible to the user through the head mounted device 16, and the virtual objects may be placed by the user and visible on a display portion of the head mounted device 16, superimposed on the user's view of the physical environment.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the conversation system disclosed by Strubbe with the teachings from Joyce to utilize the system comprises augmented reality glasses where the user can place the virtual object in the scene visible in their glasses in order to allow the user to control the experience.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARK R HEIM whose telephone number is (571)270-0120. The examiner can normally be reached M-F 9-6 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fadey Jabr can be reached at 571-272-1516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.R.H./Examiner, Art Unit 3668
/Fadey S. Jabr/Supervisory Patent Examiner, Art Unit 3668
Prosecution Insights