DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to the rejections of claims 1-20 under 35 U.S.C. §101 have been fully considered and are persuasive. The rejections are hereby withdrawn.
Applicant’s arguments with respect to the rejections of claims 1-20 under 35 U.S.C. §103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of amendments made to the independent claims, narrowing the scope of the claimed invention.
Claim Objections
Claim 20 is objected to because of the following informalities.
Claim 20 contains the terms “the HMI” and “the LLM” throughout the claims, without a declaration of “an HMI” or “an LLM”. Applicant is advised to provide clarifying amendments, in order to avoid a potential 35 U.S.C. 112(b) rejection on lack of antecedent basis.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4 and 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over US 20250013683 A1, filed January 3rd, 2024, hereinafter “Granieri”, in view of US 12332074 B1, filed April 19th, 2024, hereinafter “Carbune”, further in view of US 20260116418 A1, filed October 25th, 2024, hereinafter “Silva”.
Regarding claim 1, Granieri teaches a method of providing intention-based infotainment within a vehicle. See at least [0011] and figure 4.
comprising, with a system controller in communication with a plurality of onboard sensors within the vehicle. See at least [0091], [0093], and figure 1, wherein sensor system 120 includes sensors arranged within a passenger cabin of the vehicle 100 and is operatively connected to the processor(s) 110 of the vehicle.
collecting data related to intentions of an occupant within the vehicle. See at least [0044]-[0045], [0050]-[0052], and figure 4, step 410, wherein data is collected related to occupants of the vehicle and their inputs, gestures, or commands.
developing a request based on the data related to the intentions of the occupant within the vehicle. See at least [0053]-[0054] and figure 4, step 420, wherein the collected occupant data is used to identify a request that the user intends to make.
collecting data from the plurality of onboard sensors related to the request. See at least [0057] and figure 4, steps 430-440, wherein additional voice or image sensor data is collected based on the identified request.
receiving, via a wireless communication module, data from remote sources related to the request. See at least [0032], [0035], [0043], [0090], and figure 3, wherein the system additionally comprises a wireless communication module 180, and retrieves map data from a cloud-based environment. See at least [0058] and figure 3, steps 410, 420, and 430, wherein the data collected in these steps additionally includes map data.
formulating, with a language model in communication with the system controller via the wireless communication module, a response. See at least [0065]-[0066] and figure 4, step 480, wherein the occupant’s search request is provided to a neural network language model to generate an answer to the search query. The model can be located remotely from the vehicle, as part of the cloud-computing environment 300.
and actuating systems within the vehicle to automatically: provide, via a human machine interface (HMI), infotainment content for the occupant within the vehicle. See at least [0067] and figure 4, step 490, wherein the search request results are provided to the occupant, including controlling the infotainment system of the vehicle and communicating the search results to the occupant via a user interface.
and control, via an automated driving assistance system (ADAS), operation of the vehicle. See at least [0067]-[0069] and figure 4, step 490, wherein the response to the search request includes controlling an autonomous driving system of the vehicle, for example, executing a parking assist function.
Granieri remains silent on a large language model (LLM). As discussed above, Granieri discloses using a neural network language model to respond to the occupant’s search request. Granieri additionally does not teach the specifics of performing the infotainment and ADAS steps in one response. As discussed above, Granieri discloses the two steps as separate responses to subsequent requests.
Carbune teaches a large language model (LLM). See at least col. 10, line 50 – col. 11, line 10 and figures 1-2, wherein a large language model is used to generate textual content in response to a determined user query.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s large language model. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Silva teaches actuating vehicle systems to automatically provide infotainment content and control operation of the vehicle. See at least [0025]-[0026], wherein the LLM 240, in response to a driver input, outputs one or more assistive actions, including operational actions (controlling steering, acceleration, and braking) and infotainment actions (advising the driver or warning the driver). See at least [0028], wherein, in an example, when a request corresponding to a driver’s intention of entering a strip mall is made, the system responds by slowing down the vehicle while alerting the driver (via speech or test).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to further modify Granieri with Silva’s technique of providing content for drivers while additionally controlling operation of the vehicle. It would have been obvious to modify because doing so enables vehicular virtual assistants to better assist drivers and other vehicle occupants.
Regarding claim 2, Granieri, Carbune, and Silva in combination disclose all of the limitations of claim 1 as discussed above, and Granieri additionally teaches wherein the collecting data related to intentions of an occupant within the vehicle further includes at least one of: collecting, with an occupant monitoring system data related to what the occupant is looking at, gestures made by the occupant, and facial expressions of the occupant. See at least [0023], [0041], and [0053], wherein the step of collecting occupant data includes collecting sensor data from the cabin of the vehicle to acquire gestures made by the occupant and facial characterization of the occupant related to the occupant’s mood and emotional state. The gestures include motion indication a direction of the occupants eyes in the surrounding environment.
and collecting, with the HMI, verbal expressions made by the occupant and manual data input from the occupant. See at least [0054], wherein the step of collecting occupant data includes collecting the occupant’s spoken words, or the occupant’s written request provided digitally at an input system 130 of the vehicle.
Regarding claim 3, Granieri, Carbune, and Silva in combination disclose all of the limitations of claim 2 as discussed above, and Granieri additionally teaches wherein the collecting data related to intentions of an occupant within the vehicle further includes accessing, with a machine learning model, stored data within a database related to past occurrences of the occupant using the vehicle. See at least [0070]-[0071] and figure 5, wherein, during the steps 510-520 of collecting data related to an occupant’s intended search request, execution module 230 references a historical log of the occupant’s previous intended requests stored in data store 240. See at least [0064] and figure 5, step 530, wherein the historical occupant data is used by machine learning model(s) 290 to identify learned patterns of specific drivers.
Regarding claim 4, Granieri, Carbune, and Silva in combination disclose all of the limitations of claim 3 as discussed above, and Granieri additionally teaches wherein the developing a request based on the data related to the intentions of the occupant within the vehicle further includes: detecting, with the system controller, one of: triggering language from the occupant, via the HMI; or a triggering event, via the plurality of onboard sensors. See at least [0053]-[0054] and figure 4, step 420, wherein the step of identifying a request made by the occupant includes detecting a wake word (triggering language) or a gesture (triggering event) using the sensors of the vehicle.
Regarding claim 11, Granieri teaches a system for providing intention-based infotainment within a vehicle,. See at least [0009] and figures 1-3.
comprising: a system controller in communication with a plurality of onboard sensors within the vehicle. See at least [0091], [0093], and figure 1, wherein sensor system 120 includes sensors arranged within a passenger cabin of the vehicle 100 and is operatively connected to the processor(s) 110 of the vehicle.
and adapted to: collect data related to intentions of an occupant within the vehicle. See at least [0044]-[0045], [0050]-[0052], and figure 4, step 410, wherein data is collected related to occupants of the vehicle and their inputs, gestures, or commands.
develop a request based on the data related to the intentions of the occupant within the vehicle. See at least [0053]-[0054] and figure 4, step 420, wherein the collected occupant data is used to identify a request that the user intends to make.
collect data from the plurality of onboard sensors related to the request. See at least [0057] and figure 4, steps 430-440, wherein additional voice or image sensor data is collected based on the identified request.
receive, via a wireless communication module, data from remote sources related to the request. See at least [0032], [0035], [0043], [0090], and figure 3, wherein the system additionally comprises a wireless communication module 180, and retrieves map data from a cloud-based environment. See at least [0058] and figure 3, steps 410, 420, and 430, wherein the data collected in these steps additionally includes map data.
formulate, with a language model in communication with the system controller via the wireless communication module, a response. See at least [0065]-[0066] and figure 4, step 480, wherein the occupant’s search request is provided to a neural network language model to generate an answer to the search query. The model can be located remotely from the vehicle, as part of the cloud-computing environment 300.
and actuate systems within the vehicle to automatically: provide, via a human machine interface (HMI), infotainment content for the occupant within the vehicle. See at least [0067] and figure 4, step 490, wherein the search request results are provided to the occupant, including controlling the infotainment system of the vehicle and communicating the search results to the occupant via a user interface.
and control, via an automated driving assistance system (ADAS), operation of the vehicle. See at least [0067]-[0069] and figure 4, step 490, wherein the response to the search request includes controlling an autonomous driving system of the vehicle, for example, executing a parking assist function.
Granieri remains silent on a large language model (LLM). As discussed above, Granieri discloses using a neural network language model to respond to the occupant’s search request. Granieri additionally does not teach the specifics of performing the infotainment and ADAS steps in one response. As discussed above, Granieri discloses the two steps as separate responses to subsequent requests.
Carbune teaches a large language model (LLM). See at least col. 10, line 50 – col. 11, line 10 and figures 1-2, wherein a large language model is used to generate textual content in response to a determined user query.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s large language model. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Silva teaches actuating vehicle systems to automatically provide infotainment content and control operation of the vehicle. See at least [0025]-[0026], wherein the LLM 240, in response to a driver input, outputs one or more assistive actions, including operational actions (controlling steering, acceleration, and braking) and infotainment actions (advising the driver or warning the driver). See at least [0028], wherein, in an example, when a request corresponding to a driver’s intention of entering a strip mall is made, the system responds by slowing down the vehicle while alerting the driver (via speech or test).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to further modify Granieri with Silva’s technique of providing content for drivers while additionally controlling operation of the vehicle. It would have been obvious to modify because doing so enables vehicular virtual assistants to better assist drivers and other vehicle occupants.
Regarding claim 12, Granieri, Carbune, and Silva in combination disclose all of the limitations of claim 11 as discussed above, and Granieri additionally teaches wherein when collecting data related to intentions of an occupant within the vehicle, the system controller is further adapted to at least one of: collect, with an occupant monitoring system data related to what the occupant is looking at, gestures made by the occupant, and facial expressions of the occupant. See at least [0023], [0041], and [0053], wherein the step of collecting occupant data includes collecting sensor data from the cabin of the vehicle to acquire gestures made by the occupant and facial characterization of the occupant related to the occupant’s mood and emotional state. The gestures include motion indication a direction of the occupants eyes in the surrounding environment.
and collect, with the HMI, verbal expressions made by the occupant and manual data input from the occupant. See at least [0054], wherein the step of collecting occupant data includes collecting the occupant’s spoken words, or the occupant’s written request provided digitally at an input system 130 of the vehicle.
Regarding claim 13, Granieri, Carbune, and Silva in combination disclose all of the limitations of claim 12 as discussed above, and Granieri additionally teaches wherein when collecting data related to intentions of an occupant within the vehicle, the system controller is further adapted to access, with a machine learning model, stored data within a database related to past occurrences of the occupant using the vehicle. See at least [0070]-[0071] and figure 5, wherein, during the steps 510-520 of collecting data related to an occupant’s intended search request, execution module 230 references a historical log of the occupant’s previous intended requests stored in data store 240. See at least [0064] and figure 5, step 530, wherein the historical occupant data is used by machine learning model(s) 290 to identify learned patterns of specific drivers.
Regarding claim 14, Granieri, Carbune, and Silva in combination disclose all of the limitations of claim 13 as discussed above, and Granieri additionally teaches wherein when developing a request based on the data related to the intentions of the occupant within the vehicle, the system controller is further adapted to detect one of triggering language from the occupant, via the HMI, or a triggering event, via the plurality of onboard sensors. See at least [0053]-[0054] and figure 4, step 420, wherein the step of identifying a request made by the occupant includes detecting a wake word (triggering language) or a gesture (triggering event) using the sensors of the vehicle.
Claims 5-10 and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Granieri, Carbune, and Silva as applied to claims above, and further in view of US 20250265278 A1, with an earliest priority date of April 28th, 2023, hereinafter “Chen”.
Regarding claim 5, Granieri, Carbune, and Silva in combination disclose all of the limitations of claim 4 as discussed above, and Granieri additionally teaches wherein the developing a request based on the data related to the intentions of the occupant within the vehicle further includes: analyzing, with the machine learning model, intentions of the occupant based on the data stored within the database. See at least [0070]-[0071] and figure 5, wherein, during the steps 510-520 of collecting data related to an occupant’s intended search request, execution module 230 references a historical log of the occupant’s previous intended requests stored in data store 240. See at least [0064] and figure 5, step 530, wherein the historical occupant data is used by machine learning model(s) 290 to identify learned patterns of specific drivers.
real time data collected by the plurality of onboard sensor and data received from remote sources. See at least [0051], wherein the sensor data 250 is acquired at successive time steps. See at least [0032], [0035], [0043], [0090], and figure 3, wherein the system additionally comprises a wireless communication module 180, and retrieves map data from a cloud-based environment. See at least [0058] and figure 3, steps 410, 420, and 430, wherein the data collected in these steps additionally includes map data.
Granieri remains silent on predicting the occupant intentions, analyzing, with the LLM, the data related to the intentions of the occupant within the vehicle; identifying, with the machine learning model and the LLM, key terms and qualifier terms; and quantifying, with the system controller, using the LLM, the machine learning model, the key terms and qualifier terms.
Carbune teaches predicting intentions of the occupant based on the data stored within the database. See at least col. 8, line 54 – col. 9, line 17, wherein the module 118 predicts a user’s query intent based on information using predictive analysis and historical user behavior. Additionally, see at least col. 16, line 63 – col. 17, line 15, wherein user historical information 304is stored in service information 122.
analyzing the data related to the intentions of the occupant within the vehicle. See at least col. 8, line 54 – col. 9, line 17, wherein the module 118 predicts a user’s query intent and generates a query 131 with query information 130. See at least col. 9, lines 35-47, wherein the query information 130 is analyzed by POI selector 132.
identifying key terms and qualifier terms. See at least col. 8, lines 43-55, wherein the query information 130 includes terms that describe a type of POI and matching criteria for the POI (e.g. expensive, within walking distance, local, suitable for young children, etc.). Per [0069] of Applicant’s specification, these qualify as the key terms and qualifier terms, respectively.
and quantifying, with the system controller, the key terms and qualifier terms. See at least col. 9, lines 48-59, wherein, upon identifying a qualifier term (“within walking distance”), the POI selector 132 uses user data to quantify the term as an objective distance.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of predicting occupant intentions based on stored data, analyzing the stored intention data, identifying key terms and qualifier terms, and quantifying the key terms and qualifier terms. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Chen teaches analyzing with the LLM, identifying terms with the machine learning model and the LLM. See at least [0060]-[0062], wherein model 330 is a machine learning model comprising a large language model. See at least [0101]-[0106] and figure 4, step 501, wherein a user’s search request is input to a generative language model to obtain classification terms (key terms) such as a “city C”, “restaurant”, “Cantonese restaurant”, etc. Additionally, see at least [0107]-[0112] and figure 4, step 501, wherein the generative language model additionally obtains requirement description terms (qualifier terms) which comprise natural language such as “good environment”, “quiet atmosphere”, “few customers”, “good road conditions”, etc.
and quantifying using the LLM, machine learning model. See at least [0170]-[0171] and figure 9, wherein requirement description information (qualifier terms) are output by generative language model 920 using map search content 911, prompt information 912, and real-time location prompt information 913. See at least [0174]-[0177], wherein the generative language model obtains a true value corresponding to the requirement description information.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to further modify Granieri with Chen’s technique of analyzing data, identifying key terms and qualifier terms, and quantifying using an LLM and machine learning model. It would have been obvious to modify because doing so enables mapping services to utilize generative language models to provide users with better search results, as recognized by Chen (see at least [0053]-[0054]).
Regarding claim 6, Granieri, Carbune, Silva, and Chen in combination disclose all of the limitations of claim 5 as discussed above, and Granieri additionally teaches wherein the collecting data from the plurality of onboard sensors related to the request and the receiving, via the wireless communication module, data from remote sources related to the request further includes: identifying, with the system controller, a geographic area within which relevant data related to the request will be collected; collecting relevant data related to the request within the geographic area. See at least [0059]-[0060] and figure 4, step 450, wherein the occupant’s request is correlated with a target. The target is a portion of the surrounding environment of the occupant which contains the objects/areas of the occupant’s interest. The target is obtained by collecting map data within a virtual shape defining the target area.
Granieri remains silent on filtering the collected data based on the key terms and qualifier terms.
Carbune teaches filtering the collected data based on the key terms and qualifier terms. See at least col. 9, line 60 – col. 10, line 9, wherein the query information is used to filter points of interest from the map search results, based on “within walking distance” (qualifier term) and “restaurant” (key term).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of filtering collected data based on the key terms and qualifier terms. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Regarding claim 7, Granieri, Carbune, Silva, and Chen in combination disclose all of the limitations of claim 6 as discussed above, and Granieri remains silent on wherein the collecting data from the plurality of onboard sensors related to the request and the receiving, via the wireless communication module, data from remote sources related to the request further includes: identifying, with the system controller, using the machine learning model, unspecified key terms and unspecified qualifier terms, related to the request and not included within the request.
Carbune teaches identifying, with the system controller, using the machine learning model, unspecified key terms and unspecified qualifier terms, related to the request and not included within the request. See at least col. 9, line 60 – col. 10, line 22, wherein the map search data is filtered based on terms not included in the query information 130, such as seafood restaurants (key terms) or restaurants with fast service (qualifier terms).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of filtering collected data based on the key terms and qualifier terms. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Regarding claim 8, Granieri, Carbune, Silva, and Chen in combination disclose all of the limitations of claim 7 as discussed above, and Granieri additionally teaches wherein the collecting data from the plurality of onboard sensors related to the request and the receiving, via the wireless communication module, data from remote sources related to the request further includes: identifying, with the system controller, an extended geographic area within which relevant data related to the request, will be collected; collecting relevant data related to the request within the extended geographic area. See at least [0059]-[0060] and figure 4, step 450, wherein the occupant’s request is correlated with a target. The target is a portion of the surrounding environment of the occupant which contains the objects/areas of the occupant’s interest. The target is obtained by collecting map data within a virtual shape defining the target area. See at least [0061], wherein the target area is larger in size, or extended, based on the occupant’s request.
Granieri remains silent on including unspecified key terms and unspecified qualifier terms, and filtering the collected data based on the key terms, qualifier terms, unspecified key terms and unspecified qualifier terms.
Carbune teaches including unspecified key terms and unspecified qualifier terms, and filtering the collected data based on the key terms, qualifier terms, unspecified key terms and unspecified qualifier terms. See at least col. 9, line 60 – col. 10, line 22, wherein the map search data is filtered based on terms not included in the query information 130, such as seafood restaurants (key terms) or restaurants with fast service (qualifier terms).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of filtering collected data based on the key terms and qualifier terms. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Regarding claim 9, Granieri, Carbune, Silva, and Chen in combination disclose all of the limitations of claim 8 as discussed above, and Granieri additionally teaches wherein the formulating, with a large language model (LLM) in communication with the system controller via the wireless communication module, a response, and actuating systems within the vehicle to automatically provide, via the HMI, infotainment content for the occupant within the vehicle further includes: and at least one of: displaying a textual language response on a touch screen display of the HMI; and broadcasting, via a speaker associated with the HMI, a verbal response for the occupant. See at least [0067], figure 4, step 490, and figures 6-7, wherein the generated response is either displayed visually through a user interface of the vehicle, or output audially through a sound system of the vehicle.
Granieri remains silent on formulating, with the LLM, a natural language response for the occupant.
Carbune teaches formulating, with the LLM, a natural language response for the occupant. See at least col. 10, line 61 – col. 11, line 32, col. 14, lines 34-59 and figure 2, wherein the textual response to the user is generated using LLMs. See at least col. 25, line 66 – col. 26, line 23, wherein the machine learning modules generate natural language output.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of formulating a natural language response for the user using an LLM. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Regarding claim 10, Granieri, Carbune, Silva, and Chen in combination disclose all of the limitations of claim 9 as discussed above, and Granieri additionally teaches wherein the formulating, with a large language model (LLM) in communication with the system controller via the wireless communication module, a response, and actuating systems within the vehicle to automatically control, via an automated driving assistance system (ADAS), operation of the vehicle further includes: identifying, with the LLM and the machine learning model, a desired vehicle operation based on the request; and automatically, via the ADAS, performing the desired vehicle operation. See at least [0069], [0074], [0078]-[0079], [0082], and figures 7A-B, wherein the generated output for the user includes identifying an execution function, and performing the identified function. The function includes ADAS systems such as parking assist functions.
Regarding claim 15, Granieri, Carbune, and Silva in combination disclose all of the limitations of claim 14 as discussed above, and Granieri additionally teaches wherein when developing a request based on the data related to the intentions of the occupant within the vehicle, the system controller is further adapted to: analyze, with the machine learning model, intentions of the occupant based on the data stored within the database. See at least [0070]-[0071] and figure 5, wherein, during the steps 510-520 of collecting data related to an occupant’s intended search request, execution module 230 references a historical log of the occupant’s previous intended requests stored in data store 240. See at least [0064] and figure 5, step 530, wherein the historical occupant data is used by machine learning model(s) 290 to identify learned patterns of specific drivers.
real time data collected by the plurality of onboard sensor and data received from remote sources. See at least [0051], wherein the sensor data 250 is acquired at successive time steps. See at least [0032], [0035], [0043], [0090], and figure 3, wherein the system additionally comprises a wireless communication module 180, and retrieves map data from a cloud-based environment. See at least [0058] and figure 3, steps 410, 420, and 430, wherein the data collected in these steps additionally includes map data.
Granieri remains silent on predict the occupant intentions, analyze, with the LLM, the data related to the intentions of the occupant within the vehicle; identify, with the machine learning model and the LLM, key terms and qualifier terms; and quantify, using the LLM, the machine learning model, the key terms and qualifier terms.
Carbune teaches predict intentions of the occupant based on the data stored within the database. See at least col. 8, line 54 – col. 9, line 17, wherein the module 118 predicts a user’s query intent based on information using predictive analysis and historical user behavior. Additionally, see at least col. 16, line 63 – col. 17, line 15, wherein user historical information 304is stored in service information 122.
analyze the data related to the intentions of the occupant within the vehicle. See at least col. 8, line 54 – col. 9, line 17, wherein the module 118 predicts a user’s query intent and generates a query 131 with query information 130. See at least col. 9, lines 35-47, wherein the query information 130 is analyzed by POI selector 132.
identify key terms and qualifier terms. See at least col. 8, lines 43-55, wherein the query information 130 includes terms that describe a type of POI and matching criteria for the POI (e.g. expensive, within walking distance, local, suitable for young children, etc.). Per [0069] of Applicant’s specification, these qualify as the key terms and qualifier terms, respectively.
and quantify, with the system controller, the key terms and qualifier terms. See at least col. 9, lines 48-59, wherein, upon identifying a qualifier term (“within walking distance”), the POI selector 132 uses user data to quantify the term as an objective distance.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of predicting occupant intentions based on stored data, analyzing the stored intention data, identifying key terms and qualifier terms, and quantifying the key terms and qualifier terms. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Chen teaches analyzing with the LLM, identifying terms with the machine learning model and the LLM. See at least [0060]-[0062], wherein model 330 is a machine learning model comprising a large language model. See at least [0101]-[0106] and figure 4, step 501, wherein a user’s search request is input to a generative language model to obtain classification terms (key terms) such as a “city C”, “restaurant”, “Cantonese restaurant”, etc. Additionally, see at least [0107]-[0112] and figure 4, step 501, wherein the generative language model additionally obtains requirement description terms (qualifier terms) which comprise natural language such as “good environment”, “quiet atmosphere”, “few customers”, “good road conditions”, etc.
and quantifying using the LLM, machine learning model. See at least [0170]-[0171] and figure 9, wherein requirement description information (qualifier terms) are output by generative language model 920 using map search content 911, prompt information 912, and real-time location prompt information 913. See at least [0174]-[0177], wherein the generative language model obtains a true value corresponding to the requirement description information.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to further modify Granieri with Chen’s technique of analyzing data, identifying key terms and qualifier terms, and quantifying using an LLM and machine learning model. It would have been obvious to modify because doing so enables mapping services to utilize generative language models to provide users with better search results, as recognized by Chen (see at least [0053]-[0054]).
Regarding claim 16, Granieri, Carbune, Silva, and Chen in combination disclose all of the limitations of claim 15 as discussed above, and Granieri additionally teaches wherein when collecting data from the plurality of onboard sensors related to the request and receiving, via the wireless communication module, data from remote sources related to the request, the system controller is further adapted to: identify a geographic area within which relevant data related to the request will be collected; collect relevant data related to the request within the geographic area. See at least [0059]-[0060] and figure 4, step 450, wherein the occupant’s request is correlated with a target. The target is a portion of the surrounding environment of the occupant which contains the objects/areas of the occupant’s interest. The target is obtained by collecting map data within a virtual shape defining the target area.
Granieri remains silent on filter the collected data based on the key terms and qualifier terms.
Carbune teaches filter the collected data based on the key terms and qualifier terms. See at least col. 9, line 60 – col. 10, line 9, wherein the query information is used to filter points of interest from the map search results, based on “within walking distance” (qualifier term) and “restaurant” (key term).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of filtering collected data based on the key terms and qualifier terms. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Regarding claim 17, Granieri, Carbune, Silva, and Chen in combination disclose all of the limitations of claim 16 as discussed above, and Granieri remains silent on wherein when collecting data from the plurality of onboard sensors related to the request and receiving, via the wireless communication module, data from remote sources related to the request, the system controller is further adapted to identify, using the machine learning model, unspecified key terms and unspecified qualifier terms, related to the request and not included within the request.
Carbune teaches identify, using the machine learning model, unspecified key terms and unspecified qualifier terms, related to the request and not included within the request. See at least col. 9, line 60 – col. 10, line 22, wherein the map search data is filtered based on terms not included in the query information 130, such as seafood restaurants (key terms) or restaurants with fast service (qualifier terms).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of filtering collected data based on the key terms and qualifier terms. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Regarding claim 18, Granieri, Carbune, Silva, and Chen in combination disclose all of the limitations of claim 17 as discussed above, and Granieri additionally teaches wherein when collecting data from the plurality of onboard sensors related to the request and receiving, via the wireless communication module, data from remote sources related to the request, the system controller is further adapted to: identify an extended geographic area within which relevant data related to the request, will be collected; collect relevant data related to the request within the extended geographic area. See at least [0059]-[0060] and figure 4, step 450, wherein the occupant’s request is correlated with a target. The target is a portion of the surrounding environment of the occupant which contains the objects/areas of the occupant’s interest. The target is obtained by collecting map data within a virtual shape defining the target area. See at least [0061], wherein the target area is larger in size, or extended, based on the occupant’s request.
Granieri remains silent on including unspecified key terms and unspecified qualifier terms, and filter the collected data based on the key terms, qualifier terms, unspecified key terms and unspecified qualifier terms.
Carbune teaches including unspecified key terms and unspecified qualifier terms, and filter the collected data based on the key terms, qualifier terms, unspecified key terms and unspecified qualifier terms. See at least col. 9, line 60 – col. 10, line 22, wherein the map search data is filtered based on terms not included in the query information 130, such as seafood restaurants (key terms) or restaurants with fast service (qualifier terms).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of filtering collected data based on the key terms and qualifier terms. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Regarding claim 19, Granieri, Carbune, Silva, and Chen in combination disclose all of the limitations of claim 18 as discussed above, and Granieri additionally teaches wherein: when formulating, with a large language model (LLM) in communication with the system controller via the wireless communication module, a response, and actuating systems within the vehicle to automatically provide, via the HMI, infotainment content for the occupant within the vehicle, the system controller is further adapted to at least one of display a textual language response on a touch screen display of the HMI, and broadcast, via a speaker associated with the HMI, a verbal response for the occupant. See at least [0067], figure 4, step 490, and figures 6-7, wherein the generated response is either displayed visually through a user interface of the vehicle, or output audially through a sound system of the vehicle.
and when formulating, with a large language model (LLM) in communication with the system controller via the wireless communication module, a response, and actuating systems within the vehicle to automatically control, via an automated driving assistance system (ADAS), operation of the vehicle, the system controller is further adapted to identify, with the LLM and the machine learning model, a desired vehicle operation based on the request, and automatically, via the ADAS, perform the desired vehicle operation. See at least [0069], [0074], [0078]-[0079], [0082], and figures 7A-B, wherein the generated output for the user includes identifying an execution function, and performing the identified function. The function includes ADAS systems such as parking assist functions.
Granieri remains silent on formulate, with the LLM, a natural language response for the occupant.
Carbune teaches formulate, with the LLM, a natural language response for the occupant. See at least col. 10, line 61 – col. 11, line 32, col. 14, lines 34-59 and figure 2, wherein the textual response to the user is generated using LLMs. See at least col. 25, line 66 – col. 26, line 23, wherein the machine learning modules generate natural language output.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of formulating a natural language response for the user using an LLM. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Regarding claim 20, Granieri teaches A vehicle having a system for providing intention-based infotainment. See at least [0009] and figures 1-3.
comprising: a system controller in communication with a plurality of onboard sensors within the vehicle. See at least [0091], [0093], and figure 1, wherein sensor system 120 includes sensors arranged within a passenger cabin of the vehicle 100 and is operatively connected to the processor(s) 110 of the vehicle.
and adapted to: collect data related to intentions of an occupant within the vehicle. See at least [0044]-[0045], [0050]-[0052], and figure 4, step 410, wherein data is collected related to occupants of the vehicle and their inputs, gestures, or commands.
wherein the system controller is adapted to at least one of: collect, with an occupant monitoring system data related to what the occupant is looking at, gestures made by the occupant, and facial expressions of the occupant. See at least [0023], [0041], and [0053], wherein the step of collecting occupant data includes collecting sensor data from the cabin of the vehicle to acquire gestures made by the occupant and facial characterization of the occupant related to the occupant’s mood and emotional state. The gestures include motion indication a direction of the occupants eyes in the surrounding environment.
collect, with the HMI, verbal expressions made by the occupant and manual data input from the occupant. See at least [0054], wherein the step of collecting occupant data includes collecting the occupant’s spoken words, or the occupant’s written request provided digitally at an input system 130 of the vehicle.
and access, with a machine learning model, stored data within a database related to past occurrences of the occupant using the vehicle. See at least [0070]-[0071] and figure 5, wherein, during the steps 510-520 of collecting data related to an occupant’s intended search request, execution module 230 references a historical log of the occupant’s previous intended requests stored in data store 240. See at least [0064] and figure 5, step 530, wherein the historical occupant data is used by machine learning model(s) 290 to identify learned patterns of specific drivers.
develop a request based on the data related to the intentions of the occupant within the vehicle. See at least [0053]-[0054] and figure 4, step 420, wherein the collected occupant data is used to identify a request that the user intends to make.
wherein the system controller is adapted to: detect one of triggering language from the occupant, via the HMI, or a triggering event, via the plurality of onboard sensors. See at least [0053]-[0054] and figure 4, step 420, wherein the step of identifying a request made by the occupant includes detecting a wake word (triggering language) or a gesture (triggering event) using the sensors of the vehicle.
analyze, with the machine learning model, intentions of the occupant based on the data stored within the database. See at least [0070]-[0071] and figure 5, wherein, during the steps 510-520 of collecting data related to an occupant’s intended search request, execution module 230 references a historical log of the occupant’s previous intended requests stored in data store 240. See at least [0064] and figure 5, step 530, wherein the historical occupant data is used by machine learning model(s) 290 to identify learned patterns of specific drivers.
real time data collected by the plurality of onboard sensor and data received from remote sources. See at least [0051], wherein the sensor data 250 is acquired at successive time steps. See at least [0032], [0035], [0043], [0090], and figure 3, wherein the system additionally comprises a wireless communication module 180, and retrieves map data from a cloud-based environment. See at least [0058] and figure 3, steps 410, 420, and 430, wherein the data collected in these steps additionally includes map data.
collect data from the plurality of onboard sensors related to the request. See at least [0057] and figure 4, steps 430-440, wherein additional voice or image sensor data is collected based on the identified request.
and receive, via a wireless communication module, data from remote sources related to the request. See at least [0032], [0035], [0043], [0090], and figure 3, wherein the system additionally comprises a wireless communication module 180, and retrieves map data from a cloud-based environment. See at least [0058] and figure 3, steps 410, 420, and 430, wherein the data collected in these steps additionally includes map data.
wherein the system controller is adapted to: identify a geographic area within which relevant data related to the request will be collected; collect relevant data related to the request within the geographic area. See at least [0059]-[0060] and figure 4, step 450, wherein the occupant’s request is correlated with a target. The target is a portion of the surrounding environment of the occupant which contains the objects/areas of the occupant’s interest. The target is obtained by collecting map data within a virtual shape defining the target area.
identify an extended geographic area within which relevant data related to the request, will be collected; collect relevant data related to the request within the extended geographic area. See at least [0059]-[0060] and figure 4, step 450, wherein the occupant’s request is correlated with a target. The target is a portion of the surrounding environment of the occupant which contains the objects/areas of the occupant’s interest. The target is obtained by collecting map data within a virtual shape defining the target area. See at least [0061], wherein the target area is larger in size, or extended, based on the occupant’s request.
formulate, with a language model in communication with the system controller via the wireless communication module, a response. See at least [0065]-[0066] and figure 4, step 480, wherein the occupant’s search request is provided to a neural network language model to generate an answer to the search query. The model can be located remotely from the vehicle, as part of the cloud-computing environment 300.
and actuate systems within the vehicle to automatically: provide, via the HMI, infotainment content for the occupant within the vehicle. See at least [0067] and figure 4, step 490, wherein the search request results are provided to the occupant, including controlling the infotainment system of the vehicle and communicating the search results to the occupant via a user interface.
and control, via an automated driving assistance system (ADAS), operation of the vehicle. See at least [0067]-[0069] and figure 4, step 490, wherein the response to the search request includes controlling an autonomous driving system of the vehicle, for example, executing a parking assist function.
Granieri remains silent on predict, analyze, with the LLM, the data related to the intentions of the occupant within the vehicle; identify, with the machine learning model and the LLM, key terms and qualifier terms; and quantify, using the LLM, the machine learning model, the key terms and qualifier terms; identify, using the machine learning model, unspecified key terms and unspecified qualifier terms, related to the request and not included within the request; including unspecified key terms and unspecified qualifier terms, and filter the collected data based on the key terms, qualifier terms, unspecified key terms and unspecified qualifier terms; and an LLM. As discussed above, Granieri discloses using a neural network language model to respond to the occupant’s search request. drainer additionally does not teach the specifics of performing the infotainment and ADAS steps in one response. As discussed above, Granieri discloses the two steps as separate responses to subsequent requests.
Carbune teaches predict intentions of the occupant based on the data stored within the database. See at least col. 8, line 54 – col. 9, line 17, wherein the module 118 predicts a user’s query intent based on information using predictive analysis and historical user behavior. Additionally, see at least col. 16, line 63 – col. 17, line 15, wherein user historical information 304is stored in service information 122.
analyze the data related to the intentions of the occupant within the vehicle. See at least col. 8, line 54 – col. 9, line 17, wherein the module 118 predicts a user’s query intent and generates a query 131 with query information 130. See at least col. 9, lines 35-47, wherein the query information 130 is analyzed by POI selector 132.
identify key terms and qualifier terms. See at least col. 8, lines 43-55, wherein the query information 130 includes terms that describe a type of POI and matching criteria for the POI (e.g. expensive, within walking distance, local, suitable for young children, etc.). Per [0069] of Applicant’s specification, these qualify as the key terms and qualifier terms, respectively.
and quantify, with the system controller, the key terms and qualifier terms. See at least col. 9, lines 48-59, wherein, upon identifying a qualifier term (“within walking distance”), the POI selector 132 uses user data to quantify the term as an objective distance.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s technique of predicting occupant intentions based on stored data, analyzing the stored intention data, identifying key terms and qualifier terms, and quantifying the key terms and qualifier terms. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
identify, using the machine learning model, unspecified key terms and unspecified qualifier terms, related to the request and not included within the request. See at least col. 9, line 60 – col. 10, line 22, wherein the map search data is filtered based on terms not included in the query information 130, such as seafood restaurants (key terms) or restaurants with fast service (qualifier terms).
including unspecified key terms and unspecified qualifier terms, and filter the collected data based on the key terms, qualifier terms, unspecified key terms and unspecified qualifier terms. See at least col. 9, line 60 – col. 10, line 22, wherein the map search data is filtered based on terms not included in the query information 130, such as seafood restaurants (key terms) or restaurants with fast service (qualifier terms).
formulating a response using a large language model (LLM). See at least col. 10, line 50 – col. 11, line 10 and figures 1-2, wherein a large language model is used to generate textual content in response to a determined user query.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to modify Granieri with Carbune’s large language model. It would have been obvious to modify because doing so enables users to evaluate a large number of points of interest at a time while reducing computational resource utilization, as recognized by Carbune (see at least col. 4, line 34 – col. 5, line 16).
Silva teaches actuating vehicle systems to automatically provide infotainment content and control operation of the vehicle. See at least [0025]-[0026], wherein the LLM 240, in response to a driver input, outputs one or more assistive actions, including operational actions (controlling steering, acceleration, and braking) and infotainment actions (advising the driver or warning the driver). See at least [0028], wherein, in an example, when a request corresponding to a driver’s intention of entering a strip mall is made, the system responds by slowing down the vehicle while alerting the driver (via speech or test).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to further modify Granieri with Silva’s technique of providing content for drivers while additionally controlling operation of the vehicle. It would have been obvious to modify because doing so enables vehicular virtual assistants to better assist drivers and other vehicle occupants.
Chen teaches analyzing with the LLM, identifying terms with the machine learning model and the LLM. See at least [0060]-[0062], wherein model 330 is a machine learning model comprising a large language model. See at least [0101]-[0106] and figure 4, step 501, wherein a user’s search request is input to a generative language model to obtain classification terms (key terms) such as a “city C”, “restaurant”, “Cantonese restaurant”, etc. Additionally, see at least [0107]-[0112] and figure 4, step 501, wherein the generative language model additionally obtains requirement description terms (qualifier terms) which comprise natural language such as “good environment”, “quiet atmosphere”, “few customers”, “good road conditions”, etc.
and quantifying using the LLM, machine learning model. See at least [0170]-[0171] and figure 9, wherein requirement description information (qualifier terms) are output by generative language model 920 using map search content 911, prompt information 912, and real-time location prompt information 913. See at least [0174]-[0177], wherein the generative language model obtains a true value corresponding to the requirement description information.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to further modify Granieri with Chen’s technique of analyzing data, identifying key terms and qualifier terms, and quantifying using an LLM and machine learning model. It would have been obvious to modify because doing so enables mapping services to utilize generative language models to provide users with better search results, as recognized by Chen (see at least [0053]-[0054]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Selena M. Jin whose telephone number is (408)918-7588. The examiner can normally be reached Monday - Thursday and alternate Fridays, 7:30-4:30 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Faris Almatrahi can be reached at (313) 446-4821. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.M.J./ Examiner, Art Unit 3667
/FARIS S ALMATRAHI/ Supervisory Patent Examiner, Art Unit 3667
Prosecution Insights