Detailed Action
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . See 35 U.S.C. § 100 (note).
Art Rejections
Anticipation
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 21, 24, 28–30, 34, 36 and 41 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Evan King et al., “Get Ready for a Party”: Exploring Smarter Smart Spaces with Help from Large Language Models, arxiv.org, Cornell University Library, 201 Olin Library Cornell University, (24 March 2023) (“King”).
Claim 21 is drawn to “a method.” The following table illustrates the correspondence between the claimed method and the King reference.
Claim 21
The King Reference
“21. A method comprising:
The King Reference describes a system and corresponding method for controlling a smart home using an LLM. King at Abs., §, ¶ 11.
“listening, using at least one audio capture device, for user input to control at least one attribute of an audio device;
King’s system and method includes recording speech containing a voice command to control an audio device—namely, setup a smart speaker to reproduce a playlist. Id. Recording speech inherently requires use of an audio capture device, like a microphone.
“routing the user input through a machine learning (ML) model to determine a control action for the at least one attribute based on the user input,
King’s system and method similarly routes user speech through a machine learning model, such as a large language model (LLM) to infer the meaning in a user’s ambiguous voice command and to determine actions to take, such as reproducing a playlist with a smart speaker. Id. at § I, ¶¶ 2–32.
“the routing including converting the user input into a formatted user input including a context of the user input and format characteristics of a response,
“wherein the formatted user input includes at least three distinct sets of decision layer keys corresponding with distinct layers of the ML model; and
The provision of three distinct sets of decision layer keys that correspond with distinct layers of the ML model requires that the claimed ML model likewise has three decision layers (e.g., top level decisions, wearable audio device type controls, speaker type controls, system state changes, external API response selection controls, text summarizer controls). (Spec. at ¶ 62). Similarly, King formats a user’s input into a set of at least three keys, including user location, locations in a house, devices in each location and available controls for each device. King at § III, ¶¶ 4, 5. Thus, King’s LLM will analyze a user’s input to infer a user’s intent and make multiple decisions concerning (1) where the user is, (2) what devices exist, (3) where the devices, (4) what controls are available for each device and (5) what state each device’s controls should be in according to all the other decisions.
“causing the determined control action to be performed,
King’s system and method includes using the LLM’s analysis to control real devices, such as turning on a stereo and configuring lights to loop through a set of colors. Id. at § I, ¶ 43.
“wherein the ML model need not have been pre-trained with the user input to determine the control action for the at least one attribute of the audio device.”
King’s system and method uses an LLM model that is not trained through fine-tuning or task-specific training. Id. at §, ¶ 3.4
Table 1
For the foregoing reasons, the King reference anticipates all limitations of the claim.
Claim 24 depends on claim 21, and further requires the following:
“wherein determining the control action includes selecting the at least one attribute of the audio device based on inferred intent from the user input.”
Similarly, King describes inferring a user’s intent from a user input, such as user’s speech. King at § 3, ¶ 25. The intent is analyzed through an LLM, like GPT-3, to create a set of control actions for connected devices to adjust their states to match the user’s intent. Id. at § III(B), FIG.2 (prompt engineering)6. For the foregoing reasons, the King reference anticipates all limitations of the claim.
Claim 28 depends on claim 21, and further requires the following:
“further comprising providing an audible response to the user input after determining the control action, wherein the audible response includes a natural language response including a query for an additional user input.”
King similarly describes providing an audible response that seeks additional clarifying information. King at § VI, ¶ 37. For the foregoing reasons, the King reference anticipates all limitations of the claim.
Claim 29 depends on claim 21, and further requires the following:
“wherein the user input relates to controlling one or more attributes of a plurality of audio devices including the audio device.”
King’s system and method parse user speech inputs to determine how to controls attributes of a plurality of audio devices, such as smart speakers, stereos and TVs located throughout a house. King at FIG.2 (depicting a determination to turn on a living room speaker while leaving a bedroom TV turned off). For the foregoing reasons, the King reference anticipates all limitations of the claim.
Claim 30 depends on claim 21, and further requires the following:
“the routing comprising:
“providing a set of controllable attributes for the audio device to the ML model,
“wherein the set of controllable attributes is provided to the ML model: a) prior to the listening, and/or b) with the user input,
“providing a set of audio device context data to the ML model for use in determining the control action for the at least one attribute, and
“defining a format of a response from the ML model including the control action.”
King’s prompt engineering approach provides context concerning the user’s devices and states, or controllable attributes, to an LLM as well as defining and requesting use of a JSON format in responding. King at § III(B), ¶ 28, FIG.2. In this regard, King’s system and method provides context to the LLM along with a user’s input. Id. For the foregoing reasons, the King reference anticipates all limitations of the claim.
Claim 34 depends on claim 21, and further requires the following:
“wherein the ML model is cloud-based and includes at least one of, a large language model (LLM) or a large action model (LAM).”
The King reference describes its system and method with reference to a cloud-based LLM, such as GPT-3 used by the ChatGPT chatbot (i.e., AVR) system. King at § III(B), ¶ 3, FIG.2. For the foregoing reasons, the King reference anticipates all limitations of the claim.
Claim 41 depends on claim 21, and further requires the following:
“wherein the ML model includes at least three layers selected from: i) action routing, ii) wearable audio device type controls, iii) speaker type controls, iv) system state changes, v) external application programming interface (API) response selection controls, and vi) text summarizer controls.”
King formats a user’s input into a set of at least three keys, including user location, locations in a house, devices in each location and available controls for each device. King at § III, ¶¶ 4, 5. Thus, King’s LLM will analyze a user’s input to infer a user’s intent and make multiple decisions concerning (1) where the user is, (2) what devices exist, (3) where the devices, (4) what controls are available for each device and (5) what state each device’s controls should be in according to all the other decisions. Decisions 1–3 correspond to an action routing decision. Decision (4) corresponds to speaker type controls. Decision (5) corresponds to system state changes. For the foregoing reasons, the King reference anticipates all limitations of the claim.
Claim 36 is drawn to “an audio device.” The following table illustrates the correspondence between the claimed device and the King reference.
Claim 36
The King Reference
“36. An audio device, comprising:
The King Reference describes a system and corresponding method for controlling a smart home using an LLM. King at Abs., §, ¶ 19.
“an electro-acoustic transducer;
“at least one microphone; and
“a processor coupled with the electro-acoustic transducer and the at least one microphone,
King describes the use of a smart assistant to access ChatGPT. Id. at § III(B), ¶ 1, FIG.2. One of ordinary skill would have recognized ChatGPT as a reference to a particular web-based, automatic speech recognizing (ASR) chatbot. Since ChatGPT is web-based, it inherently requires that the smart assistant is a computer with a processor coupled to a microphone to record user speech and transmit the speech to the ChatGPT service. See id. at § I, ¶ 1, FIG.2 (describing recording user speech). The processor is also inherently coupled with an electro-acoustic transducer, such as a smart speaker, since the machine-readable responses from ChatGPT are used to control real devices in a user’s home. See id. at § III, ¶ 2.
“the processor programmed to:
“listen, using the at least one microphone, for user input to control at least one attribute of the audio device;
King’s system and method includes recording speech containing a voice command to control an audio device—namely, setup a smart speaker to reproduce a playlist. Id. at § 1, ¶ 1. Recording speech inherently requires use of an audio capture device, like a microphone.
“rout the user input through a machine learning (ML) model to determine a control action for the at least one attribute based on the user input, the routing including:
King’s system and method similarly routes user speech through a machine learning model, such as a large language model (LLM) to infer the meaning in a user’s ambiguous voice command and to determine actions to take, such as reproducing a playlist with a smart speaker. Id. at § I, ¶¶ 2–310.
“providing a set of controllable attributes for the audio device to the ML model, wherein the set of controllable attributes is provided to the ML model; a) prior to the listening, and/or b) with the user input,
King’s routing of the user’s query through an LLM includes modifying the query to include additional context along with the user input. Id. at § 3, ¶¶ 3–7, FIGs.1, 2. One piece of context includes controllable attributes—for example, power state and volume. Id.
“providing a set of audio device context data to the ML model for use in determining the control action for the at least one attribute, and
Another piece of context includes the current state of an audio device—for example, state: on and volume: 30. Id.
“defining a format of a response from the ML model including the control action; and
King forms the query in JSON format (i.e., characterized by objects formed by curly braces, arrays formed by square braces and a set of key:value pairs) and requests a similar JSON-formatted response. Id.
“cause the determined control action to be performed,
King’s system and method includes using the LLM’s analysis to control real devices, such as turning on a stereo and configuring lights to loop through a set of colors. Id. at § I, ¶ 411.
“wherein the ML model need not have been pre-trained with the user input to determine the control action for the at least one attribute of the audio device.”
King’s system and method uses an LLM model that is not trained through fine-tuning or task-specific training. Id. at §, ¶ 3.12
Table 2
For the foregoing reasons, the King reference anticipates all limitations of the claim.
Obviousness
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 22, 23 and 37 are rejected under 35 U.S.C. § 103 as being unpatentable over the combination of King and US Patent Application Publication 2019/0035397 (published 31 January 2019) (“Reily”).
Claims 39 and 43 are rejected under 35 U.S.C. § 103 as being unpatentable over the combination of King and US Patent Application Publication 2022/0328039 (published 13 October 2022) (“Avijeet”).
Claims 25, 44 and 45 are rejected under 35 U.S.C. § 103 as being unpatentable over the combination of King; Avijeet and US Patent Application Publication 2022/0301562 (published 22 September 2022) (“Robert”).
Claim 33 is rejected under 35 U.S.C. § 103 as being unpatentable over the combination of King and US Patent Application Publication 2022/0150129 (published 12 May 2022) (“Kim”).
Claims 42, 46 and 47 are rejected under 35 U.S.C. § 103 as being unpatentable over King.
Claim 22 depends on claim 21, and further requires the following:
“wherein the audio capture device performs the listening without requiring a wake word.”
Claim 37 depends on claim 36, and further requires the following:
“wherein the at least one microphone performs the listening without requiring a wake word, or after detecting a user command.”
Claims 22 and 37 are treated together because they both are drawn to features of a voice user interface. As shown in the anticipation rejection of claim 1, incorporated herein, the King reference describes a system and method that leverages prompt engineering to provide context that supplements a user’s spoken voice command. King at § III(B). The prompt engineering enables a general-purpose LLM to infer the user’s intent and provide a formatted response to control smart devices throughout the user’s home without fine-tuning or further training of the LLM. Id. The King reference does not anticipate this claim because it does not describe how a user may initiate a session with an LLM aside from reference to the ChatGPT chatbot.
The Reily reference describes that conventional speech processing engines typically require utterance of a wake word. Reily at ¶¶ 63–78. Reily teaches that this approach leads to frustration in situations where the engine requires the user to provide additional context since a user has to repeatedly initiate a session with the engine. Id. Riley teaches and suggests an improved interface that eliminates the need for a wake word when requesting additional context. Id. Based on this teaching, one of ordinary skill would have reasonably modified King’s system and method to use an improved interface like the one taught in detail by the Reily reference. One of ordinary skill would have reasonably expected that the interface would provide a more seamless and less frustrating user experience by eliminating the need to repetitively recite a wake word to initiate a session with an engine, like King’s LLM. For example, when King’s LLM requests for clarification of an ambiguous voice command, King’s voice interface will simply reproduce a natural language request and begin listening for a follow-up user input. See Reily at ¶¶ 70–78; King at § 6, ¶ 3. For the foregoing reasons, the combination of the King and the Reily references makes obvious all limitations of the claims.
Claim 23 depends on claim 21, and further requires the following:
“wherein the audio capture device performs the listening after detecting a user command.”
As shown in the anticipation rejection of claim 1, incorporated herein, the King reference describes a system and method that leverages prompt engineering to provide context that supplements a user’s spoken voice command. King at § III(B). The prompt engineering enables a general-purpose LLM to infer the user’s intent and provide a formatted response to control smart devices throughout the user’s home without fine-tuning or further training of the LLM. Id. The King reference does not anticipate this claim because it does not describe how a user may initiate a session with an LLM.
The Reily reference describes that conventional speech processing engines typically require utterance of a wake word. Reily at ¶¶ 63–78. Reily teaches that this approach leads to frustration in situations where the engine requires the user to provide additional context since a user has to repeatedly initiate a session with the engine. Id. Riley teaches and suggests an improved interface that eliminates the need for a wake word when requesting additional context. Id. Based on this teaching, one of ordinary skill would have reasonably modified King’s system and method to use an improved interface like the one taught in detail by the Reily reference. One of ordinary skill would have reasonably expected that the interface would provide a more seamless and less frustrating user experience by eliminating the need to repetitively recite a wake word to initiate a session with an engine, like King’s LLM. For example, when King’s LLM requests for clarification of an ambiguous voice command (i.e., a detected user command), King’s voice interface will simply reproduce a natural language request and begin listening for a follow-up user input. See Reily at ¶¶ 70–78; King at § 6, ¶ 3. For the foregoing reasons, the combination of the King and the Reily references makes obvious all limitations of the claim.
Claim 25 depends on claim 24, and further requires the following:
“wherein the inferred intent is determined based on a nested selection approach that includes,
“applying a local portion of the ML model run on the at least one audio capture device or the audio device to determine the control action, and
“if the at least one attribute of the audio device is not selected by applying the local portion of the ML model, applying an off-device portion of the ML model to determine the control action,
“wherein the nested selection approach includes evaluating the inferred intent relative to control functions of the audio device prior to control functions of a service utilized by the audio device, wherein,
“control functions of the audio device enable control of at least one of, transport control, volume, active noise reduction (ANR), audio device grouping, equalization, spatial audio controls, transparency mode, or channel playback, and
“control functions of the service utilized by the audio device enable control of at least one of, a song or a track, an artist, a playlist, or a content channel.”
The King reference describes a system and method that uses a cloud-based AVR, such as ChatGPT, and an LLM, such as OpenAI’s GPT-3. King at § I, ¶ 2, § III(B), ¶ 3, FIG.2. King does not describe a nested selection approach.
The Avijeet reference teaches and suggests a further improvement to King’s system and method where the ASR system is split into a local engine and a remote engine. Avijeet at ¶¶ 38, 40. According to Avijeet, the remote engine may be a large, cloud-based engine with a large vocabulary or model while the local engine is a specialized model corresponding to the most common queries at the local site, such as a user’s personal device. Id. Avijeet teaches that this split arrangement may improve latency by eliminating remote connections in the most common circumstances. See id. at ¶ 38. Avijeet also suggests that a smaller model may be more accurate. Id. at ¶ 82. Accordingly, it would have been obvious for one of ordinary skill to modify King’s system and method to similarly utilize a split, or nested, approach to ASR. A large, remote engine would use a full LLM to perform King’s operations. A small, local engine would implement a subset of the LLM to address the most common queries made by the user at the user’s device.
The local device (e.g., King’s smart home assistant) will receive speech input and process the speech input to derive the user’s intent and generate additional context about the user’s devices to produce a formatted response to control smart devices in the user’s home. See King at § III(B). The combination of King and Avijeet do not address the failure condition of what to do when the local model is unable to derive an adequate response to the user’s input.
The Robert reference, however, further teaches and suggests determining if a local model is capable of properly transcribing a user’s input. Robert at ¶ 35, FIG.5. If not, the input is forwarded to a remote server. Id. Though applied in a speech-to-text content, and not an intent analysis, the teachings of Robert reasonably suggest applying a similar technique to the King-Avijeet system and method in order to improve system robustness when a specialized local model fails to address a novel situation it has not been tuned for. For example, just like when Robert’s local model fails to understand all words in a user’s utterance, when a custom, localized version of King’s LLM receives a new command that is not among the set of common commands that the local LLM was tuned for, the local model will evaluate its ability to respond to a user’s voice command by gauging how well it understands a new command. If the model fails to accurately recognize the user’s intent, the local model will forward the user’s input to a full, remote model capable of determining the user’s intent.
The obviousness rejections of claims 43 and 44, incorporated herein, shows the obviousness of modifying King’s system and method to employ a nested selection approach, such that a local model will be queried before a remote model is queried to control the state of audio devices, including adjusting volume and selecting playlists. See King at § 1, ¶ 1, § IV, ¶ 3. In that case, through the normal learning operations of the King-Avijeet-Robert system, common requests will be processed by a local model prior to uncommon requests processed by a remote model. See Avijeet at ¶ 40. If the user frequently adjusts volume controls (e.g., “increase volume”) but infrequently adjusts a song, track, artist, playlist or content channel (e.g., “play playlist X”), volume controls will be processed prior to those other controls. See Avijeet at ¶ 40 (describing the use of a local engine for the most common questions). For the foregoing reasons, the combination of the King, Avijeet and Robert references makes obvious all limitations of the claim.
Claim 33 depends on claim 21, and further requires the following:
“wherein the ML model is run on the at least one audio capture device or the audio device,
“wherein the ML model includes a function-limited operational mode,
“wherein in response to detecting a threshold latency in network communication, the method includes running the ML model in the function-limited operational mode on the at least one audio capture device or the audio device.”
The obviousness rejection of claim 23, incorporated herein, shows the obviousness of running at least a portion of King’s LLM on a user’s local device used to capture a user’s speech input. King does not describe a function-limited operation mode or the use of the function-limited operation mode when a threshold latency is detected in network communication as claimed.
The Kim reference teaches and suggests implementing an inference model with a combination of models configured for different levels of accuracy and responsiveness to meet time constraints. Kim at ¶¶ 3–12, 32, 36, FIG.1. A front-end system selects an appropriate model based on constraints, such as latency requirements. Id. Kim teaches that this arrangement allows for the dynamic optimization of generating a response to a user’s request by balancing accuracy and latency. Id. Accordingly, it would have been obvious for one of ordinary skill in the art to have modified King’s system to similarly utilize a plurality of LLMs configured with different accuracies and latencies. King’s system would select an appropriate LLM based on a user’s latency requirements. For example, King’s system would utilize a local model that is not as accurate as a remote model because the latency of the remote model is too great (i.e., beyond a tolerable threshold for a response). For the foregoing reasons, the combination of the King and the Kim references makes obvious all limitations of the claim.
Claim 42 depends on claim 41, and further requires the following:
“wherein at least one response includes a JSON responding with keys including: Action, Data, and FriendlyResponse.”
Similarly, King’s LLM responds with a JSON-formatted response that includes action:data keys. And as shown in the obviousness rejection of claim 46, King also teaches and suggests providing a friendly response to keep the user informed. For the foregoing reasons, the King reference makes obvious all limitations of the claim.
Claim 46 depends on claim 36, and further requires the following:
“wherein the set of controllable attributes are separated into distinct groups, and
“wherein:
“if a match for the user input exists in a group of controllable attributes, a positive response is provided related to the control action, and
“if no match for the user input exists in any group of controllable attributes, a null response is provided that is distinct from the positive response.”
The King reference describes increasing the robustness of its system by training a system to recognize when an LLM’s response does not comport with control constraints (e.g., when an LLM makes invalid changes to a device state), and to prompt the user for additional information. King at 0╢ VI, ¶ 3. This reasonably suggest modifying King’s system to review an LLM’s responses as claimed. For example, an LLM may infer from a user’s input a desire to adjust a set of allowable controllable attributes (e.g., turning power on/off). Id. In some cases, however, the LLM may infer from a users’ input a desire to adjust a set of unallowed controllable attributes (e.g., turn up the volume of a toaster). Id. The former case includes a match between a user input and a group of controllable attributes. The latter cases includes a mismatch, or no match, between a user input and a group of controllable attributes. In the latter, case, King suggests providing a null response asking for the user to clarify his intent. Id. And while King does not describe providing a positive response in the former case (e.g., “understood, I am turning on the lights in your bedroom”), doing so is a reasonable extension of King’s disclosure of providing null responses in order to keep the user fully informed about system operation. For the foregoing reasons, the King reference makes obvious all limitations of the claim.
Claim 47 depends on claim 46, and further requires the following:
“wherein null responses are provided iteratively until the match for the user input is identified in the group of controllable attributes, whereby the null responses enhance changes of identifying an intended attribute for the user.”
The obviousness rejection of claim 46, incorporated herein, shows that King describes embodying a robust error handling scheme that provides a null response to a user to improve the system’s ability to infer the user’s intent and provide a set of controllable attributes. The very nature of King’s error handling, which relies on a feedback between the system and the user to refine the user’s intent, reasonably suggests repeating the error handling process as many times as necessary until a user’s intent is properly inferred and used to generate a set of controllable attributes. For the foregoing reasons, the King reference makes obvious all limitations of the claim.
Claim 43 is drawn to “a method.” The following table illustrates the correspondence between the claimed method and the King reference.
Claim 43
The King Reference
“43. A method comprising:
The King Reference describes a system and corresponding method for controlling a smart home using an LLM. King at Abs., §, ¶ 1.
“listening, using at least one audio capture device, for user input to control at least one attribute of an audio device;
King’s system and method includes recording speech containing a voice command to control an audio device—namely, setup a smart speaker to reproduce a playlist. Id. Recording speech inherently requires use of an audio capture device, like a microphone.
“routing the user input through a machine learning (ML) model to determine a control action for the at least one attribute based on the user input; and
King’s system and method similarly routes user speech through a machine learning model, such as a large language model (LLM) to infer the meaning in a user’s ambiguous voice command and to determine actions to take, such as reproducing a playlist with a smart speaker. Id. at § I, ¶¶ 2–3.
“causing the determined control action to be performed,
King’s system and method includes using the LLM’s analysis to control real devices, such as turning on a stereo and configuring lights to loop through a set of colors. Id. at § I, ¶ 4.
“wherein the ML model need not have been pre-trained with the user input to determine the control action for the at least one attribute of the audio device,
King’s system and method uses an LLM model that is not trained through fine-tuning or task-specific training. Id. at §, ¶ 3.13
“wherein determining the control action includes selecting the at least one attribute of the audio device based on inferred intent from the user input, wherein the inferred intent is determined based on a nested selection approach.”
The King reference describes a system and method that uses a cloud-based AVR, such as ChatGPT, and an LLM, such as OpenAI’s GPT-3. King at § I, ¶ 2, § III(B), ¶ 3, FIG.2. King does not describe a nested selection approach.
Table 3
The Avijeet reference teaches and suggests a further improvement to King’s system and method where the ASR system is split into a local engine and a remote engine. Avijeet at ¶¶ 38, 40, 104. According to Avijeet, the remote engine may be a large, cloud-based engine with a large vocabulary or model while the local engine is a specialized model that represents only a cache of the remote engine, where that cache includes only the most common queries at the local site, such as a user’s personal device. Id. Avijeet teaches that this split arrangement may improve latency by eliminating remote connections in the most common circumstances. See id. at ¶ 38. Avijeet also suggests that a smaller model may be more accurate. Id. at ¶ 82. Accordingly, it would have been obvious for one of ordinary skill to modify King’s system and method to similarly utilize a split, or nested, approach to ASR. A large, remote engine would use a full LLM to perform King’s operations. Meanwhile, a small, local engine would act as a cache of the remote engine to implement a subset of the LLM to address the most common queries made by the user at the user’s device. Thus, consistent with the caching concept, it would have been obvious to use the local engine for common requests while using the remote engine for other requests. For the foregoing reasons, the combination of the King and the Avijeet references makes obvious all limitations of the claim.
Claim 44 depends on claim 43, and further requires the following:
“wherein the nested selection approach includes,
“applying a local portion of the ML model run on the at least one audio capture device or the audio device to determine the control action, and
“if the at least one attribute of the audio device is not selected by applying the local portion of the ML model, applying an off-device portion of the ML model to determine the control action.”
The obviousness rejection of claim 43, incorporated herein, shows the obviousness of modifying King’s system and method to run a portion (i.e., a commonly used section) of an LLM on a user’s local device. The local device (e.g., King’s smart home assistant) will receive speech input and process the speech input to derive the user’s intent and generate additional context about the user’s devices to produce a formatted response to control smart devices in the user’s home. See King at § III(B). The combination of King and Avijeet do not address the failure condition of what to do when the local model is unable to derive an adequate response to the user’s input.
The Robert reference, however, further teaches and suggests determining if a local model is capable of properly transcribing a user’s input. Robert at ¶ 35, FIG.5. If not, the input is forwarded to a remote server. Id. Though applied in a speech-to-text content, and not an intent analysis, the teachings of Robert reasonably suggest applying a similar technique to the King-Avijeet system and method in order to improve system robustness when a specialized local model fails to address a novel situation it has not been tuned for. For example, just like when Robert’s local model fails to understand all words in a user’s utterance, when a custom, localized version of King’s LLM receives a new command that is not among the set of common commands that the local LLM was tuned for, the local model will evaluate its ability to respond to a user’s voice command by gauging how well it understands a new command. If the model fails to accurately recognize the user’s intent, the local model will forward the user’s input to a full, remote model capable of determining the user’s intent. For the foregoing reasons, the combination of the King, Avijeet and Robert references makes obvious all limitations of the claim.
Claim 45 depends on claim 43, and further requires the following:
“wherein the nested selection approach includes
“evaluating the inferred intent relative to control functions of the audio device prior to control functions of a service utilized by the audio device, wherein, control functions of the audio device enable control of at least one of, transport control, volume, active noise reduction (ANR), audio device grouping, equalization, spatial audio controls, transparency mode, or channel playback, and
“control functions of the service utilized by the audio device enable control of at least one of, a song or a track, an artist, a playlist, or a content channel.”
The obviousness rejections of claims 43 and 44, incorporated herein, shows the obviousness of modifying King’s system and method to employ a nested selection approach, such that a local model will be queried before a remote model is queried to control the state of audio devices, including adjusting volume and selecting playlists. See King at § 1, ¶ 1, § IV, ¶ 3. In that case, through the normal learning operations of the King-Avijeet-Robert system, common requests will be processed by a local model prior to uncommon requests processed by a remote model. See Avijeet at ¶ 40. If the user frequently adjusts volume controls (e.g., “increase volume”) but infrequently adjusts a song, track, artist, playlist or content channel (e.g., “play playlist X”), volume controls will be processed prior to those other controls. See Avijeet at ¶ 40 (describing the use of a local engine for the most common questions). For the foregoing reasons, the combination of the King, Avijeet and Robert references makes obvious all limitations of the claim.
Claim 39 depends on claim 36, and further requires the following:
“wherein determining the control action includes selecting the at least one attribute of the audio device based on inferred intent from the user command,
“wherein the inferred intent is determined based on a nested selection approach.”
Similarly, King describes inferring a user’s intent from a user input, such as user’s speech. King at § 3, ¶ 214. The intent is analyzed through an LLM, like GPT-3 used by the ChatGPT chatbot (i.e., AVR), to create a set of control actions for connected devices to adjust their states to match the user’s intent. Id. at § III(B), FIG.2 (prompt engineering)15.
The King reference describes a system and method that uses a cloud-based AVR, such as ChatGPT, and an LLM, such as OpenAI’s GPT-3. King at § I, ¶ 2, § III(B), ¶ 3, FIG.2. King does not describe a nested selection approach.
The Avijeet reference teaches and suggests a further improvement to King’s system and method where the ASR system is split into a local engine and a remote engine. Avijeet at ¶¶ 38, 40. According to Avijeet, the remote engine may be a large, cloud-based engine with a large vocabulary or model while the local engine is a specialized model corresponding to the most common queries at the local site, such as a user’s personal device. Id. Avijeet teaches that this split arrangement may improve latency by eliminating remote connections in the most common circumstances. See id. at ¶ 38. Avijeet also suggests that a smaller model may be more accurate. Id. at ¶ 82. Accordingly, it would have been obvious for one of ordinary skill to modify King’s system and method to similarly utilize a split, or nested, approach to ASR. A large, remote engine would use a full LLM to perform King’s operations. A small, local engine would implement a subset of the LLM to address the most common queries made by the user at the user’s device. For the foregoing reasons, the combination of the King and the Avijeet references makes obvious all limitations of the claim.
Summary
Claims 21–25, 28–30, 33, 34, 36, 37, 39 and 41–47 are rejected under at least one of 35 U.S.C. §§ 102 and 103 as being unpatentable over the cited prior art. In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Response to Applicant’s Arguments
Applicant’s Reply (07 April 2026) has substantively amended all the claims. This Office action has been updated accordingly.
Applicant’s Reply at 7–8 includes comments pertaining to the rejections included in the Non-Final Rejection (12 January 2026) and repeated with updates herein.
Concerning claims 43–45, Applicant comments that Avijeet does not first apply a local ML model to infer intent before attempting to infer intent using an off-device portion of a ML model. (Reply at 7–8). However, Avijeet describes the local ASR engine as a cache of the remote ASR engine, such that the ASR processes only the most common phrases. Avijeet at ¶ 104. Much like other computing caches (e.g., memory caching), Avijeet’s reference to a caching relationship between Avijeet’s local and remote ASR engines plainly conveys to one of ordinary skill that the local ASR engine only operates on common phrases (i.e., cache hits) while the remote ASR engine operates on all other phrases (i.e., cache misses). The Robert reference provides further details as to how that caching operation would operate in practice. Even if Robert does not describe caching, or a nested selection, in the context of intent analysis, it is relevant to the claimed subject matter and the subject matter of Avijeet because it is addressing the problem of coordinating local and remote processing.
Applicant’s additional comments have been considered, but are moot in light of the updated rejections presented in this Office action.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WALTER F BRINEY III whose telephone number is (571)272-7513. The examiner can normally be reached M-F 8 am-4:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Carolyn Edwards can be reached at 571-270-7136. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Walter F Briney III/
/CAROLYN R EDWARDS/Supervisory Patent Examiner, Art Unit 2692
Walter F Briney IIIPrimary ExaminerArt Unit 2692
5/28/2026
1 “An exciting prospect of smart homes at their advent was the potential to reduce user burden by providing seamless, unobtrusive, and “smart” interfaces to everyday devices. While smart assistants have improved significantly over the years with respect to speech recognition [25, 24] and user satisfaction [20, 16], a central challenge remains: how can these assistants be made to respond appropriately to ambiguous user commands that may be influenced by context or are otherwise impossible for system developers to anticipate beforehand? An example of such a command might be a user preparing their home to entertain for guests, who asks their smart assistant to “get ready for a party”. The hope is that the assistant—if it is truly smart—might be able to help by inferring the meaning of the statement and determining how to change the state of available devices in response: perhaps to start up the user’s party playlist on a smart speaker and change their smart lights to a festive color scheme. In practice, however, such a request is beyond the capacity of current smart home systems. Google Home will sadly admit: ;I’m sorry, I didn’t understand.;”
2 “To explore this question, we carry out a feasibility study that places GPT-3 in control of a smart home. We evaluate GPT3’s ability to provide high-quality responses to user commands of varying ambiguity given only a simple prompt and a data structure containing information about devices that it can control. Our results demonstrate that LLMs like GPT can infer the meaning behind ambiguous user commands like “get ready for a party” or “I am tired and I want to sleep” and respond with properly-formatted data describing courses of action, enabling more intuitive control of smart devices.”
3 “An implementation that puts a GPT model in control of real devices, showing that it can intuitively respond to a variety of commands. When told to “set up for a party”, it responds by turning on a stereo and configuring a group of Hue lights to loop through a festive set of colors; given the command “I’m leaving”, it turns off all available devices. We trigger these actions by inputting the LLM’s response directly into smart device APIs.”
4 “We furthermore build a proof-of-concept implementation that puts GPT-3 in control of real devices, showing LLM-driven command inference and action planning can function in practice with no fine-tuning or task-specific training required.”
5 “Qualitatively, we want these courses of action to be shaped by the model successfully inferring (1) the intent behind the user command and (2) the manner in which the state of available devices can be changed to meet the user’s intent. To that end, we first define an abstract schema for capturing smart home context before describing a method for engineering prompts to conversational LLMs that elicit useful, actionable responses.”
6 “Our prompts consist of four segments, as follows: Framing. This portion of the prompt provides direction to the conversational agent about its role in the interaction— it is being asked to make decisions as an AI that controls a smart home. We open with the phrase “You are an AI that controls a smart home.” Context. This informs the agent of the user context and devices available in the environment, which scopes the space of its actions and provides a hint as to the structure of our desired response. We continue the prompt: “Here is the state of the devices in the home, in JSON format: devices Here is information about the user: user ”, where both contexts are formatted as shown earlier. Command. This portion inserts the user command and directs the agent to manipulate the state of devices in response, as follows: “The user issues the command: command . Change the device state as appropriate.” The command is written in natural language, as a user might utter to their smart assistant. Formatting. We close the prompt by requesting the response in JSON format so that it can be easily parsed and input to a relevant smart device API: ‘Provide your response in JSON format.’”
7 “In the case of unsatisfactory responses, it would be beneficial to develop a method for learning user preferences or seeking clarifying information (e.g., “are you tired and want to sleep, or are you tired but need an energy boost?”).”
8 See n.6, above.
9 See n.1, above.
10 See n.2, above.
11 See n.3, above.
12 See n.4, above.
13 “We furthermore build a proof-of-concept implementation that puts GPT-3 in control of real devices, showing LLM-driven command inference and action planning can function in practice with no fine-tuning or task-specific training required.”
14 See n.5, above.
15 See n.6, above.
Prosecution Insights