DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 – 2, 12 – 13 and 23 – 24 are rejected under 35 U.S.C. 103 as being unpatentable over Yue et al. ("Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning"), hereinafter Yue, in view of Chen et al. ("FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"), hereinafter Chen.
Regarding claim 1, Yue discloses a method comprising:
receiving, via a computing device, a query (Abstract, lines 1-5, "Large language models (LLMs) such as GPT-4 have exhibited remarkable performance in a variety of tasks, but this strong performance often comes with the high expense of using paid API services. In this paper, we are motivated to study building an LLM cascade to save the cost of using LLMs, particularly for performing reasoning (e.g., mathematical, causal) tasks."; Section 2.1, lines 8-10, "Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
."; Section 3.1, line 1, "We evaluate our LLM cascade approaches on six datasets"; Providing a question to a large language model reads on receiving a query, and evaluating a large language model approach on datasets demonstrates the use of a computer to implement the large language model.);
causing, based on the query, input of a first prompt to a first model, wherein the first model comprises a first large language model (LLM) (Section 2.1, lines 19-22, "Both LLMs solve the question via few-shot in-context learning, e.g., for the weaker LLM, an answer
A
w
is produced by sampling from
P
L
L
M
w
(
A
w
|
E
1
||
E
2
|| . . . ||
E
M
||Q), where
E
1
||
E
2
|| . . . ||
E
M
||Q denotes a concatenation of M task demonstrations and the input question Q, forming the “prompt input” to the LLM (Brown et al., 2020)."; Forming a prompt input to a large language model from the input question Q reads on causing input of a first prompt to a first model based on the query, wherein the first model comprises a first large language model.);
receiving, via the first model, a first output (Section 2.1, lines 8-10, "Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
."; A large language model obtaining an initial answer reads on receiving a first output via the first model.);
determining, based on the first output, a confidence score, wherein the confidence score is based, at least in part, on a number of times a previous query has been the same as the query, and wherein the confidence score is based, at least in part, on feedback received the number of times the previous query has been the same as the query (Section 1, lines 22-28, "In this work, we proposed to devise this routing decision-maker from a different angle, i.e., the “answer consistency” of the weaker LLM (Wang et al., 2023). This is inspired by the observation that answers from the weaker LLM tend to be consistent in multiple sampling paths when the question is easy, but inconsistent when the question is hard. To implement this idea, we proposed two types of methods, a vote-based method that examines if the agreement of multiple answer samples on the majority-voted answer surpasses a pre-defined confidence threshold, and a verification-based method that checks if the majority-voted answers sampled from different prompts are consistent."; Section 2.2, lines 11-14, "Drawing inspiration from the prior work, we make the following hypothesis: When the weaker LLM samples highly consistent answers for a given question, it reveals a high “confidence” in solving this question and its most consistent answer is likely to be correct; in this case, there is thus no need to invoke the stronger LLM."; Section 2.2, lines 33-42, "The first method calculates the consistency of the weaker LLM’s answer samples by voting. Formally, for a single prompt, we denote the set of answers produced by the weaker LLM for each question Q as (
A
1
w
,
A
2
w
, ...,
A
K
w
), where K is the pre-defined number of samples. When sampling from two different prompts, we denote (
A
11
w
,
A
12
w
, ...,
A
1
K
1
w
) and (
A
21
w
,
A
22
w
, ...,
A
2
K
1
w
) as the answer samples produced by each of them, where
K
1
and
K
2
represent the pre-defined sample size for each prompt setting, respectively. Note that for this method, we do not distinguish answers sampled with a single prompt or multiple prompts (e.g., samples from different prompts have exactly equal weights when voting). The most consistent answer can then be selected as the one that most samples agree with, and this answer will also be regarded as the final answer
A
w
by the weaker LLM."; Calculating the consistency of the weaker LLM’s answer samples reads on determining a confidence score based on an output, and calculating the consistency of the weaker LLM’s answer by voting from a set of answers produced by the weaker LLM for each question reads on the confidence score being based, at least in part, on feedback received the number of times the previous query has been the same as the query, where the set of answers reads on feedback received when a previous query has been the same as the query.);
causing, based on the confidence score not satisfying the threshold, input of a second prompt to the second model, wherein the second prompt is based at least on the query (Section 2.1, lines 1-13, "We leverage a cascade of LLMs to save the cost of in-context LLM reasoning, as illustrated in Figure 1. Specifically, we assume two LLMs. The weaker LLM (denoted as
L
L
M
w
) yields relatively worse performance but is less costly, whereas the stronger LLM (denoted as
L
L
M
s
) enjoys better task performance but is more expensive. Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
. This answer, along with other metadata produced by the weaker LLM, will then be fed to a cascade decision maker to decide whether the answer can be accepted as the final one. If the answer is rejected, the stronger LLM should be invoked to provide a more reliable answer
A
s
"; Section 2.2, lines 45-47, "The larger the s, the more consistent the weaker LLM’s answer samples. In conjunction with a pre-defined threshold value τ, the decision maker accepts the weaker LLM’s most consistent answer
A
w
when s ≥ τ and rejects it otherwise."; Invoking the stronger LLM to provide an answer to a question when the consistency of the weaker LLM is below a pre-defined threshold value reads on causing input of a second prompt to the second model based on the confidence score not satisfying the threshold.);
receiving, via the second model, a second output (Section 2.1, lines 1-13, "We leverage a cascade of LLMs to save the cost of in-context LLM reasoning, as illustrated in Figure 1. Specifically, we assume two LLMs. The weaker LLM (denoted as
L
L
M
w
) yields relatively worse performance but is less costly, whereas the stronger LLM (denoted as
L
L
M
s
) enjoys better task performance but is more expensive. Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
. This answer, along with other metadata produced by the weaker LLM, will then be fed to a cascade decision maker to decide whether the answer can be accepted as the final one. If the answer is rejected, the stronger LLM should be invoked to provide a more reliable answer
A
s
"; Invoking the stronger LLM to provide an answer reads on receiving a second output via the second model.);
and causing, based on the query, the second output to be output via the computing device (Section 2.1, lines 1-13, "We leverage a cascade of LLMs to save the cost of in-context LLM reasoning, as illustrated in Figure 1. Specifically, we assume two LLMs. The weaker LLM (denoted as
L
L
M
w
) yields relatively worse performance but is less costly, whereas the stronger LLM (denoted as
L
L
M
s
) enjoys better task performance but is more expensive. Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
. This answer, along with other metadata produced by the weaker LLM, will then be fed to a cascade decision maker to decide whether the answer can be accepted as the final one. If the answer is rejected, the stronger LLM should be invoked to provide a more reliable answer
A
s
"; Section 3.1, line 1, "We evaluate our LLM cascade approaches on six datasets"; Invoking the stronger LLM to provide an answer to a question reads on causing the second output to be output via the computing device based on the query, and evaluating a large language model approach on datasets demonstrates the use of a computer to implement the large language model.).
Yue does not specifically disclose: determining, based on a comparison of a first performance of the first model and a second performance of a second model, a threshold, wherein the second model comprises a second LLM, and wherein the comparison of the first performance and the second performance comprises a comparison of at least one of accuracy, negative log-likelihood, or perplexity.
Chen teaches:
determining, based on a comparison of a first performance of the first model and a second performance of a second model, a threshold, wherein the second model comprises a second LLM, and wherein the comparison of the first performance and the second performance comprises a comparison of at least one of accuracy, negative log-likelihood, or perplexity (Section 2, lines 5-8, "We consider answering queries via the LLM market, which comprises K different LLM APIs, denoted by
{
f
1
·
}
i
=
1
K
. Each
f
1
: P → A is a function that, given a prompt p from the prompt space P, generates an answer from the answer distribution A. Note that to use LLM APIs, one has to convert each query q to some corresponding prompt first."; Section 3, lines 44-53, "The key components of LLM cascade consist of two elements: (i) a generation scoring function and (ii) an LLM router. The generation scoring function, denoted by g(
·
,
·
) : Q x A → [0, 1], generates a reliability score given a query and an answer produced by an LLM API. The LLM router selects m LLM APIs to include in the list. Let L ϵ
[
K
]
m
denote the indexes of the m APIs selected by the router. Given a new query, it iteratively invokes the ith API in the list to obtain an answer
f
L
i
(q). Then, it uses the scoring function to generate a score g(q,
f
L
i
(q)). It returns the generation if the score is higher than a threshold
τ
i
, and queries the next service otherwise. The scoring function can be obtained by training a simple regression model that learns whether a generation is correct from the query and a generated answer. Learning the selected list L and the threshold vectors can be modeled as a constraint optimization problem"; Learning threshold vectors for a reliability score by iteratively invoking large language models to obtain an answer to a query and generating reliability scores using a scoring function given a query and the answers produced by the large language models reads on determining a threshold based on a comparison of a first performance of the first model and a second performance of a second model, where the second model comprises a second LLM, and where the comparison of the first performance and the second performance comprises a comparison of accuracy.).
Chen is considered to be analogous to the claimed invention because it is in the same field of managing large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue to incorporate the teachings of Chen to learn threshold vectors for a reliability score by iteratively invoking large language models to obtain an answer to a query and generating reliability scores using a scoring function given a query and the answers produced by the large language models. Doing so would allow for combining large language models to use for different queries in order to reduce cost and improve accuracy (Chen; Abstract, lines 1-9).
Regarding claim 2, Yue in view of Chen discloses the method as claimed in claim 1.
Yue further discloses:
wherein the determining a confidence score comprises receiving the confidence score from the first model (Section 2.2, lines 33-42, "The first method calculates the consistency of the weaker LLM’s answer samples by voting. Formally, for a single prompt, we denote the set of answers produced by the weaker LLM for each question Q as (
A
1
w
,
A
2
w
, ...,
A
K
w
), where K is the pre-defined number of samples. When sampling from two different prompts, we denote (
A
11
w
,
A
12
w
, ...,
A
1
K
1
w
) and (
A
21
w
,
A
22
w
, ...,
A
2
K
1
w
) as the answer samples produced by each of them, where
K
1
and
K
2
represent the pre-defined sample size for each prompt setting, respectively. Note that for this method, we do not distinguish answers sampled with a single prompt or multiple prompts (e.g., samples from different prompts have exactly equal weights when voting). The most consistent answer can then be selected as the one that most samples agree with, and this answer will also be regarded as the final answer Aw by the weaker LLM."; Calculating the consistency of the weaker LLM’s answer reads on receiving the confidence score from the first model.).
Regarding claim 12, Yue discloses a method comprising:
receiving, via a computing device, a query (Abstract, lines 1-5, "Large language models (LLMs) such as GPT-4 have exhibited remarkable performance in a variety of tasks, but this strong performance often comes with the high expense of using paid API services. In this paper, we are motivated to study building an LLM cascade to save the cost of using LLMs, particularly for performing reasoning (e.g., mathematical, causal) tasks."; Section 2.1, lines 8-10, "Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
."; Section 3.1, line 1, "We evaluate our LLM cascade approaches on six datasets"; Providing a question to a large language model reads on receiving a query, and evaluating a large language model approach on datasets demonstrates the use of a computer to implement the large language model.);
causing, based on the query, input of a first prompt to a first model (Section 2.1, lines 19-22, "Both LLMs solve the question via few-shot in-context learning, e.g., for the weaker LLM, an answer
A
w
is produced by sampling from
P
L
L
M
w
(
A
w
|
E
1
||
E
2
|| . . . ||
E
M
||Q), where
E
1
||
E
2
|| . . . ||
E
M
||Q denotes a concatenation of M task demonstrations and the input question Q, forming the “prompt input” to the LLM (Brown et al., 2020)."; Forming a prompt input to a large language model from the input question Q reads on causing input of a first prompt to a first model based on the query.);
receiving, via the first model, a first output (Section 2.1, lines 8-10, "Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
."; A large language model obtaining an initial answer reads on receiving a first output via the first model.);
determining, based on the first output, a confidence score (Section 1, lines 22-28, "In this work, we proposed to devise this routing decision-maker from a different angle, i.e., the “answer consistency” of the weaker LLM (Wang et al., 2023). This is inspired by the observation that answers from the weaker LLM tend to be consistent in multiple sampling paths when the question is easy, but inconsistent when the question is hard. To implement this idea, we proposed two types of methods, a vote-based method that examines if the agreement of multiple answer samples on the majority-voted answer surpasses a pre-defined confidence threshold, and a verification-based method that checks if the majority-voted answers sampled from different prompts are consistent."; Section 2.2, lines 11-14, "Drawing inspiration from the prior work, we make the following hypothesis: When the weaker LLM samples highly consistent answers for a given question, it reveals a high “confidence” in solving this question and its most consistent answer is likely to be correct; in this case, there is thus no need to invoke the stronger LLM."; Section 2.2, lines 33-42, "The first method calculates the consistency of the weaker LLM’s answer samples by voting. Formally, for a single prompt, we denote the set of answers produced by the weaker LLM for each question Q as (
A
1
w
,
A
2
w
, ...,
A
K
w
), where K is the pre-defined number of samples. When sampling from two different prompts, we denote (
A
11
w
,
A
12
w
, ...,
A
1
K
1
w
) and (
A
21
w
,
A
22
w
, ...,
A
2
K
1
w
) as the answer samples produced by each of them, where
K
1
and
K
2
represent the pre-defined sample size for each prompt setting, respectively. Note that for this method, we do not distinguish answers sampled with a single prompt or multiple prompts (e.g., samples from different prompts have exactly equal weights when voting). The most consistent answer can then be selected as the one that most samples agree with, and this answer will also be regarded as the final answer
A
w
by the weaker LLM."; Calculating the consistency of the weaker LLM’s answer samples reads on determining a confidence score based on an output.);
causing, based on the confidence score not satisfying the threshold, input of a second prompt to the second model, wherein the second prompt is based at least on the query (Section 2.1, lines 1-13, "We leverage a cascade of LLMs to save the cost of in-context LLM reasoning, as illustrated in Figure 1. Specifically, we assume two LLMs. The weaker LLM (denoted as
L
L
M
w
) yields relatively worse performance but is less costly, whereas the stronger LLM (denoted as
L
L
M
s
) enjoys better task performance but is more expensive. Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
. This answer, along with other metadata produced by the weaker LLM, will then be fed to a cascade decision maker to decide whether the answer can be accepted as the final one. If the answer is rejected, the stronger LLM should be invoked to provide a more reliable answer
A
s
"; Section 2.2, lines 45-47, "The larger the s, the more consistent the weaker LLM’s answer samples. In conjunction with a pre-defined threshold value τ, the decision maker accepts the weaker LLM’s most consistent answer
A
w
when s ≥ τ and rejects it otherwise."; Invoking the stronger LLM to provide an answer to a question when the consistency of the weaker LLM is below a pre-defined threshold value reads on causing input of a second prompt to the second model based on the confidence score not satisfying the threshold.);
receiving, via the second model, a second output (Section 2.1, lines 1-13, "We leverage a cascade of LLMs to save the cost of in-context LLM reasoning, as illustrated in Figure 1. Specifically, we assume two LLMs. The weaker LLM (denoted as
L
L
M
w
) yields relatively worse performance but is less costly, whereas the stronger LLM (denoted as
L
L
M
s
) enjoys better task performance but is more expensive. Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
. This answer, along with other metadata produced by the weaker LLM, will then be fed to a cascade decision maker to decide whether the answer can be accepted as the final one. If the answer is rejected, the stronger LLM should be invoked to provide a more reliable answer
A
s
"; Invoking the stronger LLM to provide an answer reads on receiving a second output via the second model.);
and causing, based on the query, the second output to be output via the computing device (Section 2.1, lines 1-13, "We leverage a cascade of LLMs to save the cost of in-context LLM reasoning, as illustrated in Figure 1. Specifically, we assume two LLMs. The weaker LLM (denoted as
L
L
M
w
) yields relatively worse performance but is less costly, whereas the stronger LLM (denoted as
L
L
M
s
) enjoys better task performance but is more expensive. Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
. This answer, along with other metadata produced by the weaker LLM, will then be fed to a cascade decision maker to decide whether the answer can be accepted as the final one. If the answer is rejected, the stronger LLM should be invoked to provide a more reliable answer
A
s
"; Section 3.1, line 1, "We evaluate our LLM cascade approaches on six datasets"; Invoking the stronger LLM to provide an answer to a question reads on causing the second output to be output via the computing device based on the query, and evaluating a large language model approach on datasets demonstrates the use of a computer to implement the large language model.).
Yue does not specifically disclose: determining, based on a comparison of a first performance of the first model and a second performance of a second model, a threshold.
Chen teaches:
determining, based on a comparison of a first performance of the first model and a second performance of a second model, a threshold (Section 2, lines 5-8, "We consider answering queries via the LLM market, which comprises K different LLM APIs, denoted by
{
f
1
·
}
i
=
1
K
. Each
f
1
: P → A is a function that, given a prompt p from the prompt space P, generates an answer from the answer distribution A. Note that to use LLM APIs, one has to convert each query q to some corresponding prompt first."; Section 3, lines 44-53, "The key components of LLM cascade consist of two elements: (i) a generation scoring function and (ii) an LLM router. The generation scoring function, denoted by g(
·
,
·
) : Q x A → [0, 1], generates a reliability score given a query and an answer produced by an LLM API. The LLM router selects m LLM APIs to include in the list. Let L ϵ
[
K
]
m
denote the indexes of the m APIs selected by the router. Given a new query, it iteratively invokes the ith API in the list to obtain an answer
f
L
i
(q). Then, it uses the scoring function to generate a score g(q,
f
L
i
(q)). It returns the generation if the score is higher than a threshold
τ
i
, and queries the next service otherwise. The scoring function can be obtained by training a simple regression model that learns whether a generation is correct from the query and a generated answer. Learning the selected list L and the threshold vectors can be modeled as a constraint optimization problem"; Learning threshold vectors for a reliability score by iteratively invoking large language models to obtain an answer to a query and generating reliability scores using a scoring function given a query and the answers produced by the large language models reads on determining a threshold based on a comparison of a first performance of the first model and a second performance of a second model.).
Chen is considered to be analogous to the claimed invention because it is in the same field of managing large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue to incorporate the teachings of Chen to learn threshold vectors for a reliability score by iteratively invoking large language models to obtain an answer to a query and generating reliability scores using a scoring function given a query and the answers produced by the large language models. Doing so would allow for combining large language models to use for different queries in order to reduce cost and improve accuracy (Chen; Abstract, lines 1-9).
Regarding claim 13, arguments analogous to claim 2 are applicable.
Regarding claim 23, Yue discloses a method comprising:
receiving, via a computing device, a query (Abstract, lines 1-5, "Large language models (LLMs) such as GPT-4 have exhibited remarkable performance in a variety of tasks, but this strong performance often comes with the high expense of using paid API services. In this paper, we are motivated to study building an LLM cascade to save the cost of using LLMs, particularly for performing reasoning (e.g., mathematical, causal) tasks."; Section 2.1, lines 8-10, "Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
."; Section 3.1, line 1, "We evaluate our LLM cascade approaches on six datasets"; Providing a question to a large language model reads on receiving a query, and evaluating a large language model approach on datasets demonstrates the use of a computer to implement the large language model.);
causing, based on the query, input of a first prompt to a first model (Section 2.1, lines 19-22, "Both LLMs solve the question via few-shot in-context learning, e.g., for the weaker LLM, an answer
A
w
is produced by sampling from
P
L
L
M
w
(
A
w
|
E
1
||
E
2
|| . . . ||
E
M
||Q), where
E
1
||
E
2
|| . . . ||
E
M
||Q denotes a concatenation of M task demonstrations and the input question Q, forming the “prompt input” to the LLM (Brown et al., 2020)."; Forming a prompt input to a large language model from the input question Q reads on causing input of a first prompt to a first model based on the query.);
receiving, via the first model, a first output (Section 2.1, lines 8-10, "Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
."; A large language model obtaining an initial answer reads on receiving a first output via the first model.);
determining, based on the first output, a confidence score (Section 1, lines 22-28, "In this work, we proposed to devise this routing decision-maker from a different angle, i.e., the “answer consistency” of the weaker LLM (Wang et al., 2023). This is inspired by the observation that answers from the weaker LLM tend to be consistent in multiple sampling paths when the question is easy, but inconsistent when the question is hard. To implement this idea, we proposed two types of methods, a vote-based method that examines if the agreement of multiple answer samples on the majority-voted answer surpasses a pre-defined confidence threshold, and a verification-based method that checks if the majority-voted answers sampled from different prompts are consistent."; Section 2.2, lines 11-14, "Drawing inspiration from the prior work, we make the following hypothesis: When the weaker LLM samples highly consistent answers for a given question, it reveals a high “confidence” in solving this question and its most consistent answer is likely to be correct; in this case, there is thus no need to invoke the stronger LLM."; Section 2.2, lines 33-42, "The first method calculates the consistency of the weaker LLM’s answer samples by voting. Formally, for a single prompt, we denote the set of answers produced by the weaker LLM for each question Q as (
A
1
w
,
A
2
w
, ...,
A
K
w
), where K is the pre-defined number of samples. When sampling from two different prompts, we denote (
A
11
w
,
A
12
w
, ...,
A
1
K
1
w
) and (
A
21
w
,
A
22
w
, ...,
A
2
K
1
w
) as the answer samples produced by each of them, where
K
1
and
K
2
represent the pre-defined sample size for each prompt setting, respectively. Note that for this method, we do not distinguish answers sampled with a single prompt or multiple prompts (e.g., samples from different prompts have exactly equal weights when voting). The most consistent answer can then be selected as the one that most samples agree with, and this answer will also be regarded as the final answer
A
w
by the weaker LLM."; Calculating the consistency of the weaker LLM’s answer samples reads on determining a confidence score based on an output.);
and causing, based on the confidence score satisfying the threshold, the first output to be output via the computing device (Section 2.1, lines 1-13, "We leverage a cascade of LLMs to save the cost of in-context LLM reasoning, as illustrated in Figure 1. Specifically, we assume two LLMs. The weaker LLM (denoted as
L
L
M
w
) yields relatively worse performance but is less costly, whereas the stronger LLM (denoted as
L
L
M
s
) enjoys better task performance but is more expensive. Given a question Q, the LLM cascade first employs the weaker LLM to obtain an initial answer
A
w
. This answer, along with other metadata produced by the weaker LLM, will then be fed to a cascade decision maker to decide whether the answer can be accepted as the final one. If the answer is rejected, the stronger LLM should be invoked to provide a more reliable answer
A
s
"; Section 2.2, lines 45-47, "The larger the s, the more consistent the weaker LLM’s answer samples. In conjunction with a pre-defined threshold value τ, the decision maker accepts the weaker LLM’s most consistent answer
A
w
when s ≥ τ and rejects it otherwise."; Section 3.1, line 1, "We evaluate our LLM cascade approaches on six datasets"; Accepting the weaker LLM’s most consistent answer when the consistency of the weaker LLM is above a pre-defined threshold value reads on causing the first output to be output via the computing device based on the confidence score satisfying the threshold, and evaluating a large language model approach on datasets demonstrates the use of a computer to implement the large language model.).
Yue does not specifically disclose: determining, based on a comparison of a first performance of the first model and a second performance of a second model, a threshold.
Chen teaches:
determining, based on a comparison of a first performance of the first model and a second performance of a second model, a threshold (Section 2, lines 5-8, "We consider answering queries via the LLM market, which comprises K different LLM APIs, denoted by
{
f
1
·
}
i
=
1
K
. Each
f
1
: P → A is a function that, given a prompt p from the prompt space P, generates an answer from the answer distribution A. Note that to use LLM APIs, one has to convert each query q to some corresponding prompt first."; Section 3, lines 44-53, "The key components of LLM cascade consist of two elements: (i) a generation scoring function and (ii) an LLM router. The generation scoring function, denoted by g(
·
,
·
) : Q x A → [0, 1], generates a reliability score given a query and an answer produced by an LLM API. The LLM router selects m LLM APIs to include in the list. Let L ϵ
[
K
]
m
denote the indexes of the m APIs selected by the router. Given a new query, it iteratively invokes the ith API in the list to obtain an answer
f
L
i
(q). Then, it uses the scoring function to generate a score g(q,
f
L
i
(q)). It returns the generation if the score is higher than a threshold
τ
i
, and queries the next service otherwise. The scoring function can be obtained by training a simple regression model that learns whether a generation is correct from the query and a generated answer. Learning the selected list L and the threshold vectors can be modeled as a constraint optimization problem"; Learning threshold vectors for a reliability score by iteratively invoking large language models to obtain an answer to a query and generating reliability scores using a scoring function given a query and the answers produced by the large language models reads on determining a threshold based on a comparison of a first performance of the first model and a second performance of a second model.).
Chen is considered to be analogous to the claimed invention because it is in the same field of managing large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue to incorporate the teachings of Chen to learn threshold vectors for a reliability score by iteratively invoking large language models to obtain an answer to a query and generating reliability scores using a scoring function given a query and the answers produced by the large language models. Doing so would allow for combining large language models to use for different queries in order to reduce cost and improve accuracy (Chen; Abstract, lines 1-9).
Regarding claim 24, arguments analogous to claim 2 are applicable.
Claims 3 – 5, 14 – 16 and 25 – 27 are rejected under 35 U.S.C. 103 as being unpatentable over Yue in view of Chen, and further in view of Brown et al. ("Language Models are Few-Shot Learners"), hereinafter Brown.
Regarding claim 3, Yue in view of Chen discloses the method as claimed in claim 1, but does not specifically disclose: wherein the first performance of the first model is based on a nonnegative loss function.
Brown teaches:
wherein the first performance of the first model is based on a nonnegative loss function (Section 3, lines 1-7, 'In Figure 3.1 we display training curves for the 8 models described in Section 2. For this graph we also include 6 additional extra-small models with as few as 100,000 parameters. As observed in [KMH+20], language modeling performance follows a power-law when making efficient use of training compute. After extending this trend by two more orders of magnitude, we observe only a slight (if any) departure from the power-law. One might worry that these improvements in cross-entropy loss come only from modeling spurious details of our training corpus. However, we will see in the following sections that improvements in cross-entropy loss lead to consistent performance gains across a broad spectrum of natural language tasks."; Figure 3.1, lines 1-2, "Performance (measured in terms of cross-entropy validation loss) follows a power-law trend with the amount of compute used for training."; Measuring the performance of a language model in terms of cross-entropy validation loss reads on the performance of a model being based on a nonnegative loss function.).
Brown is considered to be analogous to the claimed invention because it is in the same field of managing large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue in view of Chen to incorporate the teachings of Brown to measure the performance of a language model in terms of cross-entropy validation loss. Doing so would allow for consistent performance gains across a broad spectrum of natural language tasks (Brown; Section 3, lines 1-7).
Regarding claim 4, Yue in view of Chen and further in view of Brown discloses the method as claimed in claim 3.
Brown further teaches:
wherein the second performance of the second model is based on the nonnegative loss function (Section 3, lines 1-7, 'In Figure 3.1 we display training curves for the 8 models described in Section 2. For this graph we also include 6 additional extra-small models with as few as 100,000 parameters. As observed in [KMH+20], language modeling performance follows a power-law when making efficient use of training compute. After extending this trend by two more orders of magnitude, we observe only a slight (if any) departure from the power-law. One might worry that these improvements in cross-entropy loss come only from modeling spurious details of our training corpus. However, we will see in the following sections that improvements in cross-entropy loss lead to consistent performance gains across a broad spectrum of natural language tasks."; Figure 3.1, lines 1-2, "Performance (measured in terms of cross-entropy validation loss) follows a power-law trend with the amount of compute used for training."; Measuring the performance of a language model in terms of cross-entropy validation loss reads on the performance of a model being based on a nonnegative loss function.).
Brown is considered to be analogous to the claimed invention because it is in the same field of managing large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue in view of Chen and further in view of Brown to further incorporate the teachings of Brown to measure the performance of a language model in terms of cross-entropy validation loss. Doing so would allow for consistent performance gains across a broad spectrum of natural language tasks (Brown; Section 3, lines 1-7).
Regarding claim 5, Yue in view of Chen and further in view of Brown discloses the method as claimed in claim 4.
Brown further teaches:
wherein the comparison of the first performance of the first model and the second performance of the second model comprises comparing a first result of a first prediction from the first model applied to the nonnegative loss function with a second result of a second prediction from the second model applied to the nonnegative loss function (Page 6, lines 7-8, "In addition to all the above, we also train a series of smaller models (ranging from 125 million parameters to 13 billion parameters) in order to compare their performance to GPT-3 in the zero, one and few-shot settings."; Section 3, lines 1-7, 'In Figure 3.1 we display training curves for the 8 models described in Section 2. For this graph we also include 6 additional extra-small models with as few as 100,000 parameters. As observed in [KMH+20], language modeling performance follows a power-law when making efficient use of training compute. After extending this trend by two more orders of magnitude, we observe only a slight (if any) departure from the power-law. One might worry that these improvements in cross-entropy loss come only from modeling spurious details of our training corpus. However, we will see in the following sections that improvements in cross-entropy loss lead to consistent performance gains across a broad spectrum of natural language tasks."; Figure 3.1, lines 1-2, "Performance (measured in terms of cross-entropy validation loss) follows a power-law trend with the amount of compute used for training."; Comparing the performance of language models, where the performance of the language models is measured in terms of cross-entropy validation loss, reads on comparing a first result of a first prediction from the first model applied to the nonnegative loss function with a second result of a second prediction from the second model applied to the nonnegative loss function.).
Brown is considered to be analogous to the claimed invention because it is in the same field of managing large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue in view of Chen and further in view of Brown to further incorporate the teachings of Brown to compare the performance of language models, where the performance of the language models is measured in terms of cross-entropy validation loss. Doing so would allow for consistent performance gains across a broad spectrum of natural language tasks (Brown; Section 3, lines 1-7).
Regarding claim 14, arguments analogous to claim 3 are applicable.
Regarding claim 15, arguments analogous to claim 4 are applicable.
Regarding claim 16, arguments analogous to claim 5 are applicable.
Regarding claim 25, arguments analogous to claim 3 are applicable.
Regarding claim 26, arguments analogous to claim 4 are applicable.
Regarding claim 27, arguments analogous to claim 5 are applicable.
Claims 6, 17 and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Yue in view of Chen, and further in view of Chuang et al. ("Learning to Route LLMs with Confidence Tokens"), hereinafter Chuang.
Regarding claim 6, Yue in view of Chen discloses the method as claimed in claim 1, but does not specifically disclose: wherein the first model produces output quicker than the second model based on the same input.
Chuang teaches:
wherein the first model produces output quicker than the second model based on the same input (Section 1, lines 34-37, "In this work, we focus on two downstream tasks that utilize this notion of confidence for practical applications: (1) routing, where queries with low confidence are routed to stronger but more costly LLMs, and (2) rejection, where the model may abstain from making a choice, valuable for preventing unsafe behaviors. rejection-to-answer."; Section 5.1, lines 13-15, "Smaller local models often have lower latency (sec) during inference as well as lower monetary cost. Thus, routing only a fraction of queries to larger, more powerful models can significantly improve the efficiency of the overall system compared to relying entirely on the more powerful models."; Smaller local models having lower latency than larger and more powerful models reads on the first model producing output quicker than the second model based on the same input.).
Chuang is considered to be analogous to the claimed invention because it is in the same field of managing large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue in view of Chen to incorporate the teachings of Chuang to route queries with low confidence to larger and more powerful models, where the smaller local models have lower latency than the larger and more powerful models. Doing so would allow for significantly improving the efficiency of the overall system compared to relying entirely on more powerful models (Chuang; Section 5.1, lines 13-15).
Regarding claim 17, arguments analogous to claim 6 are applicable.
Regarding claim 28, arguments analogous to claim 6 are applicable.
Claims 7 – 8, 18 – 19 and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Yue in view of Chen, and further in view of Zhang et al. ("EdgeShard: Efficient LLM Inference via Collaborative Edge Computing"), hereinafter Zhang.
Regarding claim 7, Yue in view of Chen discloses the method as claimed in claim 1, but does not specifically disclose: wherein a first computing device comprises the first model, wherein a second computing device comprises the second model, and wherein the first computing device comprises less computing power than the second computing device.
Zhang teaches:
wherein a first computing device comprises the first model, wherein a second computing device comprises the second model, and wherein the first computing device comprises less computing power than the second computing device (Abstract, lines 10-18, "In this work, we leverage collaborative edge computing to facilitate the collaboration among edge devices and cloud servers for jointly performing efficient LLM inference. We propose a general framework to partition the LLM model into shards and deploy on distributed devices. To achieve efficient LLM inference, we formulate an adaptive joint device selection and model partition problem and design an efficient dynamic programming algorithm to optimize the inference latency and throughput, respectively."; Section IV, lines 15-18, "We consider a network consisting of M edge devices and cloud servers. The devices have heterogeneous computation and memory capabilities, and cloud servers are much more powerful than edge devices in terms of computation capability."; Partitioning a large language model and deploying the partitions on edge devices and cloud servers reads on a first computing device comprising the first model and a second computing device comprising the second model, and the cloud servers being more powerful than the edge devices in terms of computation capability reads on the first computing device comprising less computing power than the second computing device.).
Zhang is considered to be analogous to the claimed invention because it is in the same field of managing large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue in view of Chen to incorporate the teachings of Zhang to partition a large language model and deploy the partitions on edge devices and cloud servers, where and the cloud servers are more powerful than the edge devices in terms of computation capability. Doing so would allow for optimizing the inference latency and throughput (Zhang; Abstract, lines 10-18).
Regarding claim 8, Yue in view of Chen discloses the method as claimed in claim 1, but does not specifically disclose: wherein the first model resides in one of a gateway, a cable modem, or a set-top box, and wherein the second model resides in one of a server or a cloud computing environment.
Zhang teaches:
wherein the first model resides in one of a gateway, a cable modem, or a set-top box, and wherein the second model resides in one of a server or a cloud computing environment (Abstract, lines 10-18, "In this work, we leverage collaborative edge computing to facilitate the collaboration among edge devices and cloud servers for jointly performing efficient LLM inference. We propose a general framework to partition the LLM model into shards and deploy on distributed devices. To achieve efficient LLM inference, we formulate an adaptive joint device selection and model partition problem and design an efficient dynamic programming algorithm to optimize the inference latency and throughput, respectively."; Section I, lines 32-35, "Edge computing is a promising solution to address the aforementioned challenges by deploying LLMs on edge devices (e.g., edge servers, edge gateways, and mobile phones) at the network edge closer to the data sources"; Partitioning a large language model and deploying the partitions on edge devices and cloud servers, where an edge device is a gateway, reads on the first model residing in a gateway and the second model residing in a server.).
Zhang is considered to be analogous to the claimed invention because it is in the same field of managing large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue in view of Chen to incorporate the teachings of Zhang to partition a large language model and deploy the partitions on edge devices and cloud servers, where an edge device is a gateway. Doing so would allow for optimizing the inference latency and throughput (Zhang; Abstract, lines 10-18).
Regarding claim 18, arguments analogous to claim 7 are applicable.
Regarding claim 19, arguments analogous to claim 8 are applicable.
Regarding claim 29, arguments analogous to claim 7 are applicable.
Claims 9 – 11 and 20 – 22 are rejected under 35 U.S.C. 103 as being unpatentable over Yue in view of Chen, and further in view of Lu et al. (US Patent No. 12,579,974), hereinafter Lu.
Regarding claim 9, Yue in view of Chen discloses the method as claimed in claim 1, but does not specifically disclose: wherein the query comprises an indication of a voice command and wherein the confidence score is based, at least in part, on an interpretation of the voice command.
Lu teaches:
wherein the query comprises an indication of a voice command and wherein the confidence score is based, at least in part, on an interpretation of the voice command (Column 2, line 63 - Column 3, line 8, "A system may receive a user input as speech. For example, a user may speak an input to a device. The device may send audio data, representing the spoken input, to the system. The system may perform ASR processing on the audio data to generate ASR data (e.g., text data, token data, etc.) representing the user input. The system may perform processing on the ASR data to determine an action responsive to the user input. In some embodiments, the system may be configured to process the ASR data (or other type of data representing another type of user input) using one or more language models (e.g., one or more large language models (LLMs)) to determine the action responsive to the user input."; Column 17, lines 30-35, "As described below, with respect to FIG. 10, the ASR component 1050 may determine ASR data that includes an ASR N-best list including multiple ASR hypotheses and corresponding confidence scores representing what the user may have said."; Processing user speech input to determine an action responsive to the user input reads on a query comprising an indication of a voice command, and determining a confidence score representing what the user may have said reads on the confidence score being based, at least in part, on an interpretation of the voice command.).
Lu is considered to be analogous to the claimed invention because it is in the same field of large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue in view of Chen to incorporate the teachings of Lu to process user speech input to determine an action responsive to the user input and determine a confidence score representing what the user may have said. Doing so would allow for reducing latency and use of computational resources when processing using a large language model (Lu; Column 4, lines 48-50).
Regarding claim 10, Yue in view of Chen and further in view of Lu discloses the method as claimed in claim 9.
Lu further teaches:
wherein the confidence score is based, at least in part, on a frequency of the interpretation of the voice command being the same as or similar to previous interpretations of voice commands (Column 3, line 59 - Column 4, line 2, "For an incoming user input, a cache lookup may be performed using a key for context data corresponding to the user input. For a cache hit, the stored output can be used to respond to the user input. A cache hit, as used herein, can mean that the LLM has already previously processed a user input corresponding to the same context data (e.g., a user input received under the same or similar circumstances), and the system can use the previously determined output (stored in the cache) to respond to the current user input that corresponds to the same context data."; Column 9, lines 47-60, "The cache prediction component 160 may receive data from a historic user input data storage 165 representing past user inputs received by the system 100. In some embodiments, the historic user input data storage 165 may store frequent user input data. The frequent user input data may indicate user inputs that are received by the system 100 on a frequent basis (e.g., a threshold number of times over a period of time; a number of times over the past 30 days, a number of times over the past 90 days, etc.). In other embodiments, the historic user input data storage 165 may indicate all different user inputs received by the system 100, where an individual user input is associated with a number indicating how many times it was received by the system 100 during a period of time (a frequency of receiving the user input)."; Using the previously determined output stored in the cache to respond to the current user input that corresponds to the same context data, where frequent user input data is stored in the cache, reads on the confidence score being based, at least in part, on a frequency of the interpretation of the voice command being the same as or similar to previous interpretations of voice commands.).
Lu is considered to be analogous to the claimed invention because it is in the same field of large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue in view of Chen and further in view of Lu to further incorporate the teachings of Lu to use previously determined output stored in a cache to respond to current user input that corresponds to the same context data, where frequent user input data is stored in the cache. Doing so would allow for reducing latency and use of computational resources when processing using a large language model (Lu; Column 4, lines 48-50).
Regarding claim 11, Yue in view of Chen and further in view of Lu discloses the method as claimed in claim 10.
Lu further teaches:
wherein the confidence score is based, at least in part, on a frequency of the interpretation of the voice command being the same as or similar to previous interpretations of voice commands within a time period (Column 3, line 59 - Column 4, line 2, "For an incoming user input, a cache lookup may be performed using a key for context data corresponding to the user input. For a cache hit, the stored output can be used to respond to the user input. A cache hit, as used herein, can mean that the LLM has already previously processed a user input corresponding to the same context data (e.g., a user input received under the same or similar circumstances), and the system can use the previously determined output (stored in the cache) to respond to the current user input that corresponds to the same context data."; Column 9, lines 47-60, "The cache prediction component 160 may receive data from a historic user input data storage 165 representing past user inputs received by the system 100. In some embodiments, the historic user input data storage 165 may store frequent user input data. The frequent user input data may indicate user inputs that are received by the system 100 on a frequent basis (e.g., a threshold number of times over a period of time; a number of times over the past 30 days, a number of times over the past 90 days, etc.). In other embodiments, the historic user input data storage 165 may indicate all different user inputs received by the system 100, where an individual user input is associated with a number indicating how many times it was received by the system 100 during a period of time (a frequency of receiving the user input)."; Using the previously determined output stored in the cache to respond to the current user input that corresponds to the same context data, where frequent user input data is stored in the cache when user input occurs a threshold number of times over a period of time, reads on the confidence score being based, at least in part, on a frequency of the interpretation of the voice command being the same as or similar to previous interpretations of voice commands within a time period.).
Lu is considered to be analogous to the claimed invention because it is in the same field of large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yue in view of Chen and further in view of Lu to further incorporate the teachings of Lu to use previously determined output stored in a cache to respond to current user input that corresponds to the same context data, where frequent user input data is stored in the cache when user input occurs a threshold number of times over a period of time. Doing so would allow for reducing latency and use of computational resources when processing using a large language model (Lu; Column 4, lines 48-50).
Regarding claim 20, arguments analogous to claim 9 are applicable.
Regarding claim 21, arguments analogous to claim 10 are applicable.
Regarding claim 22, arguments analogous to claim 11 are applicable.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Sen et al. (US Patent Application Publication No. 2025/0252319)
Shah et al. (US Patent Application Publication No. 2025/0252032)
Dekoninck et al. ("A Unified Approach to Routing and Cascading for LLMs")
Hao et al. ("Hybrid SLM and LLM for Edge-Cloud Collaborative Inference")
Zhang et al. ("TREACLE: Thrifty Reasoning via Context-Aware LLM and Prompt Selection")
Šakota et al. ("Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling")
Wang et al. ("Tabi: An Efficient Multi-Level Inference System for Large Language Models")
Xu et al. ("CHA: A Caching Framework for Home-based Voice Assistant Systems")
Any inquiry concerning this communication or earlier communications from the examiner should be directed to James Boggs whose telephone number is (571)272-2968. The examiner can normally be reached M-F 8:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571)272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAMES BOGGS/Examiner, Art Unit 2657
Prosecution Insights