Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The Action is responsive to the Amendments and Remarks filed on 5/7/2026. Claims 1-20 are pending claims. Claims 1, 13, and 20 are written in independent form.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-patentable subject matter. The claimed invention is directed to one or more abstract ideas without significantly more. The judicial exception is not integrated into a practical application. The claims do not include additional elements that are sufficient to amount to significantly more than judicial exception. The eligibility analysis in support of these findings is provided below.
As per Claim 1, 13, and 20,
STEP 1:In accordance with Step 1 of the eligibility inquiry (as explained in MPEP 2106), the claimed apparatus (claims 1-12), method (claims 13-19), and non-transitory computer readable medium (claim 20) are directed to one of the eligible categories of subject matter and therefore satisfies Step 1.
STEP 2A Prong One:The independent claims 1, 13, and 20 recite the following limitations directed to an abstract idea:
Extract, from a query, a first element and a second element;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating a query, and based on the observation and evaluation, making a judgement and/or opinion to extract a first and a second element from the query.
Determine an entropy level associated with the first element, the second element, and a plurality of input parameters;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating the first element, the second element, and a plurality of input parameters, and based on the observation and evaluation, making a judgement and/or opinion of an entropy level associated with the first element, the second element, and the plurality of input parameters.
Determine that the entropy level exceeds a threshold;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating an entropy level and a threshold, and based on the observation and evaluation, making a judgement and/or opinion that the entropy level exceeds the threshold.
Generate a first embedding representation associated with the first element, a second embedding representation associated with the second element, and embedding representations associated with the plurality of input parameters;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating the first element, the second element, and the plurality of input parameters, and based on the observation and evaluation, making a judgement and/or opinion of a first embedding representation/vector associated with the first element, a second embedding representation/vector associated with the second element, and embedding representations/vectors associated with the plurality of input parameters.
In response to the determination that the entropy level exceeds the threshold, determine a plurality of probabilities associated with mapping the first element and the second element to the plurality of input parameters based on distances between (i) each of the first embedding representation and the second embedding representation and (ii) the embedding representations associated with respective input parameters of the plurality of input parameters,
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating the entropy level exceeding the threshold and distances between (i) each of the first embedding representation/vector associated with the first element and the second embedding representation/vector associated with the second element and (ii) the embedding representations/vectors associated with respective input parameters of the plurality of input parameters, and based on the observation and evaluation, making a judgement and/or opinion of mapping the first element and the second element to the plurality of input parameters and determining a plurality of probabilities associated with the mapping.
Map, based on the first set of probabilities, the first element to a first input parameter of the plurality of input parameters using a first machine learning model; and
The limitation recites a mathematical concept of executing a mathematical formula/function in the form of “a first machine learning model” that takes as input “the first element”, “the plurality of input parameters”, and the first set of probabilities, and outputs a mapping or association between “the first element” and “a first input parameter”.
Map, based on the second set of probabilities, the second element to a second input parameter of the plurality of input parameters using a second machine learning model.
The limitation recites a mathematical concept of executing a mathematical formula/function in the form of “a second machine learning model” that takes as input “the second element”, “the plurality of input parameters”, and the second set of probabilities, and outputs a mapping or association between “the second element” and “a second input parameter”.
STEP 2A Prong Two:Independent Claims 1, 13, and 20 recite that the steps are performed using “at least one memory”, “at least one processor”, and “a non-transitory computer readable medium”, which is a high-level recitation of generic computer components and represents mere instructions to apply on a computer as in MPEP 2106.05(f), which does not provide integration into a practical application.
The claim recites the following additional elements:
The plurality of probabilities comprising a first set of probabilities and a second set of probabilities,
The limitation recites an insignificant extra-solution activity as selecting a particular type/grouping of data comprised in the plurality of probabilities as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The first set of probabilities comprising, for each respective input parameter of the plurality of input parameters, a respective first probability corresponding to mapping the first element to the respective input parameter, and
The limitation recites an insignificant extra-solution activity as selecting a particular type of data comprised in the first set of probabilities as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The second set of probabilities comprising, for each respective input parameter of the plurality of input parameters, a respective second probability corresponding to mapping the second element to the respective input parameter;
The limitation recites an insignificant extra-solution activity as selecting a particular type of data comprised in the second set of probabilities as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application.
STEP 2B:
The conclusions for the mere implementation using a computer are carried over and does not provide significantly more.
With respect to “The plurality of probabilities comprising a first set of probabilities and a second set of probabilities,” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to “The first set of probabilities comprising, for each respective input parameter of the plurality of input parameters, a respective first probability corresponding to mapping the first element to the respective input parameter,” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to “The second set of probabilities comprising, for each respective input parameter of the plurality of input parameters, a respective second probability corresponding to mapping the second element to the respective input parameter;” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
Looking at the claim as a whole does not change this conclusion and the claim is ineligible.
As per Dependent Claims 2-12 and 14-19,
STEP 1:In accordance with Step 1 of the eligibility inquiry (as explained in MPEP 2106), the claimed apparatus (claims 1-12), method (claims 13-19), and non-transitory computer readable medium (claim 20) are directed to one of the eligible categories of subject matter and therefore satisfies Step 1.
STEP 2A Prong One:The dependent claims 2-12 and 14-19 recite the following limitations directed to an abstract idea:
The limitation(s) of Dependent Claims 3 and 15 include the step(s) of:
Extracting the first element and the second element using a third machine learning model based on named entity recognition.
The limitation recites a mathematical concept of executing a mathematical formula/function in the form of “a third machine learning model” that performs “named entity recognition” that takes as input the query and outputs “the [extracted] first element and the [extracted] second element”.
The limitation(s) of Dependent Claims 5 and 17 include the step(s) of:
the application programming interface performing an action based on the mapping of the first element to the first input parameter and the mapping of the second element to the second input parameter.
The limitation recites a mathematical concept of executing a mathematical formula/function in the form of “performing an action” based on the input or triggering of “the mapping of the first element to the first input parameter and the mapping of the second element to the second input parameter”.
The limitation(s) of Dependent Claim 8 include the step(s) of:
Mapping the first element using the first machine learning model and the second machine learning model.
The limitation recites a mathematical concept of executing a mathematical formula/function, comprising the first machine learning model and the second machine learning model, that takes as input “the first element” and “the plurality of input parameters” and outputs a mapping or association between “the first element” and “a first input parameter”.
The limitation(s) of Dependent Claim 9 include the step(s) of:
Classify the first element as associated with a first class;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating the first element and a first class, and based on the observation and evaluation, making a judgement and/or opinion to classify the first element as associated with the first class.
Classify a third element, extracted from the query, as associated with the first class;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating a third element extracted from the query and the first class, and based on the observation and evaluation, making a judgement and/or opinion to classify the third element as associated with the first class.
Compare a number of elements associated with the first class and a number of input parameters associated with the first class; and
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating a number of elements associated with the first class and a number of input parameters associated with the first class, and based on the observation and evaluation, making a judgement and/or opinion of the similarities and differences (thus comparing) between the number of elements associated with the first class and the number of input parameters associated with the first class.
Determine the entropy level exceeds the threshold based on the number of elements associated with the first class being greater than the number of input parameters associated with the first class.
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating the number of elements associated with the first class and the number of input parameters associated with the first class, and based on the observation and evaluation, making a judgement and/or opinion that the entropy level exceeds the threshold by making a judgement and/or opinion that the number of elements associated with the first class is greater than the number of input parameters associated with the first class.
The limitation(s) of Dependent Claim 10 include the step(s) of:
Classify the first element and the second element using named entity recognition;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating the first element and the second element, and based on the observation and evaluation, making a judgement and/or opinion using named entity recognition of classifications for the first element and the second element.
The limitation(s) of Dependent Claim 12 include the step(s) of:
Determine to use the first machine learning model and the second machine learning model based on a trigger condition,
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind, or by a human using a pen and paper, by observing and evaluating a trigger condition, and based on the observation and evaluation, making a judgement and/or opinion to use the first machine learning model and the second machine learning model.
STEP 2A Prong Two:The claim(s) recite the following additional elements:
The limitation(s) of Dependent Claims 2 and 14 include the step(s) of:
Wherein the distances are based on positions of the first embedding representation, the second embedding representation, and the embedding representations associated with the plurality of input parameters in an embedding space.
The limitation recites an insignificant extra-solution activity as selecting a particular type of data used to base the distances on, as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The limitation(s) of Dependent Claims 4 and 16 include the step(s) of:
Wherein the first element comprises one or more words from the query.
The limitation recites an insignificant extra-solution activity as selecting a particular type of data comprised in the first element as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The limitation(s) of Dependent Claims 5 and 17 include the step(s) of:
Wherein the plurality of input parameters are associated with an application programming interface,
The limitation recites an insignificant extra-solution activity as selecting a particular type of data being associated with the plurality of input parameters as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The limitation(s) of Dependent Claims 6 and 18 include the step(s) of:
Wherein the first set of probabilities and the second set of probabilities include conditional probabilities associated with a mapping of the first element and the second element to the plurality of input parameters, respectively.
The limitation recites an insignificant extra-solution activity as selecting a particular type of probabilities being included in “the first set of probabilities and the second set of probabilities” as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The limitation(s) of Dependent Claims 7 and 19 include the step(s) of:
Wherein the first machine learning model is a first large language model (LLM), and the second machine learning model is a second LLM with fewer parameters than the first LLM.
The limitation recites an insignificant extra-solution activity as selecting a particular type of machine learning models and their number of parameters being used for the first and second machine learning models as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The limitation(s) of Dependent Claim 10 include the step(s) of:
Wherein the first class is a preset class type associated with the plurality of input parameters.
The limitation recites an insignificant extra-solution activity as selecting a particular type of class being used as the first class as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The limitation(s) of Dependent Claim 11 include the step(s) of:
Wherein the entropy level is a number representing a difference in the number of elements associated with the first class and the number of input parameters associated with the first class.
The limitation recites an insignificant extra-solution activity as selecting a particular type of data being used to represent the entropy level as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The limitation(s) of Dependent Claim 12 include the step(s) of:
The trigger condition comprising that the entropy level associated with the first element, the second element, and the plurality of input parameters exceeds the threshold.
The limitation recites an insignificant extra-solution activity as selecting a particular type of condition settings being used to represent “the trigger condition” as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application.
STEP 2B:
The conclusions for the mere implementation using a computer are carried over and does not provide significantly more.
With respect to Claims 2 and 14 reciting “Wherein the distances are based on positions of the first embedding representation, the second embedding representation, and the embedding representations associated with the plurality of input parameters in an embedding space.” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to Claims 4 and 16 reciting “Wherein the first element comprises one or more words from the query.” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to Claims 5 and 17 reciting “Wherein the plurality of input parameters are associated with an application programming interface” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to Claims 6 and 18 reciting “Wherein the plurality of probabilities includes conditional probabilities associated with a mapping of the first element and the second element to the plurality of input parameters.” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to Claims 7 and 19 reciting “Wherein the first machine learning model is a first large language model (LLM), and the second machine learning model is a second LLM with fewer parameters than the first LLM.” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to Claim 10 reciting “Wherein the first class is a preset class type associated with the plurality of input parameters.” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to Claim 11 reciting “Wherein the entropy level is a number representing a difference in the number of elements associated with the first class and the number of input parameters associated with the first class.” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to Claim 12 reciting “The trigger condition comprising that the entropy level associated with the first element, the second element, and the plurality of input parameters exceeds the threshold.” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
Looking at the claim as a whole does not change this conclusion and the claim is ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-8 and 12-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ray et al. (U.S. Pre-Grant Publication No. 2020/0410989, hereinafter referred to as Ray) and further in view of Tsvetkov et al. (U.S. Pre-Grant Publication No. 2024/0362412, hereinafter referred to as Tsvetkov) and Kumar Choubey et al. (U.S. Pre-Grant Publication No. 2023/0334245, hereinafter referred to as Kumar).
Regarding Claim 1:
Ray teaches an apparatus for parameter extraction of one or more queries, the apparatus comprising:
At least one memory; and
At least one processor coupled to the at least one memory and configured to:
Extract, from a query, a first element and a second element;
Ray teaches “receiving a query 602” and extracts at least first and second words of phrases in the query “by string-matching words or phrases in the query 602 with slot words or phrases in the slot vocabulary 610” (Para. [0078])
Determine an entropy level associated with the first element, the second element, and a plurality of input parameters;
Ray teaches “the parse results can include slot tags determined by the NLU model for each word of phrase for each candidate as well as entropy scores for each word or phrase. In some embodiments, the entropy scores can each be a number between 0 and 1 (inclusive), with numbers closer to 0 indicating a higher confidence in the parse result and numbers closer to 1 indicating a lower confidence in the parse result. The parse results and scores are provided for each candidate seeded at block 606.” (Para. [0079]) where the entropy scores are associated with the slot vocabulary 610 (Para. [0079]).
Determine that the entropy level exceeds a threshold;
Ray teaches comparing the “slot entropy scores” for a word or phrase to “an uncertainty threshold” (Para. [0080]). It is noted that a value can exceed both to the up and to the downside of a threshold value.
Generate a first embedding representation associated with the first element, a second embedding representation associated with the second element, and embedding representations associated with the plurality of input parameters.
Ray teaches “In various embodiments of this disclosure, the hybrid NLU model 208 uses a combination of delexicalization and a probabilistic NLU model, such as a neural network, to fill slots for utterances.” (Para. [0058]) and neural networks require data to be transformed into embedding vectors/representations in order to be input/processed by the neural network. Ray further teaches inputting both the query 602 and the slot vocabulary 610 into the same step 604 to be delexicalized where “To at least partially delexicalize utterances, the hybrid NLU model 208 uses the slot vocabulary 210 to replace words or phrases in utterances with tokens.” (Para. [0059]). Therefore, Ray teaches generating an embedding vector for each element in the query 602 (words or phrases) and the slot vocabulary (input parameters) 610 to be compared and processed by a neural network.
In response to the determination that the entropy level exceeds the threshold, determine a plurality of probabilities associated with mapping the first element and the second element to the plurality of input parameters,
Ray teaches, comparing the “slot entropy scores” for a word or phrase to “an uncertainty threshold” (Para. [0080]) and then determining “no new candidates [are] created by the candidate generation by delexicalization module at decision block 614…the processor computes a total confidence score for each candidate” where the candidate comprises slot words or phrases, and thus the total confidence score of each candidate represents a probability associated with mapping the elements to the slots (Para. [0081]).It is noted that Para. [0080] is referring to decision block 612 and Para.[0081] is referring to decision block 614 and Fig. 6 teaches that decision block 614 is performed in response to decision block 612.
The plurality of probabilities comprising a first set of probabilities and a second set of probabilities,
Ray teaches, after determining entropy scores, adding candidates and parse results to total set and initialize new candidate set (Fig. 8 Element 804) where “In some embodiments, the parsing confidence score for each candidate is determined from the slot entropy scores for the candidate…Once the processor selects the best candidate, the processor can use the selected candidate to post-process the inferred slots and output the intent and slot tags to fulfill the original user query” (Para. [0096]). Ray further teaches “The candidates are parsed by a trained hybrid NLU model, which provides parse results including determined slot tags and/or intent and an entropy score for each word or phrase in the query.” (Para. [0037]) thereby teaching that “each word or phrase” can be associated with multiple “slot tags” such that a first slot tag has a first set of associated words or phrases in the query, each one being a candidate for the first slot tag with a parsing confidence score, and a second slot tag has a second set of associated words or phrases in the query, each one being a candidate for the second slot tag with a parsing confidence score.
The first set of probabilities comprising, for each respective input parameter of the plurality of input parameters, a respective first probability corresponding to mapping the first element to the respective input parameter, and
Ray teaches, after determining entropy scores, adding candidates and parse results to total set and initialize new candidate set (Fig. 8 Element 804) where “In some embodiments, the parsing confidence score for each candidate is determined from the slot entropy scores for the candidate…Once the processor selects the best candidate, the processor can use the selected candidate to post-process the inferred slots and output the intent and slot tags to fulfill the original user query” (Para. [0096]). Ray further teaches “The candidates are parsed by a trained hybrid NLU model, which provides parse results including determined slot tags and/or intent and an entropy score for each word or phrase in the query.” (Para. [0037]) thereby teaching that “each word or phrase” can be associated with multiple “slot tags” such that a first slot tag has a first set of associated words or phrases in the query, each one being a candidate for the first slot tag with a parsing confidence score, and a second slot tag has a second set of associated words or phrases in the query, each one being a candidate for the second slot tag with a parsing confidence score.
The second set of probabilities comprising, for each respective input parameter of the plurality of input parameters, a respective second probability corresponding to mapping the second element to the respective input parameter;
Ray teaches, after determining entropy scores, adding candidates and parse results to total set and initialize new candidate set (Fig. 8 Element 804) where “In some embodiments, the parsing confidence score for each candidate is determined from the slot entropy scores for the candidate…Once the processor selects the best candidate, the processor can use the selected candidate to post-process the inferred slots and output the intent and slot tags to fulfill the original user query” (Para. [0096]). Ray further teaches “The candidates are parsed by a trained hybrid NLU model, which provides parse results including determined slot tags and/or intent and an entropy score for each word or phrase in the query.” (Para. [0037]) thereby teaching that “each word or phrase” can be associated with multiple “slot tags” such that a first slot tag has a first set of associated words or phrases in the query, each one being a candidate for the first slot tag with a parsing confidence score, and a second slot tag has a second set of associated words or phrases in the query, each one being a candidate for the second slot tag with a parsing confidence score.
Map, based on the first set of probabilities, the first element to a first input parameter of the plurality of input parameters using a first machine learning model; and
Ray teaches mapping “the original words of phrases from the query” to the slots (Para. [0081]) thereby teaching mapping a first original word or phrase to a first slot and a second original word or phrase to a second slot.
Ray teaches using a first machine learning model by teaching “the hybrid NLU model 208 uses a combination of delexicalization and a probabilistic NLU model, such as a neural network, to fill slots for utterances.” (Para. [0058]) and neural networks are understood to be a type of machine learning model.
Map, based on the second set of probabilities, the second element to a second input parameter of the plurality of input parameters using a second machine learning model.
Ray teaches mapping “the original words of phrases from the query” to the slots (Para. [0081]) thereby teaching mapping a first original word or phrase to a first slot and a second original word or phrase to a second slot.
Ray explicitly teaches all of the elements of the claimed invention as recited above except:
mapping the first element and the second element to the plurality of input parameters based on distances between (i) each of the first embedding representation and the second embedding representation and (ii) the embedding representations associated with respective input parameters of the plurality of input parameters,
Using a second machine learning model;
However, in the related field of endeavor of entropy based key-phrase extraction, Tsvetkov teaches:
Using a second machine learning model;
Tsvetkov teaches “the example solutions described herein use large language models (LLMs) such as GPT family of LLMs (GPT-2, GPT-3, GPT-4, made publicly available as ChatGPT by OpenAI) or the like” (Para. [0014]) thereby teaching not limiting the use of a single machine learning model, and instead using a “family”/plurality of machine learning models.
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Tsvetkov and Ray at the time that the claimed invention was effectively filed, to have combined the use of LLMs, as taught by Tsvetkov, with the systems and methods for natural language understanding, as taught by Ray.
One would have been motivated to make such combination because Tsvetkov teaches “LLMs can also learn from new data and improve over time, making them increasingly effective as they are used more frequently” (Para. [0014]) and it would have been obvious to a person having ordinary skill in the art to use an LLM as a machine learning model that can learn and improve over time, thus improving the overall system overtime.
Tsvetkov and Ray explicitly teach all of the elements of the claimed invention as recited above except:
mapping the first element and the second element to the plurality of input parameters based on distances between (i) each of the first embedding representation and the second embedding representation and (ii) the embedding representations associated with respective input parameters of the plurality of input parameters,
However, in the related field of endeavor of text classification, Kumar teaches:
mapping the first element and the second element to the plurality of input parameters based on distances between (i) each of the first embedding representation and the second embedding representation and (ii) the embedding representations associated with respective input parameters of the plurality of input parameters,
Kumar teaches “models use a similarity score between text and labels mapped to common embedding space,” (Para. [0014]) where “the base classifier 220 may be built based on cosine similarity…the non-conformity score may be based on the cosine distance between bag-of-words (BoW) representation of target label description (Cwk) and input text (xE).” (Para. [0024]).
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Kumar, Tsvetkov, and Ray at the time that the claimed invention was effectively filed, to have combined the conformal predictor, as taught by Kumar, with the use of LLMs, as taught by Tsvetkov, and the systems and methods for natural language understanding, as taught by Ray.
One would have been motivated to make such combination because Kumar teaches “A simpler and efficient CP base classifier may reduce the inference time the most. It is observed that CP-Token achieves the best inference time with the NLI model on ATIS, SNIPS and AG's news datasets, and with the NSP model on ATIS and AG's news datasets. On the other hand, it achieves the lowest ALS size for both models only on the ATIS dataset. Minimal complexity for calculating token-overlap adds negligible overhead to the 0shot model, thus, achieving the best speed up despite higher ALS size in several cases.” (Para. [0074]) and “the accuracy increases in many cases, suggesting that pruning label space using a CP may remove noisy labels and boost the performance.” (Para. [0076]).
Regarding Claim 2:
Kumar, Tsvetkov, and Ray further teach:
Wherein the distances are based on positions of the first embedding representation, the second embedding representation, and the embedding representations associated with the plurality of input parameters in an embedding space.
Kumar teaches “models use a similarity score between text and labels mapped to common embedding space,” (Para. [0014]) where “the base classifier 220 may be built based on cosine similarity…the non-conformity score may be based on the cosine distance between bag-of-words (BoW) representation of target label description (Cwk) and input text (xE).” (Para. [0024]).
Regarding Claim 3:
Kumar, Tsvetkov, and Ray further teach:
Wherein extracting the first element and the second element comprises extracting the first element and the second element using a third machine learning model based on named entity recognition.
Ray teaches using named entity recognition by teaching recognition of the name “Alice” as being a contact connected with “a slot tag of ‘(@contact, Alice)’” (Para. [0057]). Ray further teaches “high variability slots can include utterance words or phrases such as message text, movie names, or names of people or places in different contexts. The hybrid NLU model 208 is configured to iteratively delexicalize input utterances. In various embodiments of this disclosure, the hybrid NLU model 208 uses a combination of delexicalization and a probabilistic NLU model, such as a neural network, to fill slots for utterances.” (Para. [0058]).
Regarding Claim 4:
Kumar, Tsvetkov, and Ray further teach:
Wherein the first element comprises one or more words from the query.
Ray teaches “matching words or phrases in the query with words or phrases in a slot vocabulary knowledge base” (Para. [0037]) thereby teaching the first and second elements each being words or a phrase from the query.
Regarding Claim 5:
Kumar, Tsvetkov, and Ray further teach:
Wherein the plurality of input parameters are associated with an application programming interface, the application programming interface performing an action based on the mapping of the first element to the first input parameter and the mapping of the second element to the second input parameter.
Ray teaches “The kernel 141 can control or manage system resources (such as the bus 110, processor 120, or a memory 130) used to perform operations or functions implemented in other programs (such as the middleware 143, API 145, or application program 147). The kernel 141 provides an interface that allows the middleware 143, the API 145, or the application 147 to access the individual components of the electronic device 101 to control or manage the system resources. The application 147 can include an application that receives queries, delexicalizes the query to create one or more delexicalized candidates, parses the candidates, determines intent and slot tags from at least one of the parsed candidates, and executes an action based on the determined intent and slot tags.” (Para. [0043]) thereby teaching using the API performing/executing an action based on the determined intent and slot tags of the mapped query words or phrases with slot tags.
Regarding Claim 6:
Kumar, Tsvetkov, and Ray further teach:
Wherein the first set of probabilities and the second set of probabilities include conditional probabilities associated with a mapping of the first element and the second element to the plurality of input parameters, respectively.
Ray teaches “the processor computes a total confidence score for each candidate” where the candidate comprises slot words or phrases, and thus the total confidence score of each candidate represents a conditional probability associated with mapping the elements to the slots (Para. [0081]). Ray teaches at least one condition/rule by teaching “the new set of candidates is created by analyzing each original candidate and its parse results and scores and by using a set of domain-independent rules to potentially generate one or more new candidates from the original candidate” (Para. [0080]).It is noted that the claim limitation does not differentiate the conditional probabilities in such a way that the first set and second set of probabilities can include the same conditional probabilities, but for mapping different elements (first and second elements) to the plurality of input parameters.
Regarding Claim 7:
Kumar, Tsvetkov, and Ray further teach:
Wherein the first machine learning model is a first large language model (LLM), and the second machine learning model is a second LLM with fewer parameters than the first LLM.
Tsvetkov teaches “the example solutions described herein use large language models (LLMs) such as GPT family of LLMs (GPT-2, GPT-3, GPT-4, made publicly available as ChatGPT by OpenAI) or the like” (Para. [0014]) and the LLMs GPT-2, GPT-3, and GPT-4 all have different numbers of parameters. Therefore, when using two of them, one will have fewer parameters than the other.
Regarding Claim 8:
Kumar, Tsvetkov, and Ray further teach:
Wherein mapping the first element comprising mapping the first element using the first machine learning model and the second machine learning model.
Ray teaches mapping “the original words of phrases from the query” to the slots (Para. [0081]) thereby teaching mapping a first original word or phrase to a first slot and a second original word or phrase to a second slot.
Ray teaches using a first machine learning model by teaching “the hybrid NLU model 208 uses a combination of delexicalization and a probabilistic NLU model, such as a neural network, to fill slots for utterances.” (Para. [0058]) and neural networks are understood to be a type of machine learning model.
Tsvetkov teaches “Example solutions use LLMs to assess the “informativeness” of phrases within the text, such as by evaluating an entropy measure of words or phrases found within text” (Para. [0015]) thereby teaching using more than one machine learning model to evaluate and then map the words or phrases found within text.
Regarding Claim 12:
Kumar, Tsvetkov, and Ray further teach:
Determine to use the first machine learning model and the second machine learning model based on a trigger condition, the trigger condition comprising that the entropy level associated with the first element, the second element, and the plurality of input parameters exceeds the threshold.
Ray teaches, based on comparing the “slot entropy scores” for a word or phrase to “an uncertainty threshold” (Para. [0080]), thus determining that the entropy score exceeds the threshold, either to the upside or the downside, and determining “no new candidates [are] created by the candidate generation by delexicalization module at decision block 614…the processor computes a total confidence score for each candidate” and moving forward to map “the original words or phrases from the query” to the slots (Para. [0081]).
Ray teaches using a first machine learning model by teaching “the hybrid NLU model 208 uses a combination of delexicalization and a probabilistic NLU model, such as a neural network, to fill slots for utterances.” (Para. [0058]) and Tsvetkov teaches “Example solutions use LLMs to assess the “informativeness” of phrases within the text, such as by evaluating an entropy measure of words or phrases found within text” (Para. [0015]) thereby teaching using more than one machine learning model to evaluate and then map the words or phrases found within text to the slots. Therefore, Ray in combination with Tsvetkov teach, based on, in part, determining the entropy score(s) exceeding the uncertainty threshold, determining to proceed to map the original words or phrases from the query to the slots, which can be done using a plurality of neural networks/LLMs.
Regarding Claim 13:
All of the limitations herein are similar to some or all of the limitations of Claim 1.
Regarding Claim 14:
All of the limitations herein are similar to some or all of the limitations of Claim 2.
Regarding Claim 15:
All of the limitations herein are similar to some or all of the limitations of Claim 3.
Regarding Claim 16:
All of the limitations herein are similar to some or all of the limitations of Claim 4.
Regarding Claim 17:
All of the limitations herein are similar to some or all of the limitations of Claim 5.
Regarding Claim 18:
All of the limitations herein are similar to some or all of the limitations of Claim 6.
Regarding Claim 19:
All of the limitations herein are similar to some or all of the limitations of Claim 7.
Regarding Claim 20:
Some of the limitations herein are similar to some or all of the limitations of Claim 1.
Kumar, Tsvetkov, and Ray further teach:
A non-transitory computer readable medium storing code for optimizing parameter extraction, the code comprising instructions executable by a processor (Ray – Para. [0007]).
Claim(s) 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over Kumar, Tsvetkov, and Ray, and further in view of Traub et al. (U.S. Pre-Grant Publication No. 2012/0278321, hereinafter referred to as Traub).
Regarding Claim 9:
Tsvetkov and Ray further teach:
Classify the first element as associated with a first class;
Ray teaches “The parameters can include determined slots or slot tags that identify semantic associations of a word, a phrase, or multiple words or phrases in the utterance data, which are typically domain-specific.” (Para. [0056]) and “the slot vocabulary…is domain-dependent” (Claim 2) thereby teaching associating extracted elements of the query with a particular domain/classification.
Classify a third element, extracted from the query, as associated with the first class;
Ray teaches “The parameters can include determined slots or slot tags that identify semantic associations of a word, a phrase, or multiple words or phrases in the utterance data, which are typically domain-specific.” (Para. [0056]) and “the slot vocabulary…is domain-dependent” (Claim 2) thereby teaching associating extracted elements of the query with a particular domain/classification.
Ray further teaches handling a third element by teaching an example query containing “post message to Bob Happy new year” which is parsed into B-contact, B-message, I-message, and I-message slots with the intent to “send_message” (Fig. 9 & Para. [0057])).
Ray teaches “receiving a query 602” and extracts a plurality of words of phrases in the query “by string-matching words or phrases in the query 602 with slot words or phrases in the slot vocabulary 610” (Para. [0078])
Kumar, Tsvetkov, and Ray explicitly teach all of the elements of the claimed invention as recited above except:
Compare a number of elements associated with the first class and a number of input parameters associated with the first class; and
Determine the entropy level exceeds the threshold based on the number of elements associated with the first class being greater than the number of input parameters associated with the first class.
However, in the related field of endeavor of computing the relative entropy between a query language model and the corresponding collection language model, Traub teaches:
Compare a number of elements associated with the first class and a number of input parameters associated with the first class; and
Traub teaches “predict query performance by computing the relative entropy between a query language model and the corresponding collection language model.” (Para.[0005]) where “relative entropy expresses how different a probability distribution associated with the query result set is different from the corresponding probability distribution associated with the baseline set. In some embodiments the baseline set could be either the overall collection or the result set for a different query. Stated another way, relative entropy is a basis for a measure of distinctiveness/salience; that is, of how interesting, or distinctive, that result set is, compared to other sets.” (Para. [0080]). Therefore, Traub in combination with Tsvetkov and Ray teach comparing the number of elements of a first set (the query in Ray) with the number of elements associated with the baseline set (slot vocabulary of a particular domain in Ray).
Determine the entropy level exceeds the threshold based on the number of elements associated with the first class being greater than the number of input parameters associated with the first class.
Traub teaches “predict query performance by computing the relative entropy between a query language model and the corresponding collection language model.” (Para.[0005]) where “relative entropy expresses how different a probability distribution associated with the query result set is different from the corresponding probability distribution associated with the baseline set. In some embodiments the baseline set could be either the overall collection or the result set for a different query. Stated another way, relative entropy is a basis for a measure of distinctiveness/salience; that is, of how interesting, or distinctive, that result set is, compared to other sets.” (Para. [0080]).
Traub further teaches “When probability distributions P and Q are identical, the relative entropy of P relative to Q is zero.” (Para.[0078]) and therefore, negative when the probability distribution P is less than Q, or positive when the probability distribution P is greater than Q.
Therefore, Traub in combination with Tsvetkov and Ray teach calculating entropy as the difference in the number of elements of a first set (the query in Ray) with the number of elements associated with the baseline set (slot vocabulary of a particular domain in Ray) and Ray teaches comparing the “slot entropy scores” for a word or phrase to “an uncertainty threshold” (Para. [0080]).It is noted that the limitation does not recite how the determination of the number of elements being greater than the number of input parameters affects or influences the determination of the entropy level exceeding the threshold, just that it is “based on”.
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Traub, Kumar, Tsvetkov, and Ray at the time that the claimed invention was effectively filed, to have combined the use of multiple interpretations to queries, as taught by Traub, with the conformal predictor, as taught by Kumar, the use of LLMs, as taught by Tsvetkov, and the systems and methods for natural language understanding, as taught by Ray.
One would have been motivated to make such combination because Traub teaches, as part of “using a distinctiveness measure to improve user experience” (Para. [0174]), “an information access system can evaluate multiple interpretations of a user's input and determine which of these possible query interpretations lead to interesting queries” (Para. [0176]) and it would be obvious to a person having ordinary skill in the art that improving user experience by considering the different possible interpretations of a user’s query would lead to a better understanding of the query and eventually improve satisfaction by the user in the query response.
Regarding Claim 10:
Traub, Kumar, Tsvetkov, and Ray further teach:
Classify the first element and the second element using named entity recognition;
Ray teaches using named entity recognition by teaching recognition of the name “Alice” as being a contact connected with “a slot tag of ‘(@contact, Alice)’” (Para. [0057]). Ray further teaches “high variability slots can include utterance words or phrases such as message text, movie names, or names of people or places in different contexts. The hybrid NLU model 208 is configured to iteratively delexicalize input utterances” (Para. [0058]).
Wherein the first class is a preset class type associated with the plurality of input parameters.
Ray teaches “The slot vocabulary includes data different from training data, and is domain-dependent. In various embodiments of this disclosure, every slot word or phrase in the slot vocabulary 210 has a corresponding special delexicalization token. For example, if the word “Alice” is in the slot vocabulary 210, the word “Alice” has a corresponding special token, such as <contact>. This special token may not always correspond to the slot name, such as @contact, and may depend on the semantic type of the slot, the domain, and the relationship of the slot with other slots in the domain.” (Para. [0059]) thereby teaching preset domains associated with a plurality of slot vocabulary.
Regarding Claim 11:
Traub, Kumar, Tsvetkov, and Ray further teach:
Wherein the entropy level is a number representing a difference in the number of elements associated with the first class and the number of input parameters associated with the first class.
Traub teaches “predict query performance by computing the relative entropy between a query language model and the corresponding collection language model.” (Para.[0005]) where “relative entropy expresses how different a probability distribution associated with the query result set is different from the corresponding probability distribution associated with the baseline set. In some embodiments the baseline set could be either the overall collection or the result set for a different query. Stated another way, relative entropy is a basis for a measure of distinctiveness/salience; that is, of how interesting, or distinctive, that result set is, compared to other sets.” (Para. [0080]). Therefore, Traub in combination with Tsvetkov and Ray teach calculating entropy as the difference in the number of elements of a first set (the query in Ray) with the number of elements associated with the baseline set (slot vocabulary of a particular domain in Ray).
Response to Amendment
Applicant’s Amendments, filed on 5/7/2026, are acknowledged and accepted.
Response to Arguments
On pages 10-11 of the Remarks filed on 5/7/2026, Applicant argues that “fails to describe ‘generat[ing] a first embedding representation associated with the first element, a second embedding representation associated with the second element, and embedding representations associated with the plurality ofinput parameters; in response to the determination that the entropy level exceeds the threshold, determine[ing] a plurality of probabilities associated with mapping the first element and the second element to the plurality of input parameters based on distances between (i) each of the first embedding representation and the second embedding representation and (ii) the embedding representations associated with respective input parameters of the plurality of input parameters,’ as recited in amended claim 1.” because “Applicant submits that Ray's disclosure of a probabilistic NLU model, such as a neural network, at most indicates that Ray may use a neural network to fill slots for utterances. Applicant respectfully submits that there is no description in Ray regarding "determining the plurality of probabilities based on distances between" the specific embedding representations (i.e., the first embedding representation, the second embedding representation, and the embedding representations associated with respective input parameters of the plurality of input parameters).”Applicant’s argument is convincing that Tsvetkov and Ray in combination fail to teach at least a portion of the amended features of the independent claims, and thus necessitate the new grounds of rejection presented above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yararbas et al. (U.S. Pre-Grant Publication No. 2022/0083557) teaches receiving a search query including search terms; using a machine learning module, selecting features of the search terms and mapping an association between the search terms and a domain object, thereby generating a domain object classification; tagging the domain object with the domain object classification; and using the domain object tagged with the domain object classification to conduct a subsequent search. Conducting the subsequent search includes: receiving a subsequent search query including subsequent search terms; tokenizing the subsequent search terms; finding permutations of the tokenized subsequent search terms; matching the subsequent search terms to the domain object tagged with the domain object classification; and displaying subsequent search results via a user interface.The reference further teaches “where xi corresponds to each search word, C is a specific class label (from Y), and l represents the information gain in an entropy sense.” (Para. [0022]).
Li et al. (U.S. Pre-Grant Publication No. 2018/0060301) teaches systems, methods, and techniques by which a processing unit can build an end-to-end dialogue agent model for end-to-end learning of dialogue agents for information access and apply the end-to-end dialogue agent model with soft attention over knowledge base entries to make the dialogue system differentiable. In various examples the processing unit can apply the end-to-end dialogue agent model to a source of input, fill slots for output from the knowledge base entries, induce a posterior distribution over the entities in a knowledge base or induce a posterior distribution of a target of the requesting user over entities from a knowledge base, develop an end-to-end differentiable model of a dialogue agent, use supervised and/or imitation learning to initialize network parameters, calculate a modified version of an episodic algorithm, e.g., the REINFORCE algorithm, for training an end-to-end differentiable model based on user feedback.The reference further teaches “computing posterior distribution over knowledge base entries to make the knowledge base differentiable; computing an entropy over a goal of an individual, e.g., each, slot to make the knowledge base differentiable; using the calculated entropies and posteriors to output probability of a next action” (Paras. [0236] – [0238]).
Choi et al. (U.S. Pre-Grant Publication No. 2002/0042793) teaches “a method of order-ranking document clusters using entropy data and Bayesian SOM according to the present invention, includes the steps of recording query words given by users for search(S10), designing user files made up of the keywords used for the most recent search and their frequencies so as to reflect user preference(S20), calculating entropy among query words given by users, user profiles and keywords of each web document(S30), judging whether data for learning Kohonen neural network, which is a type of unsupervised neural network model, is sufficient or not(S40); a fifth step of ensuring number of documents using a bootstrap algorithm, a type of statistical technique, if it is determined in the fourth step that the data is not sufficient(S60); a sixth step of determining a prior information to be used as an initial value for each parameter of network through Bayesian learning, and determining an initial connection weight value of Bayesian SOM neural network model where the Kohonen neural network and Bayesian learning are coupled(S50); and a seventh step of performing a real-time document clustering for relevant documents using the entropy value calculated in the third step and Bayesian SOM neural network model(S70).” (Para. [0051]).
Wu et al. (U.S. Patent No. 8,019,748) teaches similar queries are identified so that suggested search queries may be provided to a user. A first search query having at least one term associated with a semantic weight is received. A second query having at least one term associated with a semantic weight is received. A first vector is generated based on the first query and the at least one semantic weight of the first query, and a second vector is generated based on the second query and the at least one semantic weight of the second query. A similarity measurement between the first and second search queries is made based at least in part on a comparison of the first vector to the second vector.The reference further teaches “the entropy of an N-gram determines the uniqueness of the N-gram in a group of previously known terms, such as in an index of previous search queries” (Col. 6 Lines 64-67).
Najaf-Zadeh et al. (U.S. Pre-Grant Publication NO. 2020/0020133) teaches the processor is configured to generate, for a 3D point cloud, a first frame representing a first attribute and a second frame representing a second attribute. The first and second frames include patches representing respective clusters of points from the 3D point cloud. The processor is configured to generate an occupancy map frame. The processor is configured to identify a query point that is positioned on a boundary of one of the patches. The processor is configured to perform smoothing with respect to the query point. The processor is configured to encode the frames and generate a compressed bitstream. The communication is configured to transmit the compressed bitstream.The reference further teaches “the second attribute represents color; to perform smoothing of the query point, the processor is configured to: identify neighboring points within a distance from the query point, derive a local entropy value of the neighboring points, compare the local entropy value of the neighboring points to a first threshold, when the local entropy value is smaller than the first threshold, derive a centroid representing color values of the neighboring points, determine whether a difference between the centroid and a color value of the query point is larger than a second threshold, and after determining that the difference is larger than the second threshold, modify the color value of the query point with the color value of the centroid” (Claim 4).
Wan et al. (U.S. Pre-Grant Publication No. 2023/0353585) teaches a malicious traffic identification method and a related apparatus. The malicious traffic identification method may include: determining a receiving time of first alarm traffic; obtaining, according to a preset policy, a plurality of pieces of second alarm traffic corresponding to the first alarm traffic within a target time period, where the target time period is a time period determined based on the receiving time, and a similarity between each of the plurality of pieces of second alarm traffic and the first alarm traffic is greater than a preset threshold; performing feature extraction on the plurality of pieces of second alarm traffic to obtain first feature information; and determining, based on the first feature information, whether the first alarm traffic is malicious traffic. In embodiments of this disclosure, accuracy of malicious traffic identification on a live network can be improved by using a multi-flow traceback method.The reference further teaches in Table 2 of Para. [0105]) calculating an entropy value of the request parameter value and an average value of information entropy values corresponding to a plurality of flows.
Non-Patent Literature Futami et al., "Information-theoretic Analysis of Test Data Sensitivity in Uncertainty", 23 July 2023, arXiv:2307.12456v1, <https://arxiv.org/pdf/2307.12456> (Year: 2023) teaches Bayesian inference is often utilized for uncertainty quantification tasks. A recent analysis rigorously decomposed the predictive uncertainty in Bayesian inference into two uncertainties, called aleatoric and epistemic uncertainties, which represent the inherent randomness in the data-generating process and the variability due to insufficient data, respectively. They analyzed those uncertainties in an information-theoretic way, assuming that the model is well-specified and treating the model’s parameters as latent variables. However, the existing information-theoretic analysis of uncertainty cannot explain the widely believed property of uncertainty, known as the sensitivity between the test and training data. It implies that when test data are similar to training data in some sense, the epistemic uncertainty should become small. In this work, we study such uncertainty sensitivity using our novel decomposition method for the predictive uncertainty. Our analysis successfully defines such sensitivity using information-theoretic quantities. Furthermore, we extend the existing analysis of Bayesian meta-learning and show the novel sensitivities among tasks for the first time.The reference further teaches in Equations (5) and (6) a calculation of a minimum excess risk by using a conditional entropy.
Non-Patent Literature Liang, Xue-Bin, "A Note on Fano's Inequality", May 2011, 45th Annual Conference on Information Sciences and Systems, Baltimore, MD, USA, 2011, pp. 1-3, doi: 10.1109/CISS.2011.5766186. (Year: 2011) teaches Fano's inequality is a sharp upper bound on conditional entropy in terms of the probability of error. It plays a fundamental role in the proof of converse part in coding theorems of information theory. A standard proof of Fanos inequality as seen in textbooks is based on properties of Shannon's information measures with a trick of introducing an auxiliary random variable. In this brief, it is observed that the generic ℒ1 bound on entropy can straightforwardly provide an upper bound on conditional entropy in terms of the probability of error. This generic ℒ1 bound on conditional entropy is still equally effective as the Fano bound for applications in the converse proof. Compared with the generic ℒ1 bound, Fanos inequality can be regarded as a specific bound on conditional entropy by employing the structural property of the two joint probability distributions. This viewpoint motivates us to find an identity to connect the conditional entropy and the probability of error. As a corollary, a necessary and sufficient condition for tightness of the Fano bound is also obtained.
Misiewicz et al. (U.S. Pre-Grant Publication No. 2022/0138170) teaches “the indexed set of structured data elements (e.g., the knowledge graph associated with the entity) is queried to identify candidate structured data elements which are within a distance threshold in a feature space based on the comparison of the embedding vectors. In an embodiment, the comparison includes scoring each of the structured data elements based on a distance metric function (e.g., a cosine similarity function, an Euclidean distance function, a Manhattan distance function, a Jaccard Similarity function, etc.).” (Para. [0064]) and “the processing logic identifies, based on the comparison, a set of matching structured data elements. In an embodiment, the processing logic applies a distance threshold to each of the generated distance metric function scores. In an embodiment, the set of matching structured data elements is identified as the structured data element that have a distance-comparison score that is less than the distance threshold level, wherein smaller distance results represent structured data elements that are determined to be more relevant in response to the search query. In an embodiment, the distance threshold is adjustable and can be changed or updated based on input received from a user of the entity system or based on analytics performed by the neural network in view of aggregated training data.” (Para. [0065]).
Non-Patent Literature Anonymous, "Shifting Attention To Relevance: Towards The Uncertainty Estimation Of Large Language Models", 10 February 2024, Twelfth International Conference on Learning Representations ICLR 2024 (Year: 2024) teaches although Uncertainty Quantification (UQ) presents a promising solution, its accurate implementation within the context of LLMs remains a significant hurdle. To address this critical roadblock, our research originates from a fundamental heuristic insight: tokens within auto-regressive LLM-generated text do not equally reflect the underlying meaning. Some tokens carry greater relevance and representativeness than others, owing to the phenomenon of “linguistic redundancy”, wherein a select few keywords suffice to convey the essence of lengthy sentences. Regrettably, existing methodologies treat all tokens with equal importance when estimating uncertainty, disregarding these inherent generative in equalities. Our analysis reveals a significant issue with state-of-the-art: numerous tokens (and sentences) of limited semantic significance receive equal or even excessive weighting during uncertainty estimation. To rectify this bias, we propose to jointly Shifting Attention to more Relevant (SAR) components, at both the token- and the sentence-levels for accurate uncertainty estimation. We conduct ex tensive experiments involving a range of popular “off-the-shelf” LLMs, including instruction-tuned LLMs such as Vicuna, WizardLM, and LLaMA-2-chat, as well as pretrained LLMs like OPT and LLaMA, with model sizes extending up to 33B parameters. We carry out evaluation across various free-form question-answering tasks, encompassing domains such as reading comprehension, science Q&A, and medical Q&A. Our experimental results, coupled with a comprehensive demographic analysis, demonstrate the superior performance of SAR in addressing the challenges of uncertainty estimation within the realm of LLMs.
Non-Patent Literature Su et al., "Building Natural Language Interfaces to Web APIs", 6 November 2017, International Conference on Information and Knowledge Management (Year: 2017) teaches studying the problem of natural language interface to APIs (NL2APIs), with a focus on web APIs for web services. Such NL2APIs have many potential benefits, for example, facilitating the integration of web services into virtual assistants. We propose the first end-to-end framework to build an NL2API for a given web API. A key challenge is to collect training data, i.e., NL command-API call pairs, from which an NL2API can learn the semantic mapping from ambiguous, informal NL commands to formal API calls. We propose a novel approach to collect training data for NL2API via crowdsourcing, where crowd workers are employed to generate diversified NL commands. We optimize the crowdsourcing process to further reduce the cost. More specifically, we propose a novel hierarchical probabilistic model for the crowdsourcing process, which guides us to allocate budget to those API calls that have a high value for training NL2APIs. We apply our framework to real-world APIs, and show that it can collect high-quality training data at a low cost, and build NL2APIs with good performance from scratch. We also show that our modeling of the crowdsourcing process can improve its effectiveness, such that the training data collected via our approach leads to better performance of NL2APIs than a strong baseline.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT F MAY whose telephone number is (571)272-3195. The examiner can normally be reached Monday-Friday 9:30am to 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Boris Gorney can be reached on 571-270-5626. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT F MAY/Examiner, Art Unit 2154 8/4/2026
/SYED H HASAN/Primary Examiner, Art Unit 2154
