DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment and Arguments.
Applicant’s arguments, see page 12, filed 1/14/2026, with respect to claims 1-20 rejection have been fully considered and are not persuasive. Applicant argues that “computer-based applications for employing various LLMs to independently evaluate and generate responses to produce a collaborated answer. Thus, this computer-dependent process is incapable of being practically performed in the human mind” The Examiner respectfully disagrees the large language models is an additional element (see specification paragraph 14) which is recited to be a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component. Furthermore, this would still be a mental process of a person receiving the query and creating questions based on the original query from a group of people. Additionally, the group of people answering those questions, Additionally categorizing them and scoring them. Further, this individual would be the final editor and can work alongside them and choose the final answer. Hence being a mental process. The claims are not patent eligible.
Applicant further argues that “Applicant submits that no reasonable interpretation of amended independent claims 1, 10, and 19 would characterize these claims as reciting features recited at a high-level of generality. Additionally, these newly claimed details are inextricably tied to and solve problems in current computer technology. Conventional AI evaluation systems lack consistency in generating responses and often suffer from hallucinations, producing inaccurate responses, as described in paragraphs [0002] and [0003] of the instant specification. In contrast, these newly claimed features combine and coordinate a plurality of LLMs to generate more accurate responses and reduce the possibility of hallucinations, as described in paragraphs [00111]-[00115]. Thus, these claimed features significantly enhance the ability of an AI-based system for evaluating generated responses. ”, Furthermore, applicant cites “Ex parte Desjardins et al.” The Examiner respectfully disagrees this does not solve problems within the computer technology. Since the LLM’s are being used to only generate answers and choose an answer. After the answers are generated, it’s being evaluated. Having an evaluation is not an improvement of a computer. The claims are not patent eligible.
Applicant further argues that “The Office asserts that the claims fail to recite significantly more than an abstract idea for similar reasons as those used in the step 2A analysis and do not amount to significantly more than an abstract idea. But the claims have been amended to be allowable under § 101 Step 2A analysis, as discussed above. Therefore, the amended claims are not susceptible to this ground of rejection. ” The Examiner respectfully disagrees the judicial exception is not integrated into a practical application. In particular, the claim only recites additional elements that are computer components “processor” (paragraph 37), “LLM’s” (paragraph 14), “memory” (paragraphs 38), and “application programming interface” (paragraph 51) recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using the computer components amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claims are not patent eligible.
Applicant further argues that “Dependent claims 2, 11, and 20 have been amended to recite that the identifying of the final response includes aggregating each respective initial response to each question from each of the plurality of LLMs, based on predetermined criteria. This feature provides a technological improvement by aggregating a plurality of responses from LLMs to generate a more accurate final response. Thus, these features amount to significantly more than an abstract idea. Therefore, amended dependent claims 2, 11, and 20 are allowable under § 101 for this additional reason. ” The Examiner respectfully disagrees claims 2, 11, and 20 does not prevent a human from performing the steps mentally as described above. additionally, the person ranking the plurality of questions and answers. Further presenting the best question/answer. Thus, these claims are directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claims are not patent eligible.
Therefore, the 101 rejection of claims 1- 20 are maintained.
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1, 10, and 19, Further claim 1 recites A method for facilitating automated model evaluation based on question perturbations, the method being implemented by at least one processor, the method comprising:
receiving, by the at least one processor via an application programming interface, at least one input, each of the at least one input including an inquiry in a natural language format;
generating, by the at least one processor via a rephrasing model, a plurality of questions based on the inquiry, each of the plurality of questions corresponding to a lexical variant of the inquiry;
determining, by the at least one processor via a plurality of large language models (LLMs),ana respective corresponding initial response for each respective question from among the plurality of questions and the inquiry, wherein each respective LLM from the plurality of LLMs is a different model, wherein each LLM from among the plurality of LLMs independently determines a respective corresponding initial response for each respective question from among the plurality of questions;
clustering, by the at least one processor, each initial response for each of the plurality of questions and the inquiry into at least one block based on at least one shared characteristic; and
computing, by the at least one processor, at least one metric for each of the at least one block.
Further claim 18 a memory;
The limitation of “receiving…”, “clustering…”, “determining…”, “computing…”, and “generating…” , as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. For example, a person receiving the query and creating questions created based on the original query from a group of people. Additionally, the group of people answering those questions. Alongside categorizing them and scoring them. Further, this individual would be the final editor and can work alongside them and choose the final answer.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claim only recites additional elements that are computer components “processor” (paragraph 37), “LLM’s” (paragraph 14), “memory” (paragraphs 38), and “application programming interface” (paragraph 51) recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using the computer components amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claims are not patent eligible.
Claims 2, 11, and 20, further claim 2 states The method of claim 1, further comprising: ranking, by the at least one processor, the at least one block based on the computed at least one metric; and identifying, by the at least one processor, a final response for the inquiry based on a result of the ranking, wherein the identifying of the final response includes aggregating each respective initial response to each question from each of the plurality of LLMs, based on predetermined criteria. However, this limitation does not prevent a human from performing the steps mentally as described above. additionally, the person ranking the plurality of questions and answers. Further presenting the best question/answer. Thus, these claims are directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claims are not patent eligible.
Claims 3 and 12 further claim 3 recites the method of claim 1, further comprising: generating, by the at least one processor, at least one classification report for the inquiry based on the computed at least one metric, wherein the at least one metric includes a supervised metric and an unsupervised metric. However, this limitation does not prevent a human from performing the steps mentally as described above. additionally, the person ranking the plurality of questions and answers. Further presenting the best question/answer. Finally, giving a report on the query’s that he creates of which ones are good or bad as well as scoring them based on the ranking. Thus, these claims are directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claims are not patent eligible.
Claims 4 and 13 further claim 4 recites the method of claim 1, wherein each of the plurality of questions relates to a natural language query that incorporates semantic meaning extracted from the inquiry, the semantic meaning including subject matter that corresponds to the inquiry. However, this limitation does not prevent a human from performing the steps mentally as described above. additionally, the person creating the variants of question to maintain the main idea of the original question. Where they take keywords from the original question and rephrase the remaining words of the question to create the variants. Thus, these claims are directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claims are not patent eligible.
Claims 5 and 14 further claim 5 recites The method of claim 1, wherein the generating of the plurality of questions based on the inquiry further comprises: applying, by the at least one processor via the rephrasing model, a predetermined transformation algorithm to the inquiry to generate each of the plurality of questions, wherein the predetermined transformation algorithm perturbs the inquiry to retain at least one semantic quality of the inquiry. However, this limitation does not prevent a human from performing the steps mentally as described above. additionally, the person creating the variants of question to maintain the main idea of the original question. Where they take keywords from the original question and rephrase the remaining words of the question to create the variants. Thus, these claims are directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claims are not patent eligible.
Claims 6 and 15, further claim 6 additionally recites the method of claim [[1]] 2, wherein the final response includes at least one crowdsourced answer determined from the initial responses determined by the plurality of LLMs. However, these limitations encompass a generating and answering those question while only considering the question he’s on and not thinking back on previous questions he answered. Thus, the claim is directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 7 and 16, further claim 7 additionally recites The method of claim 1, wherein the at least one metric includes at least one from among an accuracy metric that relates to a baseline accuracy, a robustness metric that relates to correctness of at least one rater, a plurality voting metric that relates to aggregated responses by a mode of a corresponding answer set, an agreement metric that relates to each initial response, and a reliability metric that relates to a measure of internal consistency. However, these limitations encompass a person generating questions based on a query and answering all the questions. Further, scoring the response to see if it meets a threshold. Thus, the claim is directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 8 and 17, further claim 8 additionally recites The method of claim 1, further comprising: associating, by the at least one processor, feedback data with each corresponding initial response when an agreement metric is below a predetermined agreement threshold, the feedback data including the at least one metric; and determining, by the at least one processor via the plurality of LLMs, a subsequent response for each of the plurality of questions and the inquiry based on the feedback data. However, these limitations encompass a person receiving a query than generating questions based on that query. Additionally answering those questions. Also looking at the responses and scoring them as well as annotating them if there are any mistakes. Thus, the claim is directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 9 and 18, further claim 9 additionally recite the method of claim 1, wherein each of the plurality of LLMs and the rephrasing model includes at least one from among a large language model, a deep learning model, a neural network model, a natural language processing model, a machine learning model, a mathematical model, and a process model. However, these limitations encompass a person receiving a query than generating questions based on that query. Additionally answering those questions. Thus, the claim is directed towards a mental process. In particular, the claim only recites additional elements that are computer components “LLM”(paragraph 76, 82), “deep learning model” (paragraph 76, 82), “neural network model” (paragraph 76), “natural language processing model” (paragraph 76, 83), “machine learning model” (paragraph 76, 78), “process model” (paragraph 76), “response models” (paragraph 86), and “rephrasing model” (paragraph 14 ) recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1,2,4,5,6,7,9,10,11,13,14,15,16,18,19, and 20 are rejected under 35 U.S.C. 103 as obvious over US Patent US 20200142888 A1, (Alakuijala; Jyrki). in view of US Patent US 20240394286 A1, (Honke; Garrett Raymond.).
Claim 1,10,19
Regarding Claim 1,10,19, Jyrki teaches
1. A method for facilitating automated model evaluation based on question perturbations, the method being implemented by at least one processor, the method comprising:
receiving, by the at least one processor via an application programming interface, at least one input, each of the at least one input including an inquiry in a natural language format;
(Paragraph 19 "In some implementations, a method implemented by one or more processors is provided that includes receiving an original query; applying tokens of the original query as input to a trained generative model; and generating multiple variants of the original query based on application of tokens of the original query to the trained generative model. The original query can be generated based on user interface input of a user via a client device. Each of the generated variants differs from the original query and generating the variants includes producing the variants based on learned parameters of the trained generative model. The trained generative model is trained to enable generation of multiple types of query variants, and the generated variants include a first variant that is a first type of the multiple types of query variants and a second variant that is a second type of the multiple types of query variants. The method further includes: generating an output based on at least one of the multiple variants and/or at least one search system response to at least one of the multiple variants; and providing, in response to the original query, the output for presentation via the client device.")
generating, by the at least one processor via a rephrasing model, at least one question based on the inquiry, each of the at least one question corresponding to a lexical variant of the inquiry;
(Paragraph 19 "In some implementations, a method implemented by one or more processors is provided that includes receiving an original query; applying tokens of the original query as input to a trained generative model; and generating multiple variants of the original query based on application of tokens of the original query to the trained generative model. The original query can be generated based on user interface input of a user via a client device. Each of the generated variants differs from the original query and generating the variants includes producing the variants based on learned parameters of the trained generative model. The trained generative model is trained to enable generation of multiple types of query variants, and the generated variants include a first variant that is a first type of the multiple types of query variants and a second variant that is a second type of the multiple types of query variants. The method further includes: generating an output based on at least one of the multiple variants and/or at least one search system response to at least one of the multiple variants; and providing, in response to the original query, the output for presentation via the client device.")
clustering, by the at least one processor, the initial response for each of the at least one question and the inquiry into at least one block based on at least one shared characteristic; and
(paragraph 8 "In some implementations, multiple variants of an original query are generated utilizing the generative model, each of the multiple variants are submitted to a search system, and corresponding response(s) received for each of the multiple variants. An output can be generated based on one or more of the responses, and the output provided in response to the original query. For example, the output can include the “best” response (e.g., as indicated by response scores provided by the search system), multiple of the “best” responses, and/or a variant and corresponding response(s) (e.g., when the variant is of a follow-up type). In this and other manners, response(s) to variant(s) of an original query can be utilized to provide output, in response to the original query, where the output directly answers the original query. Further, response(s) to variant(s) of an original query can be utilized to substantiate/corroborate response(s) to the original query and/or response(s) to other variant(s). For example, the accuracy of an “answer” to an original query can be determined based on whether affirmative answers are provided for variants of the original query. For instance, based on whether other affirmative answers are provided for variant(s) of a follow-up type and/or based on whether affirmative similar answers (similar to the answer of the original query) are available to variant(s) of equivalent, generalization, and/or language translation type(s). In this and other manners, unsubstantiated/uncorroborated response(s) can be determined and not utilized in provided output, and/or flagged as uncorroborated if utilized in provided output (e.g., flagged as “potentially fake”).")
computing, by the at least one processor, at least one metric for each of the at least one block.
(paragraph 8 "In some implementations, multiple variants of an original query are generated utilizing the generative model, each of the multiple variants are submitted to a search system, and corresponding response(s) received for each of the multiple variants. An output can be generated based on one or more of the responses, and the output provided in response to the original query. For example, the output can include the “best” response (e.g., as indicated by response scores provided by the search system), multiple of the “best” responses, and/or a variant and corresponding response(s) (e.g., when the variant is of a follow-up type). In this and other manners, response(s) to variant(s) of an original query can be utilized to provide output, in response to the original query, where the output directly answers the original query. Further, response(s) to variant(s) of an original query can be utilized to substantiate/corroborate response(s) to the original query and/or response(s) to other variant(s). For example, the accuracy of an “answer” to an original query can be determined based on whether affirmative answers are provided for variants of the original query. For instance, based on whether other affirmative answers are provided for variant(s) of a follow-up type and/or based on whether affirmative similar answers (similar to the answer of the original query) are available to variant(s) of equivalent, generalization, and/or language translation type(s). In this and other manners, unsubstantiated/uncorroborated response(s) can be determined and not utilized in provided output, and/or flagged as uncorroborated if utilized in provided output (e.g., flagged as “potentially fake”).")
Jyrki do not explicitly teach all of determining, by the at least one processor via at least one response model a plurality of large language models (LLMs),ana respective corresponding initial response for each of the at least one question respective question from among the plurality of questions and the inquiry, wherein each respective LLM from the plurality of LLMs is a different model, wherein each LLM from among the plurality of LLMs independently determines a respective corresponding initial response for each respective question from among the plurality of questions;
However, Honke teach
determining, by the at least one processor via at least one response model a plurality of large language models (LLMs),ana respective corresponding initial response for each of the at least one question respective question from among the plurality of questions and the inquiry, wherein each respective LLM from the plurality of LLMs is a different model, wherein each LLM from among the plurality of LLMs independently determines a respective corresponding initial response for each respective question from among the plurality of questions;
(Paragraph 112, provisional page 10 lines 13-22 "In some implementations, the system can generate a second prompt. The second prompt can include different text than the prompt, but with the same meaning. The system can generate the second prompt using a language model, for example. The system can provide an input that includes the prompt and an input that includes the second prompt to one or more machine learning models. The system can generate distributions for intermediate answers corresponding to the prompt and the second prompt. The system can generate the answer by performing a probabilistic inference over both distributions. In some implementations, the system can generate more prompts that include different text than the prompt but have the same meaning, and provide inputs that include the different prompts to the machine learning models."
Paragraph 87, provisional page 8 lines 13-22 "For example, a system can provide the input to one or more language models by calling the language model. In some implementations, the system can provide the input to the same language model. In some implementations, the system can provide the input to instances of the same language model. In some implementations, the system can provide the input to different language models. In yet other implementations, the system can provide input to 1) a set of instances of one language model and 2) a second different language model alone or to a set of instances of the second different language model."
Paragraph 48, provisional page 4 lines 8-13 "The language models 108 can be any appropriate machine learning models that are configured to generate an output sequence of tokens given an input sequence. For example, the input sequence can be a prompt of natural language text. The output sequence can be an answer to the prompt in natural language text. For example, the language model 108 can be a large language model."
Paragraph 26 page 1 lines 19-31 and page 2 lines 0-11"Conventional systems that draft documents or answer questions provide one document or one answer to a question. Conventional systems do not provide insight into any potential steps or chains of reasoning that led to the document or answer. Some conventional systems also do not provide insight into uncertainties or nuances in parts of the document or answer. Thus conventional systems may disregard many potential answers or parts of answers that could be interesting to a user, or whose exploration could lead to different answers. The techniques described in this specification allow for quantifying ambiguities in answers or parts of answers, and for further exploration of other potential answers or parts of answers. The system described in this specification can, for example, represent ambiguity using a distribution of intermediate answers, or parts of answers, generated by providing the same input to a language model multiple times to generate multiple intermediate answers. The system can cluster the intermediate answers in the distribution. The system can select the most probable intermediate answer as the answer, or explore the less likely intermediate answers. In addition, while a human or conventional system may only develop a single argument at a time, the system can allow for the parallel development of multiple chains of arguments from different perspectives."
Paragraph 6 "In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of obtaining a prompt comprising natural language text; obtaining a set of documents comprising natural language text; generating an input comprising at least the set of documents and the prompt; providing the input to a plurality of language models, wherein each language model is configured to generate at least an intermediate answer to the prompt from the input; generating a distribution from the intermediate answers; and generating an answer to the prompt by performing a probabilistic inference over the distribution, the answer comprising natural language text.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jyrki to incorporate the teachings of Honke to provide a “determining, by the at least one processor via at least one response model a plurality of large language models (LLMs),ana respective corresponding initial response for each of the at least one question respective question from among the plurality of questions and the inquiry, wherein each respective LLM from the plurality of LLMs is a different model, wherein each LLM from among the plurality of LLMs independently determines a respective corresponding initial response for each respective question from among the plurality of questions;” Doing so would Provide a level of confidence to the user if the answer is ambiguous, as recognized by Ankur. (Paragraph 97).
Regarding claim 10 Jyrki in view of Honke further Jyrki teaches
10. A computing device configured to implement an execution of a method for facilitating automated model evaluation based on question perturbations, the computing device comprising:
a processor;
(Paragraph 18 " In some implementations, a method implemented by one or more processors is provided that includes receiving an original query that is generated based on user interface input of a user via a client device. The method further includes applying, as input to a trained generative model: tokens of the original query, and one or more attributes associated with the user. The trained generative model is a sequence to sequence deep neural network model with one or more memory layers. The method further includes generating at least one variant of the original query based on application of the tokens and the one or more attributes to the trained generative model. The method further includes generating an output based on at least one of: the at least one variant, and at least one search system response to the at least one variant. The method further includes providing, in response to the original query, the output for presentation via the client device.")
a memory; and
(Paragraph 18 " In some implementations, a method implemented by one or more processors is provided that includes receiving an original query that is generated based on user interface input of a user via a client device. The method further includes applying, as input to a trained generative model: tokens of the original query, and one or more attributes associated with the user. The trained generative model is a sequence to sequence deep neural network model with one or more memory layers. The method further includes generating at least one variant of the original query based on application of the tokens and the one or more attributes to the trained generative model. The method further includes generating an output based on at least one of: the at least one variant, and at least one search system response to the at least one variant. The method further includes providing, in response to the original query, the output for presentation via the client device.")
a communication interface coupled to each of the processor and the memory,
(Paragraph 18 " In some implementations, a method implemented by one or more processors is provided that includes receiving an original query that is generated based on user interface input of a user via a client device. The method further includes applying, as input to a trained generative model: tokens of the original query, and one or more attributes associated with the user. The trained generative model is a sequence to sequence deep neural network model with one or more memory layers. The method further includes generating at least one variant of the original query based on application of the tokens and the one or more attributes to the trained generative model. The method further includes generating an output based on at least one of: the at least one variant, and at least one search system response to the at least one variant. The method further includes providing, in response to the original query, the output for presentation via the client device.")
For remaining limitation see claim one. Claim 10 is similar to claim 1 and therefore are rejected for similar reasons as in claim 1.
Claim 2,11,20
Regarding Claim 2,11,20, Jyrki in view of Honke further Jyrki teaches
2. The method of claim 1, further comprising:
ranking, by the at least one processor, the at least one block based on the computed at least one metric; and
(paragraph 8 "In some implementations, multiple variants of an original query are generated utilizing the generative model, each of the multiple variants are submitted to a search system, and corresponding response(s) received for each of the multiple variants. An output can be generated based on one or more of the responses, and the output provided in response to the original query. For example, the output can include the “best” response (e.g., as indicated by response scores provided by the search system), multiple of the “best” responses, and/or a variant and corresponding response(s) (e.g., when the variant is of a follow-up type). In this and other manners, response(s) to variant(s) of an original query can be utilized to provide output, in response to the original query, where the output directly answers the original query. Further, response(s) to variant(s) of an original query can be utilized to substantiate/corroborate response(s) to the original query and/or response(s) to other variant(s). For example, the accuracy of an “answer” to an original query can be determined based on whether affirmative answers are provided for variants of the original query. For instance, based on whether other affirmative answers are provided for variant(s) of a follow-up type and/or based on whether affirmative similar answers (similar to the answer of the original query) are available to variant(s) of equivalent, generalization, and/or language translation type(s). In this and other manners, unsubstantiated/uncorroborated response(s) can be determined and not utilized in provided output, and/or flagged as uncorroborated if utilized in provided output (e.g., flagged as “potentially fake”).")
Jyrki in view of Honke further Honke teaches
identifying, by the at least one processor, a final response for the inquiry based on a result of the ranking, wherein the identifying of the final response includes aggregating each respective initial response to each question from each of the plurality of LLMs, based on predetermined criteria.
(paragraph 90, provisional page 8 lines 24-32 "The system can generate a distribution (450). The system can generate the distribution from intermediate answers. For example, a visualization of a distribution is shown below in FIG. 5. In some implementations, generating a distribution can include clustering the intermediate answers based on the similarity of each intermediate answer to each other intermediate answer. For example, the similarity can be a cosine similarity between embeddings of the intermediate answers."
Paragraph 108 provisional page 9 lines 23-31"In implementations where the system provides the input to different language models, the system can generate an answer by performing a probabilistic inference over each distribution. In these implementations, the system can determine an initial answer for each distribution, and then determine a final answer from the initial answers, for example, by repeating at least steps 430-450 for a new input. For example, the system can generate an input that includes the initial answers as multiple choice options for a multiple choice question derived from the prompt, and the set of documents for the prompt."
Paragraph 101, provisional page 9 lines 12-15 "In some examples where the prompt includes an open-ended question, each intermediate answer can include at least part of an answer to the question. For example, the system can generate multiple intermediate answers using a language model. The system can cluster the intermediate answers into multiple clusters. The system can cluster the intermediate answers into multiple clusters using embeddings, bag of words, or other clustering analyses. As an example, the system can generate embeddings for each of the intermediate answers. The system can cluster, e.g., using unsupervised clustering, each of the intermediate answers in the embedding space."
Paragraph 105 provisional page 9 lines 16-19 "In some examples, the system can use the language model to determine which leaves are most relevant to the prompt. The system can determine relevance by volume, length, emotional valence, or other text-based evaluations. In some examples, the system can generate the answer by providing the answers (e.g., the chain of intermediate answers leading to the leaf) generated for the most relevant leaves and a request to select the best answer to the one or more language models.")
See claim one for rationale.
Claim 4,13
Regarding Claim 4,13, Jyrki in view of Honke further Jyrki teaches
The method of claim 1, wherein each of the plurality of questions relates to a natural language query that incorporates semantic meaning extracted from the inquiry, the semantic meaning including subject matter that corresponds to the inquiry.
(Fig 8A and Fig 8B shows the question being a NL query and having variants of the original question by paraphrasing it
Paragraph 22 " In some implementations, a method implemented by one or more processors is provided that includes receiving an original query generated based on user interface input of a user via a client device. The method further includes determining a predicted task for the user and applying, as input to a trained generative model: tokens of the original query, and one or more task attributes of the predicted task for the user. The method further includes generating at least one variant of the original query based on application of the tokens and the one or more task attributes to the trained generative model. The method further includes: generating an output based on the at least one variant and/or at least one search system response to the at least one variant; and providing, in response to the original query, the output for presentation via the client device.")
Claim 5,14
Regarding Claim 5,14, Jyrki in view of Honke further Jyrki teaches the method of claim 1, wherein the generating of the plurality of questions based on the inquiry further comprises: applying, by the at least one processor via the rephrasing model, a predetermined transformation algorithm to the inquiry to generate each of the plurality of questions,
(Paragraph 19 "In some implementations, a method implemented by one or more processors is provided that includes receiving an original query; applying tokens of the original query as input to a trained generative model; and generating multiple variants of the original query based on application of tokens of the original query to the trained generative model. The original query can be generated based on user interface input of a user via a client device. Each of the generated variants differs from the original query and generating the variants includes producing the variants based on learned parameters of the trained generative model. The trained generative model is trained to enable generation of multiple types of query variants, and the generated variants include a first variant that is a first type of the multiple types of query variants and a second variant that is a second type of the multiple types of query variants. The method further includes: generating an output based on at least one of the multiple variants and/or at least one search system response to at least one of the multiple variants; and providing, in response to the original query, the output for presentation via the client device."
paragraph 11 " In some implementations, multiple generative models can be generated, with each of the generative models being trained based on training data that is based on past query submissions associated with particular attributes, such as particular attributes of a user, particular temporal attributes, and/or other attributes. For example, the first generative model can be generated based on training data that is based on past query submissions associated with an on-line shopping task. For instance, the past query submissions can be identified based on being submitted to an on-line shopping search system, based on users selecting shopping content (e.g., certain ads) in association with the submissions, based on search results being shopping centric, based on users completing a transaction following the submissions, etc. A second generative model can be generated based on training data that is based on past query submissions associated with different particular attributes. For example, the second generative model can be generated based on training data that is based on past query submissions associated with a traveling to a location task (e.g., to any location, any restaurant location, a meeting location, etc.). For instance, the past query submissions can be identified based on being submitted before and/or during travel to a location, based on being submitted temporally close to a scheduled calendar entry, etc. For a submitted query of a user, a task of the user can be predicted, and a generative model corresponding to the predicted task selected for generating variants for that submitted query. For example, if a calendar entry and/or electronic communications of the user indicate the user is travelling to a location (or will soon be travelling to the location), the second generative model in the preceding example can be selected based on that model being associated with a travelling to a location task. In this manner, a generative model can be selected, from a plurality of available generative models, such that the selected generative model is tailored to a task of the user, such as a predicted task being engaged in, or to be engaged in. This may result in generation of query variants, utilizing the selected generative model, that are more appropriate for the current task of the user. As described above and elsewhere herein, in various implementations a generative model can be a multitask model and enable generation of query variants of various disparate types. Some of those various implementations enable use of the generative model to generate variants that expand a user query and enable exploration of multiple paths of extending the query. Such variants can be provided for presentation to the user (e.g., optionally without first issuing queries based on such variants), simultaneously or sequentially, to enable the user to explore various paths for extending the query. Additionally, or alternatively, responses to such variants can be obtained from a search system, and the responses provided for presentation to the user to enable the user to explore the various responses for the extensions to the query."
Paragraph 22 " In some implementations, a method implemented by one or more processors is provided that includes receiving an original query generated based on user interface input of a user via a client device. The method further includes determining a predicted task for the user and applying, as input to a trained generative model: tokens of the original query, and one or more task attributes of the predicted task for the user. The method further includes generating at least one variant of the original query based on application of the tokens and the one or more task attributes to the trained generative model. The method further includes: generating an output based on the at least one variant and/or at least one search system response to the at least one variant; and providing, in response to the original query, the output for presentation via the client device.")
wherein the predetermined transformation algorithm perturbs the inquiry to retain at least one semantic quality of the inquiry.
(Paragraph 86 "At block 660, the system determines whether to generate further variants. In some implementations, the system determines whether to generate further variants based on properties of the so-far generated variants and/or based on response(s) from a search system for the so-far generated variants. For example, the system can determine whether to generate further variants based on whether response(s) to the so-far generated variant(s) were found by the search system and/or quality measure(s) of the response(s). For instance, the system can generate further variants if no responses were found and/or if quality measure(s) fail to satisfy one or more quality criteria.")
Claim 6,15
Regarding Claim 6,15, Jyrki in view of Honke further Honke teaches the method of claim [[1]] 2, wherein the final response includes at least one crowdsourced answer determined from the initial responses determined by the plurality of LLMs.
(paragraph 90, provisional page 8 lines 24-32 "The system can generate a distribution (450). The system can generate the distribution from intermediate answers. For example, a visualization of a distribution is shown below in FIG. 5. In some implementations, generating a distribution can include clustering the intermediate answers based on the similarity of each intermediate answer to each other intermediate answer. For example, the similarity can be a cosine similarity between embeddings of the intermediate answers."
Paragraph 108 provisional page 9 lines 23-31"In implementations where the system provides the input to different language models, the system can generate an answer by performing a probabilistic inference over each distribution. In these implementations, the system can determine an initial answer for each distribution, and then determine a final answer from the initial answers, for example, by repeating at least steps 430-450 for a new input. For example, the system can generate an input that includes the initial answers as multiple choice options for a multiple choice question derived from the prompt, and the set of documents for the prompt."
Paragraph 101, provisional page 9 lines 12-15 "In some examples where the prompt includes an open-ended question, each intermediate answer can include at least part of an answer to the question. For example, the system can generate multiple intermediate answers using a language model. The system can cluster the intermediate answers into multiple clusters. The system can cluster the intermediate answers into multiple clusters using embeddings, bag of words, or other clustering analyses. As an example, the system can generate embeddings for each of the intermediate answers. The system can cluster, e.g., using unsupervised clustering, each of the intermediate answers in the embedding space."
Paragraph 105 provisional page 9 lines 16-19 "In some examples, the system can use the language model to determine which leaves are most relevant to the prompt. The system can determine relevance by volume, length, emotional valence, or other text-based evaluations. In some examples, the system can generate the answer by providing the answers (e.g., the chain of intermediate answers leading to the leaf) generated for the most relevant leaves and a request to select the best answer to the one or more language models.")
See claim one for rationale.
Claim 7,16
Regarding Claim 7,16, Jyrki in view of Honke further Jyrki teaches The method of claim 1, wherein the at least one metric includes at least one from among an accuracy metric that relates to a baseline accuracy, a robustness metric that relates to correctness of at least one rater, a plurality voting metric that relates to aggregated responses by a mode of a corresponding answer set, an agreement metric that relates to each initial response, and a reliability metric that relates to a measure of internal consistency.
(paragraph 8 "In some implementations, multiple variants of an original query are generated utilizing the generative model, each of the multiple variants are submitted to a search system, and corresponding response(s) received for each of the multiple variants. An output can be generated based on one or more of the responses, and the output provided in response to the original query. For example, the output can include the “best” response (e.g., as indicated by response scores provided by the search system), multiple of the “best” responses, and/or a variant and corresponding response(s) (e.g., when the variant is of a follow-up type). In this and other manners, response(s) to variant(s) of an original query can be utilized to provide output, in response to the original query, where the output directly answers the original query. Further, response(s) to variant(s) of an original query can be utilized to substantiate/corroborate response(s) to the original query and/or response(s) to other variant(s). For example, the accuracy of an “answer” to an original query can be determined based on whether affirmative answers are provided for variants of the original query. For instance, based on whether other affirmative answers are provided for variant(s) of a follow-up type and/or based on whether affirmative similar answers (similar to the answer of the original query) are available to variant(s) of equivalent, generalization, and/or language translation type(s). In this and other manners, unsubstantiated/uncorroborated response(s) can be determined and not utilized in provided output, and/or flagged as uncorroborated if utilized in provided output (e.g., flagged as “potentially fake”).")
Claim 9,18
Regarding Claim 9,18, Jyrki in view of Honke further Jyrki teaches
The method of claim 1, wherein each of the plurality of LLMs and the rephrasing model includes at least one from among a large language model, a deep learning model, a neural network model, a natural language processing model, a machine learning model, a mathematical model, and a process model.
( Paragraph 5 "In some implementations where the generative model is a neural network model with memory layers, the generative model is a sequence to sequence model. For example, the sequence to sequence model can be one where tokens of a query can be applied as input to the model (e.g., on a token-by-token basis or combined basis), and an encoding of the tokens generated over layers of the network. Further, the generated encoding can be decoded over additional layers of the network, where the resulting decoding indicates (directly or indirectly) a variant of the query. For instance, the resulting decoding can be applied to softmax layer(s) of the network to generate the variant of the query. In some versions of those implementations, the generative model has the same or similar architecture as a sequence to sequence neural machine translation model and is trained utilizing query variant specific training data. The query variant specific training data can be, for example, based on: query pairs that each have “clicks” on the same documents (e.g., to train for equivalent query variant generation); query pairs submitted in succession (e.g., to train for follow-up query variant generation); and/or original, canonical query pairs (e.g., to train for canonicalization query variant generation). Such a model can be optionally pre-trained based on translation training data.")
Claims 3 and 12 are rejected under 35 U.S.C. 103 as obvious over US Patent US 20200142888 A1, (Alakuijala; Jyrki). in view of US Patent US 20240394286 A1, (Honke; Garrett Raymond.). in further view of US Patent US 20220188661 A1, (Tappin; Isabella.).
Claim 3,12
Regarding Claim 3,12, Jyrki in view of Honke do not explicitly teach all of
The method of claim 1, further comprising: generating, by the at least one processor, at least one classification report for the inquiry based on the computed at least one metric,
wherein the at least one metric includes a supervised metric and an unsupervised metric.
However, Isabella teaches
3. The method of claim 1, further comprising:
generating, by the at least one processor, at least one classification report for the inquiry based on the computed at least one metric,
Paragraph 37 "Identifying and clarifying ambiguity in the original query, and interactively obtaining supplemental information and/or clarification from the user via the graphical user interface 250 of FIG. 2. In some implementations, the query subsystem 210 may generate options based on the original user query for interactive refinement and selection by the user during the query analysis process in order to resolve ambiguity, narrow the scope of the query, and more accurately identify user intention and variables, timelines, and metrics associated with the original user query."
Paragraph 40 "The functionalities above for the query processing subsystem 210 form a core of the query entry portion of the pattern machine architecture, in which the input query is extracted, cleansed, expanded, supplemented, and analyzed to generate one or more formalized questions with well-defined signals/events and predictable metrics and variables in a single domain or in a combination of multiple domains. For example, the query processing subsystem 210 maps metrics in an input query, after expansion, supplementation, and/or cleansing, to related signals of the input query that are pre-computed and constantly updated. Alternatively, historical processed queries may be recommended. A more detailed example pipeline for the query processing subsystem 210 is shown in 300 of FIG. 3."
Paragraph 91 "Returning to the query recommendation subsystem 270 of FIG. 2, an example data/logic flow of such a subsystem is illustrated as 800 in FIG. 8. The query recommendation data/logic flow 800, for example, may be configured to recommend follow-up queries to the user based on inputs such as the user's input query, the user's historical queries, historical queries from all the users, the knowledge base, and optionally based on the answer generated by the answer generation subsystem described above. The purpose of query recommendation is that it may provide the user with optional suggested follow-up queries for the user to explore additional information related to the original query and/or the answer to the original query. The user may choose from the suggested queries listed, for example, as part of the answering page on the graphical user interface, to run a new query. The selected new query may then be processed by the knowledge pattern machine to generate further predictive answers and reports."
Paragraph 171 " As described above and throughout this disclosure, the knowledge pattern machine system is configured to be dynamic and real-time in various aspects. For example, data scraping is performed continuously and in real-time, capable of catching new data and updating the old data as becomes available. The knowledge graph and the various signal and event databases are also updated continuously and in real-time as the underlying data changes. In addition, the predictive answer and reports generated for user queries and stored in the answer/report database are also updated continuously and in real time as the underlying data, including the information in the knowledge graph and the event/signal databases, changes over time. Further, the knowledge graph may be configured to be dynamic in that it records changing information entities and relationships/correlating information on a time series, as described in more detail above.")
wherein the at least one metric includes a supervised metric and an unsupervised metric.
(Paragraph 45 " For example, OOS type may be used to represent queries or questions that are unethical to answer. Ethical boundaries may be set based on topic-dependent parameters. This type of question would mean that the entire question or most of the question is unethical. In some implementations, a model for determining unethical questions may be established by starting from sample out-of-scope topics, using both language models and vector embedding to identify similar queries. For biased queries, determination of the bias in the queries may vary depending on the topical domain. Certain opinionated language can typically indicate strong bias. For example, some input queries may contain bias and/or profanity in the form of strong opinion towards the entities in the queries or are sarcastic. These queries may be first identified and then further processed to remove the bias. Various models may be designed and trained based on labeled datasets (pre-classified user queries) for classifying an input query into these various example query types. For example, supervised/reinforcement learning may be employed. The underlying models may be based on neural networks. A bias detector may be built for each domain. A bias detector may also combine a general bias detection component supplemented by domain bias detecting components. Bias may be automatically removed once identified.”)
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jyrki in view of Honke to incorporate the teachings of Isabella to provide a “The method of claim 1, further comprising: generating, by the at least one processor, at least one classification report for the inquiry based on the computed at least one metric, wherein the at least one metric includes a supervised metric and an unsupervised metric.” Doing so Be able to share reports with others, as recognized by Isabella. (Paragraph 175).
Claims 8 and 17are rejected under 35 U.S.C. 103 as obvious over US Patent US 20200142888 A1, (Alakuijala; Jyrki). in view of US Patent US 20240394286 A1, (Honke; Garrett Raymond.). in further view of US Patent US 20210049476 A1, (Davis; Matthew).
Claim 8,17
Regarding Claim 8,17, Jyrki in view of Honke do not explicitly teach all of The method of claim 1, further comprising: associating, by the at least one processor, feedback data with each corresponding initial response when an agreement metric is below a predetermined agreement threshold, the feedback data including the at least one metric; and determining, by the at least one processor via the plurality of LLMs, a subsequent response for each of the plurality of questions and the inquiry based on the feedback data.
However, Matthew teaches
The method of claim 1, further comprising: associating, by the at least one processor, feedback data with each corresponding initial response when an agreement metric is below a predetermined agreement threshold, the feedback data including the at least one metric;
(Paragraph 27 "For example, consider a compendium including information on available vacation trips. If an incoming query asks about vacations in England, and an embodiment scores a potential response in the compendium with a value higher than the threshold value (perhaps a score of 0.8, with a threshold of 0.5, all on a 0-1 scale), it is likely that this potential response includes information about vacations in England, and is thus a good response to the query. If an incoming query asks about a particular software product, an embodiment will score all the potential responses in the compendium with very low values, under the lower threshold (perhaps giving each a score of 0.1, with a lower threshold of 0.25, all on the same 0-1 scale) because this is a compendium of vacation information, not software product information, and the query is simply outside the compendium's designed scope. However, if an incoming query asks about vacations in Italy, and an embodiment scores all potential response in the compendium with values in between the two thresholds (perhaps scores of 0.3 or 0.4, all on the same 0-1 scale), this query represents information the compendium should have but does not. Consequently, a new response to this new query should be added to the compendium.")
and determining, by the at least one processor via the plurality of LLMs, a subsequent response for each of the plurality of questions and the inquiry based on the feedback data. (Paragraph 28 "An embodiment also determines that a compendium of natural language responses to natural language queries requires a response to a query by analyzing feedback from a response provided to the query. In particular, due to a false positive score, the embodiment might have provided an incorrect response. However, the querier is likely to provide feedback that the response is incorrect. By conducting a natural language analysis on the provided feedback, an embodiment can conclude that the response is incorrect, and that a new response to this query should be added to the compendium. For example, if an incoming query asks about vacations in Italy, and an embodiment provided a response about vacations in France, the querier might respond with an additional query such as, “I asked about Italy, not France!” By analyzing this additional query, an embodiment concludes that the previously-provided response was incorrect, and that a new, correct, response is needed."
paragraph 29 "An embodiment uses the results of a natural language analysis of the query to construct a search for information that could constitute a new response to be added to the compendium. One embodiment can use, as a data source for the search, a corpus of documents or other narrative text that has been analyzed and indexed by a natural language analysis tool. Another embodiment can utilize any search engine tool that has an application program interface (API) capable of being used by a software application to find and retrieve the information. The information is typically in the form of narrative text.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jyrki in view of Honke to incorporate the teachings of Matthew to provide a “The method of claim 1, further comprising: associating, by the at least one processor, feedback data with each corresponding initial response when an agreement metric is below a predetermined agreement threshold, the feedback data including the at least one metric; and determining, by the at least one processor via the plurality of LLMs, a subsequent response for each of the plurality of questions and the inquiry based on the feedback data.” Doing so would Increase the accuracy/correctness of the response, as recognized by Matthew. (Paragraph 20).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALI M HASSAN whose telephone number is (571)272-5331. The examiner can normally be reached Monday - Friday 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras Shah can be reached at (571)270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALI M HASSAN/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653 04/13/2026
Prosecution Insights