DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The IDS dated 4/20/2026 has been considered and placed in the application file.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1, 12, and 19, Further claim 1 recites A method comprising:
receiving a text query from a user;
identifying first text chunks based on similarities between the text query and a plurality of text chunks;
determining first score information associated with each of the first text chunks;
generating a first prompt based on the first score information, the first prompt including the text query and the first text chunks;
transmitting the first prompt to a text generation model;
receiving a response to the first prompt from the text generation model;
presenting the response and the first text chunks;
receiving a rating of one of the first text chunks from the user; and
updating the first score information associated with the one of the first text chunks based on the rating.
Further claim 12 states A system comprising:
a memory storing executable program code; and
at least one processing unit to execute the program code to cause the system to perform operations comprising:
Further claim 19 states One or more non-transitory computer-readable recording media storing program code, the program code executable by at least one processing unit of a computing system to cause the computing system to perform operations comprising:
The limitation of “receiving…”, “identifying …”, “determining…”, “generating…” , “transmitting …” , “receiving …” , “presenting …” , “receiving…” , and “updating …” , as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. For example, a person receiving a query from a user and answering by going through multiple articles. Further when going through the articles scoring/ranking the passages to the query. When the best passage is found answering the query with an answer and the best passage from the articles. Further when presenting it the individual gives a rating to the best passage and updates the score.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claim only recites additional elements that are computer components “text generation model” (paragraph 22), “processor” (paragraph 32) and “memory” (paragraphs 32 & 48) recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using the computer components amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claims are not patent eligible.
Claims 2 and 13 additionally claim 2 recites The method of Claim 1, wherein presenting the response and the first text chunks comprises: determining a composite score based on the first score information associated with each of the first text chunks; and presenting an indicator of the composite score with the response. However, this limitation does not prevent a human from performing the steps mentally as described above. Further, the person giving a cumulative score based on all the passages to the user. Thus, these claims are directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claims are not patent eligible.
Claims 3 additionally recites The method of Claim 2, wherein the first score information associated with each of the first text chunks includes a first score associated with each of the first text chunks, and wherein presenting the response and the first text chunks comprises: presenting each of the first text chunks with an indicator of the first score associated with the first text chunk. However, these limitations encompass a person having an answer and a score for each passage being presented to the user. Thus, these claims are directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claims are not patent eligible.
Claims 4 additionally recites the method of Claim 3, wherein receiving the rating of the one of the first text chunks comprises: receiving a selection of an indicator associated with the one of the first text chunks which is different from the presented indicator of the first score associated with the one of the first text chunks. However, these limitations encompass a person having an answer and a score for each passage being presented to the user. Further the user putting his score of the passage, which is different than the original score. Thus, these claims are directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claims are not patent eligible.
Claims 5 additionally recites the method of Claim 1, wherein the first score information associated with each of the first text chunks includes a first score associated with each of the first text chunks, and wherein presenting the response and the first text chunks comprises: presenting each of the first text chunks with an indicator of the first score associated with the first text chunk. However, these limitations encompass a person a score for each passage being presented to the user. Further the user putting his score of the passage, which is different than the original score. Thus, these claims are directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claims are not patent eligible.
Claims 6 additionally recites The method of Claim 5, wherein receiving the rating of the one of the first text chunks comprises: receiving a selection of an indicator associated with the one of the first text chunks which is different from the presented indicator of the first score associated with the one of the first text chunks. However, these limitations encompass a person having an answer and a score for each passage being presented to the user. Further the user having a different score for that passage. Thus, these claims are directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claims are not patent eligible.
Claims 7 and 15 additionally claim 7 recites The method of Claim 1, wherein the first score information associated with each of the first text chunks includes a first score associated with each of the first text chunks, and wherein the first prompt comprises instructions to associate the first text chunks with an importance based on their respective first scores. However, these limitations encompass a person answering a query with instructions to rank the chunks of importance to the query from a user and further ranking and scoring the passages he’s reading for an answer based on the query. Thus, the claim is directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 8, 16, and 20 additionally claim 8 recites The method of Claim 1, further comprising: receiving a second text query from a second user; identifying second text chunks based on similarities between the second text query and the plurality of text chunks; determining second score information associated with each of the second text chunks; generating a second prompt based on the second score information, the second prompt including the second text query and the second text chunks; transmitting the second prompt to the text generation model; receiving a second response to the second prompt from the text generation model; presenting the second response and the second text chunks; receiving a second rating of one of the second text chunks from the second user; and updating the second score information associated with the one of the second text chunks based on the second rating. However, these limitations a person receiving another query from a user and answering by going through multiple articles. Further when going through the articles scoring/ranking the passages to the query. When the best passage is found answering the query with an answer and the best passage from the articles. Further when presenting it the individual gives a rating to the best passage and updates the score. In particular, the claim only recites additional elements that are computer components “text generation model” (paragraph 22) is recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea. Therefore, the claim is not patent eligible.
Claims 9 and 19 additionally claim 9 recite The method of Claim 8, further comprising: wherein the identified second text chunks include the one of the first text chunks, and wherein the determined second score information associated with the one of the first text chunks is the updated first score information. However, these limitations encompass a person receiving a query from a user and going through passages and giving them scores. These scores having already a score from the previous query where the newer score updates the previous one. Thus, the claim is directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 10 additionally recite The method of Claim 8, wherein the first score information associated with each of the first text chunks includes a first score associated with each of the first text chunks, wherein presenting the response and the first text chunks comprises presenting each of the first text chunks with an indicator of the first score associated with the first text chunk, wherein the second score information associated with each of the second text chunks includes a second score associated with each of the second text chunks, and wherein presenting the second response and second text chunks comprises presenting each of the second text chunks with an indicator of the second score associated with the second text chunk. However, these limitations encompass a person receiving a query from a user and going through passages and giving them scores. These scores having already a score from the previous query where the newer score updates the previous one. Furthermore, the answer, the first score, second score, and the according passages are shown to the user. Thus, the claim is directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 11 additionally recite The method of Claim 10, wherein receiving the rating of the one of the first text chunks comprises: receiving a selection of an indicator associated with the one of the first text chunks which is different from the presented indicator of the first score associated with the one of the first text chunks, and wherein receiving the second rating of the one of the second text chunks comprises: receiving a second selection of a second indicator associated with the one of the second text chunks which is different from the presented indicator of the second score associated with the one of the second text chunks. However, these limitations encompass a person receiving a query from a user and going through passages and giving them scores. These scores having already a score from the previous query where the newer score updates the previous one. Furthermore, the answer, the first score, second score, and the according passages are shown to the user. Even further the user would be able to give a score to the first and second score being shown to him. Thus, the claim is directed towards a mental process. Similar to above, no additional limitations are provided that provide a practical application, or amount to significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim 14 contains limitations similar to those found in claims 3 and 4 and therefore are not patent eligible for the same reasons.
Claim 18 contains limitations similar to those found in claims 10 and 11 and therefore are not patent eligible for the same reasons.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-6, 8, 10-14, 16, and 19 are rejected under 35 U.S.C. 103 as obvious over US 20250005051 A1, (Khosla; Sopan) in view of US 20140297571 A1, (Beamon; Bridget B.).
Claim 1, 12, and 19
Regarding Claim 1, Khosla teach
1. A method comprising:
receiving a text query from a user;
(paragraph 70 "At block 402, the natural language question answering service 102 receives a natural language question (or prompt) from one of the customer computing devices 122 (e.g., where a user of the device entered the question via UI). As stated herein, the user of the customer computing devices 122 may be a customer of a network-based service associated with the natural language question answering service 102. Additionally, or alternatively, the question may be submitted or input via APIs. The question may also be generated by another generative model (e.g., not the LLM component 106) such that two models can effectively communicate with one another without human intervention. In this context, the user may seek an answer specifically related to the network-based service they have subscribed to. For example, the user may ask the natural language question answering service natural language question answering service 102 how to create a bucket in a network-based storage service. As another example, the user may ask the natural language question answering service 102 to create an API call, or create and run an API call (e.g., the user asks “please create a bucket for me named ‘bucket3’ in my network-based storage service”).")
identifying first text chunks based on similarities between the text query and a plurality of text chunks;
(paragraph 71 "At block 404, the natural language question answering service 102 may determine via the aggregator component 104, relevant passages to retrieve to answer the question (or prompt). As stated above, the aggregator component 104 may utilize partial string matching techniques to determine words or phrases from question input that may have been misspelled or mis-keyed by a user (e.g., user put in “Londin” but meant “London”). This is done to at least determine the meaning of the question."
paragraph 72 "At block 406, aggregator component 104 retrieves passages (e.g., documents, links, API calls, multimedia, etc.) related to the answer from the search systems 124. The aggregator component 104 may retrieve whole documents of from the search systems 124 or retrieve certain text (e.g., inline text) from documents but not the whole document. As stated herein, the passages may be retrieved in QA pair form which accompany the passages, among other forms of passages. The aggregator component 104 generates a prompt with some of the retrieved passages and the question where the prompt may comprise partial passages and associated QA pairs and some form of the original natural language question (or prompt). The aggregator component 104 may use a similarity score to determine whether the retrieved passages, and QA pairs, are above a threshold when compared to the natural language question (e.g., if above a threshold then the passages and QA pairs are allowed to be returned to the LLM component 106 as a prompt). As described above, the natural language question answering service 102 may optionally determine user context via the user context component 105 where the user context may also be provided to the LLM component 106 for the LLM component 106 to generate an answer."
Paragraph 45 "The aggregator component 104 may also use a similarity score to determine which passages retrieved are relevant (e.g., not out of scope). The retrieved passages may be sent through a dense encoder and to get their dense embeddings. Scores may be generated by the aggregator component 104 for each passage in relativity to the natural language question (e.g., how well the passage is related to the question). The passages which pass a certain threshold (e.g., greater than 0.5) may be kept while passages under a certain threshold may not be kept (e.g., less than or equal to 0.5). The passages that are not kept may be deemed out of scope by the aggregator component 104."
Paragraph 46 "…Given a question or query, the aggregator component 104 may use DPR techniques to retrieve relevant passages from an index based on the similarity between their representations and the representation of a query or question. Once the relevant passages are retrieved, the aggregator component 104 may use a downstream model to extract answers from the question asked."
Passages/certain text is being interpreted as chunks)
determining first score information associated with each of the first text chunks;
(Paragraph 45 "The aggregator component 104 may also use a similarity score to determine which passages retrieved are relevant (e.g., not out of scope). The retrieved passages may be sent through a dense encoder and to get their dense embeddings. Scores may be generated by the aggregator component 104 for each passage in relativity to the natural language question (e.g., how well the passage is related to the question). The passages which pass a certain threshold (e.g., greater than 0.5) may be kept while passages under a certain threshold may not be kept (e.g., less than or equal to 0.5). The passages that are not kept may be deemed out of scope by the aggregator component 104.")
generating a first prompt based on the first score information, the first prompt including the text query and the first text chunks;
(paragraph 12 " …For purposes of the present application, prompt can correspond to a few selected passages (e.g., from all the passages retrieved) and the QA pairs of the selected passages, along with the natural language question (e.g., or a form of the question where question may be a prompt or a command)."
Paragraph 22 "The aggregator component 104 can retrieve passages from the search systems 124 based on the natural language question and create a prompt for the LLM component 106. For example, the aggregator component 104 may analyze the natural language question by using string matching techniques (e.g., partial string matching, dense passage retrieval, etc.) to determine the meaning of the natural language question. After determining the meaning of the natural language question, the aggregator component 104 may then determine which of the search systems 124 the aggregator component 104 may retrieve passages from (e.g., retrieve from a network-based storage service QA pair system but not a network-based AI service QA pair system) based on the natural language question. Based on the retrieved passages (e.g., documents, text of the documents, pictures of the documents, or video of the documents, etc.) from the search systems 124, the aggregator component 104 may create a prompt
for the LLM component 106 to answer, where the prompt from the aggregator component 104 to the LLM component 106 may contain some of the retrieved passages and a form of the natural language question. The aggregator component 104 may be a machine learning model trained on Retrieval Augmented Generation (RAG) techniques."
paragraph 72 "… The aggregator component 104 may use a similarity score to determine whether the retrieved passages, and QA pairs, are above a threshold when compared to the natural language question (e.g., if above a threshold then the passages and QA pairs are allowed to be returned to the LLM component 106 as a prompt). As described above, the natural language question answering service 102 may optionally determine user context via the user context component 105 where the user context may also be provided to the LLM component 106 for the LLM component 106 to generate an answer.")
transmitting the first prompt to a text generation model;
(Paragraph 22 "The aggregator component 104 can retrieve passages from the search systems 124 based on the natural language question and create a prompt for the LLM component 106. For example, the aggregator component 104 may analyze the natural language question by using string matching techniques (e.g., partial string matching, dense passage retrieval, etc.) to determine the meaning of the natural language question…"
Paragraph 24 " The LLM component 106 may receive the prompt from the aggregator component 104, the user context (optionally) from user context component 105, and generate one or more answers based on the prompt and the user context….")
receiving a response to the first prompt from the text generation model;
(paragraph 24 "The LLM component 106 may receive the prompt from the aggregator component 104, the user context (optionally) from user context component 105, and generate one or more answers based on the prompt and the user context. The LLM component 106 may be trained on at least QA pairs generated from the search systems 124 or knowledge graphs of customers of network-based services (e.g., a network-based on-demand computing service which is serverless, etc.). The LLM component 106 may take the prompt and the user context received from the aggregator component 104 and utilize a generative AI model (e.g., Retrieval Augmented Generation (RAG) utilizing natural language processing (NLP) architecture) to determine an answer to the natural language question. For example, if the customer computing devices 122 sends a natural language question regarding how to setup a type of network-based storage, the LLM component 106 may utilize a trained generative AI model (e.g., trained on QA pairs from a network-based storage service and customer knowledge graphs) to determine an answer (e.g., where the answer provides the instructions and potential API calls to setup the network-based storage) from the prompt received from the aggregator component 104."
Paragraph 73 "At block 408, the natural language question answering service 102 determines via the LLM component 106, an answer (e.g., in human readable text) based on the prompt (e.g., and in some cases user context). The LLM component 106 may utilize RAG techniques to determine the answer. Moreover, the LLM component 106 may utilize customer information such as resource graphs which comprise subscription information of the customer (e.g., current storage usage, current processing usage, periods of CPU usage, etc.) to determine or refine an answer for the customer (e.g., the LLM component 106 determines a customer's service subscription will expire in the next month so includes in the answer that the customer should renew her service). The LLM component 106 may also generate API calls (or run them for the customer) based on the question (e.g., customer wants an API call to create a bucket in a network-based storage service and the LLM component 106 generates it). Additionally, the LLM component 106 may pre-determine questions for customers based on their activity (e.g., referencing a knowledge graph and determining that the customer likes links to other passages rather than answers with long text in the answer itself).")
presenting the response and the first text chunks;
(paragraph 68 " At (11), the watermarking component 110 adds patterns to the answer to make the answer proprietary to the natural language question answering service 102 and verifiable against subsequent copying. In other words, the 110 may embed patterns into the generated text of the answer from the LLM component 106 that is invisible to humans but algorithmically detectable from a short span of tokens (e.g., group of words or where a token equals a single word or group of characters). Tokens may be selected prior to watermarking and the tokens may be promoted during the watermarking of the generated answer. At (11), the watermarking component 110 sends the watermarked answer and retrieved passages to the customer computing devices 122 such that a user of the customer computing devices 122 may view the answer and retrieved passages."
Paragraph 76 "At block 414, the natural language question answering service 102 determines if the answer was generated in error (e.g., hallucinated). If the answer was generated in error, the routine ends. If not, at block 416, the answer and retrieved passages are sent to the customer computing device 122.")
Khosla do not explicitly teach all of receiving a rating of one of the first text chunks from the user; and
updating the first score information associated with the one of the first text chunks based on the rating.
However, Beamon teach
receiving a rating of one of the first text chunks from the user; and
(Paragraph 28 "In addition, with the mechanisms of the illustrative embodiments, the QA system stores information regarding the portions of the corpus of information used in support of the candidate answer being a correct answer for the input question, and in some illustrative embodiments, evidence that detracts from the candidate answer being a correct answer. These portions of the corpus of information, referred to hereafter as "evidence passages," may be output in correlation with the candidate answers to which they apply for review and evaluation by the user via a graphical user interface (GUI). These evidence passages may have relevance scores associated with them indicating a degree to which the evidence passage is believed to be relevant to the input question, as may be determined by the QA system through an analysis of the context of the evidence passage. These matching scores may be used to rank the output of the evidence passages in the GUI and provide an indicator to the user of the relative importance of the evidence passage to the consideration of the candidate answer as a correct answer for the input question."
paragraph 30 "Moreover, GUI elements are provided for selection of evidence passages to be removed from the set of evidence passages associated with a candidate answer or increasing a relevance score associated with the evidence passage. That is, GUI elements are provided that allow a user to override the determined relevance of the evidence passage to the corresponding candidate answer by either eliminating the evidence passage altogether or modifying its relevance score based on the user's subjective determination of the relevance of the evidence passage, either for or against the candidate answer being a correct answer for the input question. Changes made to the set of evidence passages stored for the candidate answer may be automatically used to update the confidence score associated with the candidate answer and modify the ranked listing of candidate answers in the GUI.")
updating the first score information associated with the one of the first text chunks based on the rating.
(Paragraph 121 " FIG. 9 is a flowchart outlining an example operation for modifying an evidence passage portion of a graphical user interface in accordance with one illustrative embodiment. As shown in FIG. 9, the operation starts by receiving a user input to an evidence passage portion of a user collaboration GUI generated in accordance with the illustrative embodiments described herein (step 910). A determination is made as to whether the user input indicates a removal of an evidence passage from the evidence passage portion of the GUI (step 920), such as by selecting a removal GUI element 736 associated with an evidence passage portion 732 in FIG. 7A, for example. If so, the evidence passage is removed from the GUI output (step 930), such as shown and described above with regard to FIG. 7C, for example. If the user input does not indicate a removal of an evidence passage, then a determination is made as to whether the user input is to modify a relevance score for an evidence passage (step 940), e.g., by manipulating a relevance score element 738 of a evidence passage 732 in FIG. 7A, for example. If so, then the output of the evidence passage is updated to reflect the increase/decrease of the relevance score for the evidence passage (step 950). It should be appreciated that, in such a case, the output may be updated so as to re-organize a ranked listing of the evidence passages in the evidence passage portion of the GUI based on the change to the relevance scores, e.g., such that the order of evidence passages is updated by moving the evidence passage to a position in the ranked listing corresponding to its new updated relevance score. Otherwise, if the input is not a removal or modification of the relevance score of an evidence passage, the user input is for drilling-down into the source document information and the operation branches to the operation outlined in FIG. 11 (step 960)."
Paragraph 122 "After either step 930 or 950, the modification to the evidence passage is communicated to the QA system (step 970) which re-evaluates the evidence passages and candidate answers to generate new relevance scores, confidence scores, ranking of candidate answers, ranking of evidence passages relative to one another within a ranked listing of evidence passages for a candidate answer, or the like (step 980). The updated QA system output is provided to the GUI engine which updates the GUI to reflect the new relevance scores, confidence scores, rankings, and the like, generated as a result of the modifications made by the user via the GUI (step 990). The operation then terminates.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Khosla to incorporate the teachings of Beamon to provide a “receiving a rating of one of the first text chunks from the user; and updating the first score information associated with the one of the first text chunks based on the rating.” Doing so would Improve accuracy, system performance, and confidence of the system, as recognized by Beamon. (Paragraph 46).
Regarding Claim 12, Khosla in view of Beamon, further Khosla teaches
12. A system comprising:
a memory storing executable program code; and
(Paragraph 30 "The network interface 208 may provide connectivity to one or more networks or computing systems, such as the network 116 of FIG. 1. The processing unit 206 may thus receive information and instructions from other computing systems or services via a network. The processing unit 206 may also communicate to and from memory 214 and further provide output information for an optional display (e.g., via a UI) via the input/output device interface 212. In some embodiments, the aggregator component 104 may include more (or fewer) components than those shown in FIG. 2A.")
at least one processing unit to execute the program code to cause the system to perform operations comprising:
(Paragraph 30 "The network interface 208 may provide connectivity to one or more networks or computing systems, such as the network 116 of FIG. 1. The processing unit 206 may thus receive information and instructions from other computing systems or services via a network. The processing unit 206 may also communicate to and from memory 214 and further provide output information for an optional display (e.g., via a UI) via the input/output device interface 212. In some embodiments, the aggregator component 104 may include more (or fewer) components than those shown in FIG. 2A.")
Claim 12 contains limitations similar to those found in claims 1 and therefore are not patent eligible for the same reasons.
Regarding Claim 19, Khosla in view of Beamon, further Khosla teaches
19. One or more non-transitory computer-readable recording media storing program code, the program code executable by at least one processing unit of a computing system to cause the computing system to perform operations comprising:
(Paragraph 30 "The network interface 208 may provide connectivity to one or more networks or computing systems, such as the network 116 of FIG. 1. The processing unit 206 may thus receive information and instructions from other computing systems or services via a network. The processing unit 206 may also communicate to and from memory 214 and further provide output information for an optional display (e.g., via a UI) via the input/output device interface 212. In some embodiments, the aggregator component 104 may include more (or fewer) components than those shown in FIG. 2A.")
Claim 19 contains limitations similar to those found in claims 1 and therefore are not patent eligible for the same reasons.
Claim 2, 13
Regarding Claim 2 and 13, Khosla in view of Beamon, further Beamon teaches
2. The method of Claim 1, wherein presenting the response and the first text chunks comprises:
determining a composite score based on the first score information associated with each of the first text chunks; and
(Paragraph 81 " In one illustrative embodiment, the relevance score for a particular portion of the corpus of data/information, or evidence passage, may be combined with relevance scores for other portions of the corpus of data/information to generate a confidence score as a whole for the hypothesis such that the confidence score is a function of the relevance scores contributing to the evaluation of the particular hypothesis. At this stage 550, there may be thousands of pieces of evidence, e.g., portions of the corpus of data/information, that are evaluated and hundreds of thousands of scores generated by the many different reasoning algorithms utilized."
Paragraph 82 "In the synthesis stage 560, the large number of relevance scores generated by the various reasoning algorithms may be synthesized into confidence scores for the various hypotheses. This process may involve applying weights to the various scores, where the weights have been determined through training of the statistical model employed by the QA system and/or dynamically updated, as described hereafter. The weighted scores may be processed in accordance with a statistical model generated through training of the QA system that identifies a manner by which these scores may be combined to generate a confidence score or measure for the individual hypotheses or candidate answers. This confidence score or measure summarizes the level of confidence that the QA system has about the evidence that the candidate answer is inferred by the input question, i.e. that the candidate answer is the correct answer for the input question.")
presenting an indicator of the composite score with the response.
(Paragraph 83 "The resulting confidence scores or measures are processed by a final confidence merging and ranking stage 570 which may compare the confidence scores and measures, compare them against predetermined thresholds, or perform any other analysis on the confidence scores to determine which hypotheses/candidate answers are the most likely to be the answer to the input question. The hypotheses/candidate answers may be ranked according to these comparisons to generate a ranked listing of hypotheses/candidate answers (hereafter simply referred to as "candidate answers"). From the ranked listing of candidate answers, at stage 580, a final answer and confidence score, or final set of candidate answers and confidence scores, may be generated and output to the submitter of the original input question."
Paragraph 104 "The candidate answer portion 720 further includes a field 722 for listing the input question for which the candidate answers were generated, a field 724 for outputting a ranked listing of candidate answers with corresponding GUI elements 726 for removal of the candidate answers from the ranked listing of candidate answers, and a field 728 for free-form entry of a new candidate answer to be added to the ranked listing of candidate answers. GUI elements 729 may further be provided for indicating a corresponding confidence measure associated with the candidate answer as generated by the QA system. In some illustrative embodiments, the GUI elements 729 associated with the candidate answers may be user selectable and modifiable so that the user may specify his/her own subjective determination of the confidence of the candidate answer. This may cause changes in the relative ranking of the candidate answers within the candidate answer portion 720 as well as modifying the graphical display of the GUI elements 729 to reflect the user specified confidence in the corresponding candidate answer.")
See claim one for rationale.
Claim 3
Regarding Claim 3, Khosla in view of Beamon, further Khosla teaches
The method of Claim 2, wherein the first score information associated with each of the first text chunks includes a first score associated with each of the first text chunks, and wherein presenting the response and the first text chunks comprises:
(Paragraph 45 "The aggregator component 104 may also use a similarity score to determine which passages retrieved are relevant (e.g., not out of scope). The retrieved passages may be sent through a dense encoder and to get their dense embeddings. Scores may be generated by the aggregator component 104 for each passage in relativity to the natural language question (e.g., how well the passage is related to the question). The passages which pass a certain threshold (e.g., greater than 0.5) may be kept while passages under a certain threshold may not be kept (e.g., less than or equal to 0.5). The passages that are not kept may be deemed out of scope by the aggregator component 104."
paragraph 68 " At (11), the watermarking component 110 adds patterns to the answer to make the answer proprietary to the natural language question answering service 102 and verifiable against subsequent copying. In other words, the 110 may embed patterns into the generated text of the answer from the LLM component 106 that is invisible to humans but algorithmically detectable from a short span of tokens (e.g., group of words or where a token equals a single word or group of characters). Tokens may be selected prior to watermarking and the tokens may be promoted during the watermarking of the generated answer. At (11), the watermarking component 110 sends the watermarked answer and retrieved passages to the customer computing devices 122 such that a user of the customer computing devices 122 may view the answer and retrieved passages."
Paragraph 76 "At block 414, the natural language question answering service 102 determines if the answer was generated in error (e.g., hallucinated). If the answer was generated in error, the routine ends. If not, at block 416, the answer and retrieved passages are sent to the customer computing device 122.")
Khosla in view of Beamon, further Beamon teaches presenting each of the first text chunks with an indicator of the first score associated with the first text chunk.
(Paragraph 108 " Each evidence passage entry 732 in the evidence passage portion 730 is rendered such that sub-portions 734 of the evidence passage entry 732 may be user selectable to automatically generate a new candidate answer in the candidate answer portion 720. In addition, each evidence passage entry 732 may further have an associated GUI element 736 for removing the evidence passage from the evidence passages associated with a candidate answer. Furthermore, GUI element 738 is provided for outputting an indication of a relevance score associated with the evidence passage. The GUI element 738 may be user manipulated so as to adjust the indication of the relevance score and thus, modify the relevance score based on the user's subjective determination of the relevance of the evidence passage to the candidate answer, e.g., selecting a different number of bars, stars, entry of a numerical value indicative of relevance on a scale of relevance scores, or the like. In addition, drill-down GUI elements 739 are provided for user selection to drill-down to source document information for output to the user for further evaluation of the corresponding evidence passage and its related context, source document veracity information, and the like.")
See claim one for rationale.
Claim 4
Regarding Claim 4, Khosla in view of Beamon, further Beamon teaches
4. The method of Claim 3, wherein receiving the rating of the one of the first text chunks comprises:
receiving a selection of an indicator associated with the one of the first text chunks which is different from the presented indicator of the first score associated with the one of the first text chunks.
(Paragraph 108 " Each evidence passage entry 732 in the evidence passage portion 730 is rendered such that sub-portions 734 of the evidence passage entry 732 may be user selectable to automatically generate a new candidate answer in the candidate answer portion 720. In addition, each evidence passage entry 732 may further have an associated GUI element 736 for removing the evidence passage from the evidence passages associated with a candidate answer. Furthermore, GUI element 738 is provided for outputting an indication of a relevance score associated with the evidence passage. The GUI element 738 may be user manipulated so as to adjust the indication of the relevance score and thus, modify the relevance score based on the user's subjective determination of the relevance of the evidence passage to the candidate answer, e.g., selecting a different number of bars, stars, entry of a numerical value indicative of relevance on a scale of relevance scores, or the like. In addition, drill-down GUI elements 739 are provided for user selection to drill-down to source document information for output to the user for further evaluation of the corresponding evidence passage and its related context, source document veracity information, and the like."
Paragraph 112 "FIG. 7C illustrates a GUI output generated by the GUI engine in response to a user selection to remove an evidence passage from the evidence passage portion of the GUI in accordance with one illustrative embodiment. The removal of an evidence passage from the evidence passage portion 730 is best seen when comparing FIG. 7C to FIG. 7A which depicts the initial GUI output generated. As shown in FIG. 7C, when compared to FIG. 7A, the evidence passage 750 has been removed by the user's selection of the removal GUI element 736 corresponding to the evidence passage 750. That is, while the evidence passage 750 referencing Vladimir Putin's dog Koni was initially found by the QA system to be relevant to a candidate answer of the input question, the user may determine that Koni is not in fact a close advisor to Vladimir Putin and thus, the evidence passage may be removed from the evidence passage listing for the candidate answer. As a result, the evidence passage 750 is removed in response to the user selecting the corresponding removal GUI element 736."
Paragraph 113 " In addition, because this evidence passage 750 contributed to the confidence score associated with the candidate answer, the confidence score representation 729 associated with the corresponding candidate answer may be updated to reflect any change in the confidence score due to the elimination of the evidence passage. This update is automatically performed in response to the user's input removing the evidence passage, effectively indicating that the evidence passage is not relevant to the evaluation of the candidate answer. Thus, for example, if the Koni evidence passage was negatively affecting the confidence score for the corresponding candidate answer, the removal of the evidence passage 750 may result in a confidence score for the corresponding candidate answer being increased as the evidence passage 750 is no longer detracting from the confidence score for the candidate answer."
Indicator of the first score is relevance score indicator 738)
See claim one for rationale.
Claim 5
Regarding Claim 5, Khosla in view of Beamon, further Beamon teaches
5. The method of Claim 1, wherein the first score information associated with each of the first text chunks includes a first score associated with each of the first text chunks, and wherein presenting the response and the first text chunks comprises:
(Paragraph 99 "The entries for the evidence passages that are output via the evidence portion of the GUI, as generated by the evidence passage engine 636, may include a representation of the evidence portion and an associated relevance score for the evidence portion as generated by the evaluations performed by the QA system 610. Moreover, the entries may include links to the source documents for the evidence passages for purpose of implementing the drill-down functionality previously described. The drill-down functionality may be facilitated by logic provided in the evidence passage engine 636 which is invoked in response to user input being received via the user interface 642 selecting the link in the entry for the evidence passage.")
presenting each of the first text chunks with an indicator of the first score associated with the first text chunk.
(Paragraph 108 " Each evidence passage entry 732 in the evidence passage portion 730 is rendered such that sub-portions 734 of the evidence passage entry 732 may be user selectable to automatically generate a new candidate answer in the candidate answer portion 720. In addition, each evidence passage entry 732 may further have an associated GUI element 736 for removing the evidence passage from the evidence passages associated with a candidate answer. Furthermore, GUI element 738 is provided for outputting an indication of a relevance score associated with the evidence passage. The GUI element 738 may be user manipulated so as to adjust the indication of the relevance score and thus, modify the relevance score based on the user's subjective determination of the relevance of the evidence passage to the candidate answer, e.g., selecting a different number of bars, stars, entry of a numerical value indicative of relevance on a scale of relevance scores, or the like. In addition, drill-down GUI elements 739 are provided for user selection to drill-down to source document information for output to the user for further evaluation of the corresponding evidence passage and its related context, source document veracity information, and the like.")
See claim one for rationale.
Claim 6
Regarding Claim 6, Khosla in view of Beamon, further Beamon teaches
6. The method of Claim 5, wherein receiving the rating of the one of the first text chunks comprises:
receiving a selection of an indicator associated with the one of the first text chunks which is different from the presented indicator of the first score associated with the one of the first text chunks.
(Paragraph 108 " Each evidence passage entry 732 in the evidence passage portion 730 is rendered such that sub-portions 734 of the evidence passage entry 732 may be user selectable to automatically generate a new candidate answer in the candidate answer portion 720. In addition, each evidence passage entry 732 may further have an associated GUI element 736 for removing the evidence passage from the evidence passages associated with a candidate answer. Furthermore, GUI element 738 is provided for outputting an indication of a relevance score associated with the evidence passage. The GUI element 738 may be user manipulated so as to adjust the indication of the relevance score and thus, modify the relevance score based on the user's subjective determination of the relevance of the evidence passage to the candidate answer, e.g., selecting a different number of bars, stars, entry of a numerical value indicative of relevance on a scale of relevance scores, or the like. In addition, drill-down GUI elements 739 are provided for user selection to drill-down to source document information for output to the user for further evaluation of the corresponding evidence passage and its related context, source document veracity information, and the like.")
See claim one for rationale.
Claim 8, 16,
Regarding Claim 8,16, Khosla in view of Beamon, further Khosla teaches
8. The method of Claim 1, further comprising:
receiving a second text query from a second user;
(paragraph 63 "The LLM component 106 may also be configured to process multiple questions (or prompts) from a user (e.g., customer of a network-based service or services) of a customer computing device 122 such that the LLM component 106 may utilize previous asked questions (or prompts) and previously provided answers to form an evidence pool. The evidence pool may be used to provide an answer to a current question. The LLM component 106 may store the multiple previous questions as conversational context. The LLM component 106 may utilize this evidence pool, current natural language question, and conversional context, to answer the current question. The LLM component 106 may update the evidence pool in different ways. In one example, the LLM component 106 may update the evidence pool with any questions and/or answers generated by the LLM component 106. In another example, the LLM component 106 may update the evidence pool if the LLM component 106 determines that the current evidence pool does not contain an answer to a natural language question. In this instance, the LLM component 106 may request more up to date passages to populate the evidence pool (e.g., from the aggregator component aggregator component 104). In another example, the LLM component 106 may use the evidence pool to determine if a current natural language question should be rewritten based on information in the evidence pool.")
identifying second text chunks based on similarities between the second text query and the plurality of text chunks;
(Paragraph 45 "The aggregator component 104 may also use a similarity score to determine which passages retrieved are relevant (e.g., not out of scope). The retrieved passages may be sent through a dense encoder and to get their dense embeddings. Scores may be generated by the aggregator component 104 for each passage in relativity to the natural language question (e.g., how well the passage is related to the question). The passages which pass a certain threshold (e.g., greater than 0.5) may be kept while passages under a certain threshold may not be kept (e.g., less than or equal to 0.5). The passages that are not kept may be deemed out of scope by the aggregator component 104."
Paragraph 46 "…Given a question or query, the aggregator component 104 may use DPR techniques to retrieve relevant passages from an index based on the similarity between their representations and the representation of a query or question. Once the relevant passages are retrieved, the aggregator component 104 may use a downstream model to extract answers from the question asked."
Passages/certain text is being interpreted as chunks)
determining second score information associated with each of the second text chunks;
(Paragraph 45 "The aggregator component 104 may also use a similarity score to determine which passages retrieved are relevant (e.g., not out of scope). The retrieved passages may be sent through a dense encoder and to get their dense embeddings. Scores may be generated by the aggregator component 104 for each passage in relativity to the natural language question (e.g., how well the passage is related to the question). The passages which pass a certain threshold (e.g., greater than 0.5) may be kept while passages under a certain threshold may not be kept (e.g., less than or equal to 0.5). The passages that are not kept may be deemed out of scope by the aggregator component 104.")
generating a second prompt based on the second score information, the second prompt including the second text query and the second text chunks;
(Paragraph 45 "The aggregator component 104 may also use a similarity score to determine which passages retrieved are relevant (e.g., not out of scope). The retrieved passages may be sent through a dense encoder and to get their dense embeddings. Scores may be generated by the aggregator component 104 for each passage in relativity to the natural language question (e.g., how well the passage is related to the question). The passages which pass a certain threshold (e.g., greater than 0.5) may be kept while passages under a certain threshold may not be kept (e.g., less than or equal to 0.5). The passages that are not kept may be deemed out of scope by the aggregator component 104."
Paragraph 47 "The aggregator component 104 can create a prompt based on the relevant passages determined and the natural language question. The aggregator component 104 may create a prompt that contains some of the passages retrieved and QA pairs that are associated with those passages. The aggregator component 104 may also create the prompt by adding QA pairs of those passages to the prompt. For example, the aggregator component 104 can take some passages from a document on how to setup a network-based storage bucket and at least part of the natural language question to formulate a prompt (e.g., and also take the QA pairs that are associated with that document). Additionally, the aggregator component 104 may add the question, or some form of the question, to the prompt.")
transmitting the second prompt to the text generation model;
(paragraph 61 "At (4), the aggregator component 104 sends some of the passages retrieved and corresponding QA pairs to the user context component 105. At (5), the user context component 105 determines user context associated with a user of the customer computing device 122 and forwards the QA pairs, passages retrieved, and user context to the LLM component 106. It should be noted that (5) is optional and the aggregator component 104 may directly send the passages retrieved and the QA pairs to the LLM component 106 without user context. At (6), the LLM component 106 receives the prompt and user context and determines an answer to the natural language question. The LLM component 106 may utilize APIs to receive the prompt and the user context from the user context component 105 where the prompt may be formatted in a certain manner to be sent or communicated over an API.")
receiving a second response to the second prompt from the text generation model;
(paragraph 61 "At (4), the aggregator component 104 sends some of the passages retrieved and corresponding QA pairs to the user context component 105. At (5), the user context component 105 determines user context associated with a user of the customer computing device 122 and forwards the QA pairs, passages retrieved, and user context to the LLM component 106. It should be noted that (5) is optional and the aggregator component 104 may directly send the passages retrieved and the QA pairs to the LLM component 106 without user context. At (6), the LLM component 106 receives the prompt and user context and determines an answer to the natural language question. The LLM component 106 may utilize APIs to receive the prompt and the user context from the user context component 105 where the prompt may be formatted in a certain manner to be sent or communicated over an API.")
presenting the second response and the second text chunks;
(paragraph 68 " At (11), the watermarking component 110 adds patterns to the answer to make the answer proprietary to the natural language question answering service 102 and verifiable against subsequent copying. In other words, the 110 may embed patterns into the generated text of the answer from the LLM component 106 that is invisible to humans but algorithmically detectable from a short span of tokens (e.g., group of words or where a token equals a single word or group of characters). Tokens may be selected prior to watermarking and the tokens may be promoted during the watermarking of the generated answer. At (11), the watermarking component 110 sends the watermarked answer and retrieved passages to the customer computing devices 122 such that a user of the customer computing devices 122 may view the answer and retrieved passages.")
Khosla in view of Beamon, further Beamon teaches
receiving a second rating of one of the second text chunks from the second user; and
(Paragraph 30 "Moreover, GUI elements are provided for selection of evidence passages to be removed from the set of evidence passages associated with a candidate answer or increasing a relevance score associated with the evidence passage. That is, GUI elements are provided that allow a user to override the determined relevance of the evidence passage to the corresponding candidate answer by either eliminating the evidence passage altogether or modifying its relevance score based on the user's subjective determination of the relevance of the evidence passage, either for or against the candidate answer being a correct answer for the input question. Changes made to the set of evidence passages stored for the candidate answer may be automatically used to update the confidence score associated with the candidate answer and modify the ranked listing of candidate answers in the GUI."
paragraph 32 "The changes made, via the GUI, to the ranked listing of candidate answers and sets of evidence passages associated with the candidate answers, may be stored for later retrieval and use. Such information may be used to assist in training of the QA system, such as by adjusting scoring parameters, adjusting weights associated with documents or sources of content in the corpus of content, and other operational parameters of the QA system. During runtime, the stored set of candidate answers and corresponding evidence passages may be used to assist in responding to the same or similar answers being submitted by the same or other users at a later time. For example, when an input question is received and parsed to generate one or more queries, the stored information may be searched to find entries having similar queries to that of the input question so that corresponding candidate answers and supporting evidence passages may be quickly retrieved and used to generate an answer or set of candidate answers for the input question. Moreover, the stored set of candidate answers and corresponding evidence passages may be used by an analyst to compare to subsequent executions on a same or similar question to evaluate if and how the corpus of information has been modified since the stored set of candidate answers and corresponding evidence passages was generated. There is a plethora of potential uses of the results generated by the operation of the illustrative embodiments, any of which are intended to be within the spirit and scope of the illustrative embodiments."
Paragraph 86 "Moreover, the GUI may include GUI elements for invoking logic and functionality of the GUI for removing evidence passages from the listing of associated evidence passages for the various candidate answers and/or modifying a relevance score associated with the evidence passage. In this way, the user essentially supersedes the evaluation made by the QA system pipeline 500 and instead imposes the user's subjective determination as to the relevance of an evidence passage by either eliminating it altogether or increasing/reducing the relevance score associated with the evidence passage to indicate the user's own subjective evaluation of the evidence passage's relevance to the candidate answer being the correct answer for the input question.")
updating the second score information associated with the one of the second text chunks based on the second rating.
(Paragraph 88 " Should the user eliminate the evidence passage or modify the evidence passage's relevance score in some manner, the QA system pipeline 500 may automatically adjust the relevance scores, confidence scores, and ranked listing of candidate answers based on the change to the evidence passage. In this way, the QA system pipeline 500 may dynamically adjust its output based on user collaboration with the QA system to provide the user's subject determination of the relevance, reliability, and correctness of the evidence passages and/or the candidate answers themselves."
Paragraph 100 "Moreover, the evidence passage engine 636 generates the evidence passage portion of the GUI with GUI elements for removing evidence passages or modifying the corresponding relevance scores associated with the evidence passages based on user input. In response to a user providing a user input via the user interface 642 that selects a GUI element for removing an evidence passage, the corresponding evidence passage is eliminated from the GUI output and the change is submitted to the QA system 610 for dynamic re-evaluation of the candidate answers. Similarly, in response to the user providing a user input for modifying the relevance score for the evidence passage, the change is communicated to the QA system 610 which may dynamically re-evaluate the candidate answers based on the received change."
Paragraph 101 "The dynamic update engine 640 comprises logic for coordinating the user modifications and selections of GUI elements received via the user interface 642. This may involve coordinating the updating of the evidence passage portion and candidate answer portions of the GUI as well as the submission of the modifications to the QA system 610 for re-evaluation of the candidate answers and/or evidence passages associated with the candidate answers. The resulting candidate answers and associated evidence passages generated via the operation of the QA system 610 and the user collaboration provided via the GUI engine 630 may be stored in the candidate answer evidence passage storage system 650 for later retrieval and use.")
See claim one for rationale.
Claim 10
Regarding Claim 10, Khosla in view of Beamon, further Beamon teaches
10. The method of Claim 8, wherein the first score information associated with each of the first text chunks includes a first score associated with each of the first text chunks,
(Paragraph 28 "In addition, with the mechanisms of the illustrative embodiments, the QA system stores information regarding the portions of the corpus of information used in support of the candidate answer being a correct answer for the input question, and in some illustrative embodiments, evidence that detracts from the candidate answer being a correct answer. These portions of the corpus of information, referred to hereafter as "evidence passages," may be output in correlation with the candidate answers to which they apply for review and evaluation by the user via a graphical user interface (GUI). These evidence passages may have relevance scores associated with them indicating a degree to which the evidence passage is believed to be relevant to the input question, as may be determined by the QA system through an analysis of the context of the evidence passage. These matching scores may be used to rank the output of the evidence passages in the GUI and provide an indicator to the user of the relative importance of the evidence passage to the consideration of the candidate answer as a correct answer for the input question.")
wherein presenting the response and the first text chunks comprises presenting each of the first text chunks with an indicator of the first score associated with the first text chunk,
(paragraph 80 "During normal runtime operation of the QA system, as part of the hypothesis generation stage 540 and the deep analysis performed during the stage 550, portions of the corpus, i.e. evidence passages, are collected and stored as evidence in support of (justifying) or in non-support of (non-justifying) a particular hypothesis (or answer) being correct for the input question (Q). After candidate answer list generation is performed, as described hereafter, the evidence passages themselves may be scored based on the justifying passage model and then ranked. The resulting ranked list of evidence passages may be output in an evidence passage portion of a graphical user interface, as further described hereafter. The ranked listings may be associated with the particular candidate answers for which they are justifying/non-justifying, however at the system level, the ranked listing and the candidate answers may become uncoupled, and each listing may have evidence passages ranked relative to other evidence passages within the same ranked listing. Moreover, entries in the output of the ranked list of evidence passages may have graphical user interface elements indicating a score associated with the particular evidence passage, referred to herein as a relevance score. The generation of relevance scores for evidence passages is further described hereafter with reference to FIG. 12."
Paragraph 99 "The entries for the evidence passages that are output via the evidence portion of the GUI, as generated by the evidence passage engine 636, may include a representation of the evidence portion and an associated relevance score for the evidence portion as generated by the evaluations performed by the QA system 610. Moreover, the entries may include links to the source documents for the evidence passages for purpose of implementing the drill-down functionality previously described. The drill-down functionality may be facilitated by logic provided in the evidence passage engine 636 which is invoked in response to user input being received via the user interface 642 selecting the link in the entry for the evidence passage.")
wherein the second score information associated with each of the second text chunks includes a second score associated with each of the second text chunks, and
(Paragraph 32 "The changes made, via the GUI, to the ranked listing of candidate answers and sets of evidence passages associated with the candidate answers, may be stored for later retrieval and use. Such information may be used to assist in training of the QA system, such as by adjusting scoring parameters, adjusting weights associated with documents or sources of content in the corpus of content, and other operational parameters of the QA system. During runtime, the stored set of candidate answers and corresponding evidence passages may be used to assist in responding to the same or similar answers being submitted by the same or other users at a later time. For example, when an input question is received and parsed to generate one or more queries, the stored information may be searched to find entries having similar queries to that of the input question so that corresponding candidate answers and supporting evidence passages may be quickly retrieved and used to generate an answer or set of candidate answers for the input question. Moreover, the stored set of candidate answers and corresponding evidence passages may be used by an analyst to compare to subsequent executions on a same or similar question to evaluate if and how the corpus of information has been modified since the stored set of candidate answers and corresponding evidence passages was generated. There are a plethora of potential uses of the results generated by the operation of the illustrative embodiments, any of which are intended to be within the spirit and scope of the illustrative embodiments."
Paragraph 95 "The question input engine 634 provides fields for the user to enter an input question and may then format the question for submission to the QA system 610. Based on the input question received via the question input engine 634, the QA system 610 performs the input question parsing and analysis, query generation, query application and candidate answer generation, candidate answer and evidence passage evaluation and scoring, etc. as previously described above. The QA system 610 operates on the corpus of data/information 620 to generate the candidate answers (hypotheses), retrieve evidence passages, and perform the various evaluations previously described. The result of the QA system 610 operation is a set of candidate answers, evidence passages associated with the candidate answers, and corresponding relevance and confidence scores which may all be stored in the candidate answer evidence passage storage 650. Moreover, this information may further include links to the source documents in the corpus 620 and other information regarding the veracity and relevancy of the source documents.")
wherein presenting the second response and second text chunks comprises presenting each of the second text chunks with an indicator of the second score associated with the second text chunk.
(Paragraph 32 "The changes made, via the GUI, to the ranked listing of candidate answers and sets of evidence passages associated with the candidate answers, may be stored for later retrieval and use. Such information may be used to assist in training of the QA system, such as by adjusting scoring parameters, adjusting weights associated with documents or sources of content in the corpus of content, and other operational parameters of the QA system. During runtime, the stored set of candidate answers and corresponding evidence passages may be used to assist in responding to the same or similar answers being submitted by the same or other users at a later time. For example, when an input question is received and parsed to generate one or more queries, the stored information may be searched to find entries having similar queries to that of the input question so that corresponding candidate answers and supporting evidence passages may be quickly retrieved and used to generate an answer or set of candidate answers for the input question. Moreover, the stored set of candidate answers and corresponding evidence passages may be used by an analyst to compare to subsequent executions on a same or similar question to evaluate if and how the corpus of information has been modified since the stored set of candidate answers and corresponding evidence passages was generated. There are a plethora of potential uses of the results generated by the operation of the illustrative embodiments, any of which are intended to be within the spirit and scope of the illustrative embodiments."
Paragraph 98 "The evidence passage engine 636 comprises logic for generating a portion of the GUI output that lists the evidence passage contributing to the confidence score for each individual candidate answer. That is, the evidence passage portion of the GUI may be organized by candidate answer with the evidence passages contributing to the confidence score of the candidate answer being displayed in association with the candidate answer. The output of the evidence passages in the evidence passage portion of the GUI is done such that sub-portions of the evidence passages, e.g., words, phrases, sentences, and the like, are selectable by a user via the user interface 642 and the user's own user interface input devices, e.g., keyboard, mouse, microphone, etc., so as to create new candidate answers that are automatically added to the ranked listing of candidate answers in response to such selection."
Paragraph 99 "The entries for the evidence passages that are output via the evidence portion of the GUI, as generated by the evidence passage engine 636, may include a representation of the evidence portion and an associated relevance score for the evidence portion as generated by the evaluations performed by the QA system 610. Moreover, the entries may include links to the source documents for the evidence passages for purpose of implementing the drill-down functionality previously described. The drill-down functionality may be facilitated by logic provided in the evidence passage engine 636 which is invoked in response to user input being received via the user interface 642 selecting the link in the entry for the evidence passage.")
See claim one for rationale.
Claim 11
Regarding Claim 11, Khosla in view of Beamon, further Beamon teaches
11. The method of Claim 10, wherein receiving the rating of the one of the first text chunks comprises:
receiving a selection of an indicator associated with the one of the first text chunks which is different from the presented indicator of the first score associated with the one of the first text chunks, and
(Paragraph 108 "Each evidence passage entry 732 in the evidence passage portion 730 is rendered such that sub-portions 734 of the evidence passage entry 732 may be user selectable to automatically generate a new candidate answer in the candidate answer portion 720. In addition, each evidence passage entry 732 may further have an associated GUI element 736 for removing the evidence passage from the evidence passages associated with a candidate answer. Furthermore, GUI element 738 is provided for outputting an indication of a relevance score associated with the evidence passage. The GUI element 738 may be user manipulated so as to adjust the indication of the relevance score and thus, modify the relevance score based on the user's subjective determination of the relevance of the evidence passage to the candidate answer, e.g., selecting a different number of bars, stars, entry of a numerical value indicative of relevance on a scale of relevance scores, or the like. In addition, drill-down GUI elements 739 are provided for user selection to drill-down to source document information for output to the user for further evaluation of the corresponding evidence passage and its related context, source document veracity information, and the like."
Paragraph 112 "FIG. 7C illustrates a GUI output generated by the GUI engine in response to a user selection to remove an evidence passage from the evidence passage portion of the GUI in accordance with one illustrative embodiment. The removal of an evidence passage from the evidence passage portion 730 is best seen when comparing FIG. 7C to FIG. 7A which depicts the initial GUI output generated. As shown in FIG. 7C, when compared to FIG. 7A, the evidence passage 750 has been removed by the user's selection of the removal GUI element 736 corresponding to the evidence passage 750. That is, while the evidence passage 750 referencing Vladimir Putin's dog Koni was initially found by the QA system to be relevant to a candidate answer of the input question, the user may determine that Koni is not in fact a close advisor to Vladimir Putin and thus, the evidence passage may be removed from the evidence passage listing for the candidate answer. As a result, the evidence passage 750 is removed in response to the user selecting the corresponding removal GUI element 736."
Paragraph 113 "In addition, because this evidence passage 750 contributed to the confidence score associated with the candidate answer, the confidence score representation 729 associated with the corresponding candidate answer may be updated to reflect any change in the confidence score due to the elimination of the evidence passage. This update is automatically performed in response to the user's input removing the evidence passage, effectively indicating that the evidence passage is not relevant to the evaluation of the candidate answer. Thus, for example, if the Koni evidence passage was negatively affecting the confidence score for the corresponding candidate answer, the removal of the evidence passage 750 may result in a confidence score for the corresponding candidate answer being increased as the evidence passage 750 is no longer detracting from the confidence score for the candidate answer.")
wherein receiving the second rating of the one of the second text chunks comprises:
receiving a second selection of a second indicator associated with the one of the second text chunks which is different from the presented indicator of the second score associated with the one of the second text chunks.
(Paragraph 32 "The changes made, via the GUI, to the ranked listing of candidate answers and sets of evidence passages associated with the candidate answers, may be stored for later retrieval and use. Such information may be used to assist in training of the QA system, such as by adjusting scoring parameters, adjusting weights associated with documents or sources of content in the corpus of content, and other operational parameters of the QA system. During runtime, the stored set of candidate answers and corresponding evidence passages may be used to assist in responding to the same or similar answers being submitted by the same or other users at a later time. For example, when an input question is received and parsed to generate one or more queries, the stored information may be searched to find entries having similar queries to that of the input question so that corresponding candidate answers and supporting evidence passages may be quickly retrieved and used to generate an answer or set of candidate answers for the input question. Moreover, the stored set of candidate answers and corresponding evidence passages may be used by an analyst to compare to subsequent executions on a same or similar question to evaluate if and how the corpus of information has been modified since the stored set of candidate answers and corresponding evidence passages was generated. There are a plethora of potential uses of the results generated by the operation of the illustrative embodiments, any of which are intended to be within the spirit and scope of the illustrative embodiments.")
See claim one for rationale.
Claim 14 contains limitations similar to those found in claims 3 and 4 and therefore are not patent eligible for the same reasons.
Claim 18 contains limitations similar to those found in claims 10 and 11 and therefore are not patent eligible for the same reasons.
Claims 7 and 15 are rejected under 35 U.S.C. 103 as obvious over US 20250005051 A1, (Khosla; Sopan) in view of US 20140297571 A1, (Beamon; Bridget B.) in further view of Ye, Fuda, et al. "R2AG: Incorporating retrieval information into retrieval augmented generation." Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.
Claim 7 and 15
Regarding Claim 7 and 15, Khosla in view of Beamon, further Khosla teaches
The method of Claim 1, wherein the first score information associated with each of the first text chunks includes a first score associated with each of the first text chunks,
(Paragraph 45 "The aggregator component 104 may also use a similarity score to determine which passages retrieved are relevant (e.g., not out of scope). The retrieved passages may be sent through a dense encoder and to get their dense embeddings. Scores may be generated by the aggregator component 104 for each passage in relativity to the natural language question (e.g., how well the passage is related to the question). The passages which pass a certain threshold (e.g., greater than 0.5) may be kept while passages under a certain threshold may not be kept (e.g., less than or equal to 0.5). The passages that are not kept may be deemed out of scope by the aggregator component 104.")
Khosla in view of Beamon do not explicitly teach and wherein the first prompt comprises instructions to associate the first text chunks with an importance based on their respective first scores.
However, Ye teaches and wherein the first prompt comprises instructions to associate the first text chunks with an importance based on their respective first scores.
(page 3 and 4 section 3.2 retrieval feature extraction "…Inspired by works in retrieval downstream tasks (Ma et al., 2022; Ye and Li, 2024), we align these representations into retrieval features by computing relevance, precedent similarity, and neighbor similarity scores… The relevance score ri is between the query and the i-th document and is also used to sort the documents... "
Page 4 section 3.4 retrieval aware prompting "In the generation process, it is crucial for the LLM to utilize the retrieval information effectively. As shown in the upper part of Figure 2, we introduce a retrieval-aware prompting strategy that injects the retrieval information extracted by R2-Former into the LLM’s generation process."
Page 5 section 3.4 retrieval aware prompting "For nuanced analysis of each document, the corresponding retrieval information embeddings are then prepended to the front of each document’s embeddings. They are external knowledge and function as an anchor, guiding the LLM to focus on useful documents. "
Appendix B "In R2AG, retrieval information, we append k special tokens(“<R>”) in front of each document to facilitate the incorporation of retrieval information."
Page 5 section 3.4 Retrieval aware prompting "For nuanced analysis of each document, the corresponding retrieval information embeddings are then prepended to the front of each document’s embeddings. They are external knowledge and function as an anchor, guiding the LLM to focus on useful documents. The final input embeddings can be arranged as:")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Khosla in view of Beamon to incorporate the teachings of Ye to provide a “and wherein the first prompt comprises instructions to associate the first text chunks with an importance based on their respective first scores.” Doing so would Eliminate a semantic gap between LLMs and retrievers, as recognized by Ye. (page 1 introduction).
Claims 9 and 17 are rejected under 35 U.S.C. 103 as obvious over US 20250005051 A1, (Khosla; Sopan) in view of US 20140297571 A1, (Beamon; Bridget B.) in further view of US 20220005367 A1, (Gera; Ralucca.)
Claim 9 and 17
Regarding Claim 9 and 17, Khosla in view of Beamon, further Beamon teaches
9. The method of Claim 8, further comprising:
wherein the identified second text chunks include the one of the first text chunks, and
(Paragraph 32 "The changes made, via the GUI, to the ranked listing of candidate answers and sets of evidence passages associated with the candidate answers, may be stored for later retrieval and use. Such information may be used to assist in training of the QA system, such as by adjusting scoring parameters, adjusting weights associated with documents or sources of content in the corpus of content, and other operational parameters of the QA system. During runtime, the stored set of candidate answers and corresponding evidence passages may be used to assist in responding to the same or similar answers being submitted by the same or other users at a later time. For example, when an input question is received and parsed to generate one or more queries, the stored information may be searched to find entries having similar queries to that of the input question so that corresponding candidate answers and supporting evidence passages may be quickly retrieved and used to generate an answer or set of candidate answers for the input question. Moreover, the stored set of candidate answers and corresponding evidence passages may be used by an analyst to compare to subsequent executions on a same or similar question to evaluate if and how the corpus of information has been modified since the stored set of candidate answers and corresponding evidence passages was generated. There are a plethora of potential uses of the results generated by the operation of the illustrative embodiments, any of which are intended to be within the spirit and scope of the illustrative embodiments.")
See claim one for rationale.
Khosla in view of Beamon do not explicitly teach wherein the determined second score information associated with the one of the first text chunks is the updated first score information.
However, Gera teaches wherein the determined second score information associated with the one of the first text chunks is the updated first score information.
(Paragraph 27 "The system 100 requires identifying relevant connections between the users to accurately recommend appropriate new CHUNKlets to users; that is to say it relies on the overlaying social network that emerges between the users of the CHUNK Learning system. The subject technology discloses methods to generate a social network to inform system 100 by having the social network assign and modify a score of each CHUNKlet, to be used for recommendations to other users. The social network's nodes are the individual learner profiles and the edges (weighted and undirected) connect nodes with similar attributes. Attributes from each student profile are extracted and saved. Examples of such attributes are the current degree, branch of military service, previous degrees, and extracurricular interests."
Paragraph 26 "The subject technology uses a two-step process for recommending CHUNKs and CHUNKlets to each user to support personalized education, based on its current structure. First, relevant CHUNKs and CHUNKlets are determined based on the user's academic requirements and goals. Second, for each relevant CHUNK, the relevant CHUNKlets are ranked based on the user's learner profile and the social connections they share with other users in the system. The goal of this process is to maximize the chances that the learner will engage with CHUNKlets that are both useful and interesting to the learner. In one embodiment, the recommendation of a CHUNKlet is based on the learner profile's keywords. As such, personalizing the chosen CHUNKlet for each user is complemented by creating and utilizing a social network that ranks relevant CHUNKIets for each user. The newly proposed rating for each CHUNKlet is generated using the learner's profile and how that information links him/her to similar users, building on the CHUNKlet feedback provided from previous learners in the network. This maximizes the chances learners use methods that work for them. The built-in rating system is used to (1) collect data from users that have completed a CHUNK or CHUNKlet and (2) affect the ranking the CHUNK or CHUNKlet receives for other related users in the network, with the strength of that effect being determined by the strength of the individual's social connection with other users.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Khosla in view of Beamon to incorporate the teachings of Gera to provide a “wherein the determined second score information associated with the one of the first text chunks is the updated first score information.” Doing so would Improve the recommendation the system gives, solves the cold start problem, and also maximizes the chances the learner engages in useful chunklets, as recognized by Gera. (Paragraph 32 & 26).
Claims 20 are rejected under 35 U.S.C. 103 as obvious over US 20250005051 A1, (Khosla; Sopan) in view of US 20140297571 A1, (Beamon; Bridget B.) in further view of Pan, Ruotong, et al. "Not all contexts are equal: Teaching llms credibility-aware generation." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.
Claim 20
Regarding Claim 20 Khosla in view of Beamon, further Khosla teaches
20. The one or more non-transitory computer-readable recording media of Claim 19, wherein the first prompt comprises instructions to associate the first text portions with an importance based on their respective first scores, the operations further comprising:
receiving a second text query from a second user;
(paragraph 63 "The LLM component 106 may also be configured to process multiple questions (or prompts) from a user (e.g., customer of a network-based service or services) of a customer computing device 122 such that the LLM component 106 may utilize previous asked questions (or prompts) and previously provided answers to form an evidence pool. The evidence pool may be used to provide an answer to a current question. The LLM component 106 may store the multiple previous questions as conversational context. The LLM component 106 may utilize this evidence pool, current natural language question, and conversional context, to answer the current question. The LLM component 106 may update the evidence pool in different ways. In one example, the LLM component 106 may update the evidence pool with any questions and/or answers generated by the LLM component 106. In another example, the LLM component 106 may update the evidence pool if the LLM component 106 determines that the current evidence pool does not contain an answer to a natural language question. In this instance, the LLM component 106 may request more up to date passages to populate the evidence pool (e.g., from the aggregator component aggregator component 104). In another example, the LLM component 106 may use the evidence pool to determine if a current natural language question should be rewritten based on information in the evidence pool.")
determining, from the plurality of stored text portions, second text portions which are semantically similar to the second query;
(Paragraph 45 "The aggregator component 104 may also use a similarity score to determine which passages retrieved are relevant (e.g., not out of scope). The retrieved passages may be sent through a dense encoder and to get their dense embeddings. Scores may be generated by the aggregator component 104 for each passage in relativity to the natural language question (e.g., how well the passage is related to the question). The passages which pass a certain threshold (e.g., greater than 0.5) may be kept while passages under a certain threshold may not be kept (e.g., less than or equal to 0.5). The passages that are not kept may be deemed out of scope by the aggregator component 104."
Paragraph 46 "…Given a question or query, the aggregator component 104 may use DPR techniques to retrieve relevant passages from an index based on the similarity between their representations and the representation of a query or question. Once the relevant passages are retrieved, the aggregator component 104 may use a downstream model to extract answers from the question asked."
Passages/certain text is being interpreted as chunks)
determining second score information associated with each of the second text chunks;
(Paragraph 45 "The aggregator component 104 may also use a similarity score to determine which passages retrieved are relevant (e.g., not out of scope). The retrieved passages may be sent through a dense encoder and to get their dense embeddings. Scores may be generated by the aggregator component 104 for each passage in relativity to the natural language question (e.g., how well the passage is related to the question). The passages which pass a certain threshold (e.g., greater than 0.5) may be kept while passages under a certain threshold may not be kept (e.g., less than or equal to 0.5). The passages that are not kept may be deemed out of scope by the aggregator component 104.")
transmitting the second prompt to the text generation model;
(paragraph 61 "At (4), the aggregator component 104 sends some of the passages retrieved and corresponding QA pairs to the user context component 105. At (5), the user context component 105 determines user context associated with a user of the customer computing device 122 and forwards the QA pairs, passages retrieved, and user context to the LLM component 106. It should be noted that (5) is optional and the aggregator component 104 may directly send the passages retrieved and the QA pairs to the LLM component 106 without user context. At (6), the LLM component 106 receives the prompt and user context and determines an answer to the natural language question. The LLM component 106 may utilize APIs to receive the prompt and the user context from the user context component 105 where the prompt may be formatted in a certain manner to be sent or communicated over an API.")
receiving a second response to the second prompt from the text generation model;
(paragraph 61 "At (4), the aggregator component 104 sends some of the passages retrieved and corresponding QA pairs to the user context component 105. At (5), the user context component 105 determines user context associated with a user of the customer computing device 122 and forwards the QA pairs, passages retrieved, and user context to the LLM component 106. It should be noted that (5) is optional and the aggregator component 104 may directly send the passages retrieved and the QA pairs to the LLM component 106 without user context. At (6), the LLM component 106 receives the prompt and user context and determines an answer to the natural language question. The LLM component 106 may utilize APIs to receive the prompt and the user context from the user context component 105 where the prompt may be formatted in a certain manner to be sent or communicated over an API.")
presenting the second response and the second text chunks;
(paragraph 68 " At (11), the watermarking component 110 adds patterns to the answer to make the answer proprietary to the natural language question answering service 102 and verifiable against subsequent copying. In other words, the 110 may embed patterns into the generated text of the answer from the LLM component 106 that is invisible to humans but algorithmically detectable from a short span of tokens (e.g., group of words or where a token equals a single word or group of characters). Tokens may be selected prior to watermarking and the tokens may be promoted during the watermarking of the generated answer. At (11), the watermarking component 110 sends the watermarked answer and retrieved passages to the customer computing devices 122 such that a user of the customer computing devices 122 may view the answer and retrieved passages.")
Khosla in view of Beamon, further Beamon teaches
receiving a second rating of one of the second text chunks from the second user; and
(Paragraph 30 "Moreover, GUI elements are provided for selection of evidence passages to be removed from the set of evidence passages associated with a candidate answer or increasing a relevance score associated with the evidence passage. That is, GUI elements are provided that allow a user to override the determined relevance of the evidence passage to the corresponding candidate answer by either eliminating the evidence passage altogether or modifying its relevance score based on the user's subjective determination of the relevance of the evidence passage, either for or against the candidate answer being a correct answer for the input question. Changes made to the set of evidence passages stored for the candidate answer may be automatically used to update the confidence score associated with the candidate answer and modify the ranked listing of candidate answers in the GUI."
paragraph 32 "The changes made, via the GUI, to the ranked listing of candidate answers and sets of evidence passages associated with the candidate answers, may be stored for later retrieval and use. Such information may be used to assist in training of the QA system, such as by adjusting scoring parameters, adjusting weights associated with documents or sources of content in the corpus of content, and other operational parameters of the QA system. During runtime, the stored set of candidate answers and corresponding evidence passages may be used to assist in responding to the same or similar answers being submitted by the same or other users at a later time. For example, when an input question is received and parsed to generate one or more queries, the stored information may be searched to find entries having similar queries to that of the input question so that corresponding candidate answers and supporting evidence passages may be quickly retrieved and used to generate an answer or set of candidate answers for the input question. Moreover, the stored set of candidate answers and corresponding evidence passages may be used by an analyst to compare to subsequent executions on a same or similar question to evaluate if and how the corpus of information has been modified since the stored set of candidate answers and corresponding evidence passages was generated. There is a plethora of potential uses of the results generated by the operation of the illustrative embodiments, any of which are intended to be within the spirit and scope of the illustrative embodiments."
Paragraph 86 "Moreover, the GUI may include GUI elements for invoking logic and functionality of the GUI for removing evidence passages from the listing of associated evidence passages for the various candidate answers and/or modifying a relevance score associated with the evidence passage. In this way, the user essentially supersedes the evaluation made by the QA system pipeline 500 and instead imposes the user's subjective determination as to the relevance of an evidence passage by either eliminating it altogether or increasing/reducing the relevance score associated with the evidence passage to indicate the user's own subjective evaluation of the evidence passage's relevance to the candidate answer being the correct answer for the input question.")
updating the second score information associated with the one of the second text chunks based on the second rating.
(Paragraph 88 " Should the user eliminate the evidence passage or modify the evidence passage's relevance score in some manner, the QA system pipeline 500 may automatically adjust the relevance scores, confidence scores, and ranked listing of candidate answers based on the change to the evidence passage. In this way, the QA system pipeline 500 may dynamically adjust its output based on user collaboration with the QA system to provide the user's subject determination of the relevance, reliability, and correctness of the evidence passages and/or the candidate answers themselves."
Paragraph 100 "Moreover, the evidence passage engine 636 generates the evidence passage portion of the GUI with GUI elements for removing evidence passages or modifying the corresponding relevance scores associated with the evidence passages based on user input. In response to a user providing a user input via the user interface 642 that selects a GUI element for removing an evidence passage, the corresponding evidence passage is eliminated from the GUI output and the change is submitted to the QA system 610 for dynamic re-evaluation of the candidate answers. Similarly, in response to the user providing a user input for modifying the relevance score for the evidence passage, the change is communicated to the QA system 610 which may dynamically re-evaluate the candidate answers based on the received change."
Paragraph 101 "The dynamic update engine 640 comprises logic for coordinating the user modifications and selections of GUI elements received via the user interface 642. This may involve coordinating the updating of the evidence passage portion and candidate answer portions of the GUI as well as the submission of the modifications to the QA system 610 for re-evaluation of the candidate answers and/or evidence passages associated with the candidate answers. The resulting candidate answers and associated evidence passages generated via the operation of the QA system 610 and the user collaboration provided via the GUI engine 630 may be stored in the candidate answer evidence passage storage system 650 for later retrieval and use.")
See claim one for rationale.
Khosla in view of Beamon, do not explicitly teach generating a second prompt based on the second scores, the second prompt including the second query and the second text portions, and the second prompt comprising instructions to associate the second text portions with an importance based on their respective second scores;
However, Pan teaches generating a second prompt based on the second scores, the second prompt including the second query and the second text portions, and the second prompt comprising instructions to associate the second text portions with an importance based on their respective second scores;
(see figure 12 for the retrieval credibility setting, see fig 7 shows the evaluation
page 3 section multi granularity credibility annotation "…Then, the retriever assesses the match between each retrieval unit and the query, assigning a relevance score, and classifies documents into three levels: high, medium, and low, using either equal count or equal interval methods…"
Page 3 section 2 credibility aware generation "…Then, these documents Dx with their credibility C are synthesized with the user input x as augmented input. LM generates responses y based on this augmented input, formally represented as y = LM x,{[ci,di]}|Dx|i=1…")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Khosla to incorporate the teachings of Beamon to provide a “generating a second prompt based on the second scores, the second prompt including the second query and the second text portions, and the second prompt comprising instructions to associate the second text portions with an importance based on their respective second scores;” Doing so would Make the accuracy and comprehensiveness of the response improve, as recognized by Beamon. (page 8 section 5.3.2 analysis of discarding low credibility documents).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALI M HASSAN whose telephone number is (571)272-5331. The examiner can normally be reached Monday - Friday 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras Shah can be reached at (571)270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALI M HASSAN/ Examiner, Art Unit 2653
/Paras D Shah/ Supervisory Patent Examiner, Art Unit 2653
07/14/2026
Prosecution Insights