Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/09/2026 has been entered.
Detailed Action
Applicant amended claims 1 and 13, previously canceled claims 11 and 14 and presented claims 1-10, 12-13 and 15-20 for reconsideration on 04/09/2026.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1 and 13 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 1 recites “in response to the feedback meeting a storage criteria, providing, via the computing device, to the programmatic search engine, the textual inquiry and the LLM response for storage and use in generating responses to later textual inquiries”.
Spec. ¶¶ 84-85 discloses: “The computing device 102 may generally compare the feedback received from the client device 104 to a storage criteria that may generally conditions under which an LLM response is approved for storage at the memory 110. For example, such storage criteria may comprise "when the feedback is "YES", an LLM response is approved" and/or such storage criteria may comprise "when the feedback corresponds to text indicating approval of the LLM response, an LLM response is approved", and the like… When the feedback meets the storage criteria, the computing device 102 may provide the programmatic search engine 106 with the textual inquiry, originally received from the client device 104, and the LLM response provided to the client device 104 as an answer to the textual inquiry for storage at the memory 110, for example as a respective new pair of a textual inquiry 112 and a corresponding textual response 114, for use in generating responses to later textual inquiries from client devices”.
Accordingly, the stored LLM response is a response approved by the user’s feedback. Claim further recites updating the stored LLM response which has already been approved by the user with a later LLM generated response and marking the stored LLM response as “deteriorated” based on semantic or textual comparison with a later LLM generated response. It is not clear how an approved stored response is different semantically or textually from a later LLM generated response such that the approved stored response is considered as “deteriorated”.
Claim 13 is rejected for same reason sated above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 12-13, 15 and 20 are rejected under 35 U.S.C. 103(a) as being unpatentable over SHIRRELL et al., Pub. No.: US 2025/0284985 A1 (SHIRRELL), in view of Avusingi et al., Pub. No.: US 2025/0252270 A1 (Avusingi).
Claim 1. SHIRRELL teaches:
A method comprising:
receiving, at a computing device, from a client device, a textual inquiry; (SHIRRELL, ¶¶ 15-17, a received query/prompt is provided to LLM for generating a response or a response is selected from a previously stored query/response: “a received prompt may be compared to a set of stored prompts. If a certain degree of similarity between the received prompt and a stored prompt is detected, the system may output a response associated with the stored prompt. For example, a first client device may be associated with a database administrator and the client device may send a prompt requesting the number of entries in the database. A system…may receive the prompt, use an LLM to interpret it and generate a response. This prompt and response may then be stored. A second client device may send a second prompt asking for the same information. In this example, the system may first consult a store of saved prompts, and determine that the second prompt is the same as the first. In this example, the system may respond with the same response, previously generated by the LLM… if none of the stored prompts match the received prompt or lack a required degree of similarity, the LLM may be leveraged to generate a new response to the prompt”; ¶¶ 23-24)
providing, via the computing device, the textual inquiry to a programmatic search engine that compares the textual inquiry to previously stored textual inquiries, the previously stored textual inquiries stored in association with respective related textual responses; (SHIRRELL, ¶¶ 15-16, see above, a received query/prompt is compared with stored query/prompt and answers)
receiving, via the computing device, from the programmatic search engine, one or more of the respective related textual responses with respective comparison scores between the textual inquiry and associated stored textual inquiries; (SHIRRELL, ¶¶ 15-17, see above, based on the similarity of a received query with stored query, the system provides a corresponding previously stored response; ¶ 26, “prompt engine 106 may select a stored prompt with the highest similarity to the received prompt…prompt engine 106 may set a similarity threshold…prompt engine 106 may require that the stored prompt be at least 90% similar to the received prompt. If none of the stored prompts at data store 214 are above the threshold, prompt engine 106 may use LLM 212 to generate a new response”)
based on the respective comparison scores, providing, via the computing device, at least the textual inquiry to a large language model (LLM) engine; (SHIRRELL, ¶¶ 15-17, see above, “if none of the stored prompts match the received prompt or lack a required degree of similarity, the LLM may be leveraged to generate a new response to the prompt”; ¶ 26, “prompt engine 106 may require that the stored prompt be at least 90% similar to the received prompt. If none of the stored prompts at data store 214 are above the threshold, prompt engine 106 may use LLM 212 to generate a new response”)
receiving, via the computing device, from the LLM engine, an LLM response to the textual inquiry; (SHIRRELL, ¶¶ 15-17, see above, “If none of the stored prompts at data store 214 are above the threshold, prompt engine 106 may use LLM 212 to generate a new response”)
providing, via the computing device, to the client device, the LLM response; (SHIRRELL, ¶ 49, “prompt engine 106 transmits the response. The response may be transmitted to client device 102 via network 104…prompt engine 106 may further save the received prompt and generated response at data store 214. This may be useful to generate future responses as well as to train LLM 212”)
receiving, via the computing device, from the client device, feedback on the LLM response; and (SHIRRELL, ¶ 30, a label is a user provided feedback: “The label may be a binary value such as 0 if the response was incorrect, and 1 if it was correct. The label may be a continuous value such as 10% correct, 50% correct, or 90% correct…the label may be received from client device 102…the label may be received from service 108”)
in response to the feedback meets a storage criteria, providing, via the computing device, to the programmatic search engine, the textual inquiry and the LLM response for storage and use in generating responses to later textual inquiries; and (SHIRRELL, ¶¶ 30-31, a labeled response is stored and used for generating later response: “if the label indicates the actual response successfully addressed the prompt, LLM may be optimized to generate a response similar to the actual response. If the label indicates the actual response failed to address the prompt, LLM 212 may be trained to generate a different response”)
SHIRRELL discloses updating the LLM response after the textual inquiry and the LLM response are stored as in ¶¶ 63-73.
SHIRRELL did not specifically disclose but Avusingi discloses
again providing, via the computing device, the textual inquiry to the LLM engine, and receiving an updated LLM response; comparing, via the computing device, the LLM response and the updated LLM response to determine a respective difference comparison score therebetween, wherein the respective difference comparison score comprises a numerical measure of one or more of a semantic difference and a textual difference between the LLM response and the updated LLM response, and wherein as the respective difference comparison score increases the one or more of a semantic difference and the textual difference between the LLM response and the updated LLM response increases;, and in response to the respective difference comparison score being above a first threshold: marking, via the computing device, the LLM response as deteriorated; and thereafter periodically generating the updated LLM response and again determining the respective difference comparison score therebetween; and in response to the respective difference comparison score being above a second threshold, larger than the first threshold, controlling, via the computing device, the programmatic search engine to delete at least the LLM response. (Avusingi, wherein answers to a given question are generated iteratively and scored “in an attempt to obtain a better and higher quality answer” and wherein the better and higher quality answer is an answer with a high score while an answer with lower score is not selected, e.g., marked as deteriorated and deleted; ¶ 7, “disclosed techniques include a mechanism by which to improve the LLM based on user feedback. For instance, a computing system may obtain user feedback regarding the quality and/or helpfulness of an answer provided by the LLM. Such user feedback may be utilized to update the LLM, enabling the LLM to provide higher quality answers in the future…the computing system may automatically resubmit the original inquiry to the LLM and obtain a new answer from the LLM for the original inquiry, in an attempt to obtain a better and higher quality answer. In such examples, the new answer may once again be evaluated for quality, and if the new answer satisfies the quality threshold, then the new answer may be annotated and provided as output for display to a user with annotations indicating a measure of quality based on the evaluation by the computing system”)
SHIRRELL ¶¶ 63-73, discloses generating new responses by running the same query periodically. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing again providing, via the computing device, the textual inquiry to the LLM engine, and receiving an updated LLM response; comparing, via the computing device, the LLM response and the updated LLM response to determine a respective difference comparison score therebetween, wherein the respective difference comparison score comprises a numerical measure of one or more of a semantic difference and a textual difference between the LLM response and the updated LLM response, and wherein as the respective difference comparison score increases the one or more of a semantic difference and the textual difference between the LLM response and the updated LLM response increases;, and in response to the respective difference comparison score being above a first threshold: marking, via the computing device, the LLM response as deteriorated; and thereafter periodically generating the updated LLM response and again determining the respective difference comparison score therebetween; and in response to the respective difference comparison score being above a second threshold, larger than the first threshold, controlling, via the computing device, the programmatic search engine to delete at least the LLM response because doing so would explicitly provide for scoring LLM generated response, comparing scores assigned to each generated response and selecting a response with higher score as a valid response for achieving the same predictable result of updating an LLM generated response.
Claim 13. SHIRRELL teaches:
A computing device comprising: a communication interface; a controller; and a computer-readable storage medium having stored thereon program instructions that, when executed by the controller, cause the controller to perform a set of operations comprising:
receiving, from a client device, a textual inquiry; (SHIRRELL, ¶¶ 15-17, a received query/prompt is provided to LLM for generating a response or a response is selected from a previously stored query/response: “a received prompt may be compared to a set of stored prompts. If a certain degree of similarity between the received prompt and a stored prompt is detected, the system may output a response associated with the stored prompt. For example, a first client device may be associated with a database administrator and the client device may send a prompt requesting the number of entries in the database. A system…may receive the prompt, use an LLM to interpret it and generate a response. This prompt and response may then be stored. A second client device may send a second prompt asking for the same information. In this example, the system may first consult a store of saved prompts, and determine that the second prompt is the same as the first. In this example, the system may respond with the same response, previously generated by the LLM… if none of the stored prompts match the received prompt or lack a required degree of similarity, the LLM may be leveraged to generate a new response to the prompt”; ¶¶ 23-24)
providing the textual inquiry to a programmatic search engine that compares the textual inquiry to previously stored textual inquiries, the previously stored textual inquiries stored in association with respective related textual responses; (SHIRRELL, ¶¶ 15-16, see above, a received query/prompt is compared with stored query/prompt and answers)
receiving, from the programmatic search engine, one or more of the respective related textual responses with respective comparison scores between the textual inquiry and associated stored textual inquiries; (SHIRRELL, ¶¶ 15-17, see above, based on the similarity of a received query with stored query, the system provides a corresponding previously stored response; ¶ 26, “prompt engine 106 may select a stored prompt with the highest similarity to the received prompt…prompt engine 106 may set a similarity threshold…prompt engine 106 may require that the stored prompt be at least 90% similar to the received prompt. If none of the stored prompts at data store 214 are above the threshold, prompt engine 106 may use LLM 212 to generate a new response”)
based on the respective comparison scores, providing, via the computing device, at least the textual inquiry to a large language model (LLM) engine; (SHIRRELL, ¶¶ 15-17, see above, “if none of the stored prompts match the received prompt or lack a required degree of similarity, the LLM may be leveraged to generate a new response to the prompt”; ¶ 26, “prompt engine 106 may require that the stored prompt be at least 90% similar to the received prompt. If none of the stored prompts at data store 214 are above the threshold, prompt engine 106 may use LLM 212 to generate a new response”)
receiving, from the LLM engine, an LLM response to the textual inquiry; (SHIRRELL, ¶¶ 15-17, see above, “If none of the stored prompts at data store 214 are above the threshold, prompt engine 106 may use LLM 212 to generate a new response”)
providing, to the client device, the LLM response; (SHIRRELL, ¶ 49, “prompt engine 106 transmits the response. The response may be transmitted to client device 102 via network 104…prompt engine 106 may further save the received prompt and generated response at data store 214. This may be useful to generate future responses as well as to train LLM 212”)
receiving, from the client device, feedback on the LLM response; and (SHIRRELL, ¶ 30, label is a user provided feedback: “The label may be a binary value such as 0 if the response was incorrect, and 1 if it was correct. The label may be a continuous value such as 10% correct, 50% correct, or 90% correct…the label may be received from client device 102…the label may be received from service 108”)
in response to the feedback meeting a storage criteria, providing, via the computing device, to the programmatic search engine, the textual inquiry and the LLM response for storage and use in generating responses to later textual inquiries; and (SHIRRELL, ¶¶ 30-31, a labeled response is stored and used for generating later response: “if the label indicates the actual response successfully addressed the prompt, LLM may be optimized to generate a response similar to the actual response. If the label indicates the actual response failed to address the prompt, LLM 212 may be trained to generate a different response”)
SHIRRELL discloses updating the LLM response after the textual inquiry and the LLM response are stored as in ¶¶ 63-73.
SHIRRELL did not specifically disclose but Avusingi discloses again providing the textual inquiry to the LLM engine, and receiving an updated LLM response; comparing the LLM response and the updated LLM response to determine a respective difference comparison score therebetween, wherein the respective difference comparison score represents a difference between the LLM response and the updated LLM response, and wherein as the respective difference comparison score increases the difference between the LLM response and the updated LLM response increases; in response to the respective difference comparison score being above a first threshold: marking the LLM response as deteriorated; and thereafter periodically generating the updated LLM response and again determining the respective difference comparison score therebetween; and in response to the respective difference comparison score being above a second threshold, larger than the first threshold, controlling the programmatic search engine to delete at least the LLM response. (Avusingi, wherein answers to a given question are generated iteratively and scored “in an attempt to obtain a better and higher quality answer” and wherein the better and higher quality answer is an answer with a high score while an answer with lower score is not selected, e.g., marked as deteriorated and deleted: ¶ 7, “disclosed techniques include a mechanism by which to improve the LLM based on user feedback. For instance, a computing system may obtain user feedback regarding the quality and/or helpfulness of an answer provided by the LLM. Such user feedback may be utilized to update the LLM, enabling the LLM to provide higher quality answers in the future…the computing system may automatically resubmit the original inquiry to the LLM and obtain a new answer from the LLM for the original inquiry, in an attempt to obtain a better and higher quality answer. In such examples, the new answer may once again be evaluated for quality, and if the new answer satisfies the quality threshold, then the new answer may be annotated and provided as output for display to a user with annotations indicating a measure of quality based on the evaluation by the computing system”)
SHIRRELL ¶¶ 63-73, discloses generating new responses by running the same query periodically. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing again providing, via the computing device, the textual inquiry to the LLM engine, and receiving an updated LLM response; comparing, via the computing device, the LLM response and the updated LLM response to determine a respective difference comparison score therebetween, wherein the respective difference comparison score comprises a numerical measure of one or more of a semantic difference and a textual difference between the LLM response and the updated LLM response, and wherein as the respective difference comparison score increases the one or more of a semantic difference and the textual difference between the LLM response and the updated LLM response increases;, and in response to the respective difference comparison score being above a first threshold: marking, via the computing device, the LLM response as deteriorated; and thereafter periodically generating the updated LLM response and again determining the respective difference comparison score therebetween; and in response to the respective difference comparison score being above a second threshold, larger than the first threshold, controlling, via the computing device, the programmatic search engine to delete at least the LLM response because doing so would explicitly provide for scoring LLM generated response, comparing scores assigned to each generated response and selecting a response with higher score as a valid response for achieving the same predictable result of updating an LLM generated response.
SHIRRELL as modified by Avusingi teaches:
Claim 3. The method of claim 1, further comprising:
comparing the respective comparison scores to a threshold score; and (SHIRRELL, ¶¶ 15-17, 26, “certain degree of similarity between the received prompt and a stored prompt is detected…prompt engine 106 may set a similarity threshold”);
in response to none of the respective comparison scores meet the threshold score, providing the textual inquiry to the LLM engine comprises: providing the textual inquiry to the LLM engine without any of the respective related textual responses. (Note that according to “based on the respective comparison scores, providing, via the computing device, at least the textual inquiry to a large language model (LLM) engine” as recited in claim 1, no “respective related textual responses” is provided to the LLM engine. SHIRRELL, ¶¶ 15-17, “if none of the stored prompts match the received prompt or lack a required degree of similarity, the LLM may be leveraged to generate a new response to the prompt”; ¶ 26, “prompt engine 106 may require that the stored prompt be at least 90% similar to the received prompt. If none of the stored prompts at data store 214 are above the threshold, prompt engine 106 may use LLM 212 to generate a new response”)
Claim 15 is rejected under the same rationale as provided above.
Claim 12. The method of claim 1, wherein a memory, accessible by the programmatic search engine, that stores the previously stored textual inquiries in association with respective related textual responses is initially empty, and the method further comprises:
populating the memory with one or more textual inquiries from one or more client devices and associated LLM responses from the LLM engine. (SHIRRELL, ¶ 24, “Data store 214 may be a memory device used to store data. Data store 214 may include various pieces of information such as previously received prompts and corresponding generated responses”)
Claim 20. The method of claim 1, further comprising, after the textual inquiry and the LLM response are stored: (SHIRRELL, ¶¶ 63-73 wherein updating the LLM response after the textual inquiry and the LLM response are stored)
again providing the textual inquiry to the LLM engine, and receiving an updated LLM response; (Avusingi, ¶ 7, wherein answers to a given question are generated iteratively “in an attempt to obtain a better and higher quality answer”)
comparing the LLM response and the updated LLM response to determine a respective difference comparison score therebetween; (Avusingi, ¶ 7, each generated response is assigned a score, a response with a stratifying score is selected as updated response)
in response to the respective difference comparison score is being a first threshold: marking the LLM response as deteriorated; and thereafter periodically generating the updated LLM response and again determining the respective difference comparison score therebetween; or when the respective difference comparison score is above a second threshold, larger than the first threshold, controlling the programmatic search engine to delete at least the LLM response. (Avusingi, ¶ 7, each generated response is assigned a score, a response with a stratifying score is selected as updated response, a response with unsatisfying score is not selected)
Claims 2, 4, 5, 16 and 17 are rejected under 35 U.S.C. 103(a) as being unpatentable over SHIRRELL and Avusingi as applied to claims 1 and 13 above, in view of DU et al., Pub. No.: US 2025/0259005 A1 (DU).
Claim 2. SHIRRELL as modified taught the method of claim 1; SHIRRELL as modified did not specifically disclose but DU discloses:
after receiving the LLM response: syntactically comparing the textual inquiry with previously stored textual inquiries to identify a group of one or more inquiries that are syntactically related to the textual inquiry; (DU, ¶¶ 89- 91, wherein “The previous queries 202 can comprise one or more output answers 132 responsive to one or more previous input question(s) 102…the previous queries 202 does not contain an answer responsive to input question 102 but instead contains an answer that is partially responsive to the input question 102” suggest that previous queries are syntactically/partially related to the current query by containing an answer responsive to the input question and therefore after receiving an LLM response for the input question, the response includes the partial response consistent with “an answer that is partially responsive to the input question 102”)
providing a pair of the textual inquiry and the LLM response with the group of the one or more inquiries and respective responses to the LLM engine to cause the LLM engine to classify the respective response in the group as being contradictory or not contradictory with the LLM response; (DU, ¶ 91, wherein, “the previous queries 202 does not contain an answer responsive to input question 102 but instead contains an answer that is partially responsive to the input question 102…The semantic search can return one or more previous queries that are most similar to the input question 102 as well as the output answers corresponding thereto. The returned previous queries may be input to the query LLM 106 as semantic context for generating the query response 116” suggests that the answers corresponding to the most previous similar queries are selected/classified as not contradictory because answers are “partially responsive to the input question 102”)
determining a contradiction rate from classifications of the respective responses; and (DU, ¶ 91, wherein “semantic search previous queries 206 model may determine…that the previous queries 202 does not contain an answer responsive to input question 102 but instead contains an answer that is partially responsive to the input question 102… while the subset of data corresponding to good A and day C can be determined using the previous query in (1), the subset of data corresponding to good B and day C is not known” suggests that a contradiction rate is zero)
in response to the contradiction rate being below a threshold contradiction rate, providing, to the programmatic search engine, the textual inquiry and the LLM response for storage and use in generating responses to later textual inquiries. (DU, ¶ 91, wherein “semantic search previous queries 206 model may determine…that the previous queries 202 does not contain an answer responsive to input question 102 but instead contains an answer that is partially responsive to the input question 102… while the subset of data corresponding to good A and day C can be determined using the previous query in (1), the subset of data corresponding to good B and day C is not known” and “The previous queries and the corresponding output answers can provide a "warm start" for the query LLM 106 in generating an appropriate query response 116 for the input question 102 that is more likely to pass the evaluations of the query judge LLMs 109” suggests that a contradiction rate is zero and “an appropriate query response” is generated for storage)
SHIRRELL ¶¶ 16, 43, 59 discloses identifying previously received query and generated corresponding results and using them for generating respond for a current query: “Method 500 may be used by prompt engine 106 to identify previously received, stored prompts. The responses associated with the previously received prompt may then be used to respond to the current prompt. This process is beneficial to ensure that responses to similar or related prompts are consistent”. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing after receiving the LLM response: syntactically comparing the textual inquiry with previously stored textual inquiries to identify a group of one or more inquiries that are syntactically related to the textual inquiry; providing a pair of the textual inquiry and the LLM response with the group of the one or more inquiries and respective responses to the LLM engine to cause the LLM engine to classify the respective response in the group as being contradictory or not contradictory with the LLM response; determining a contradiction rate from classifications of the respective responses; and when the contradiction rate is below a threshold contradiction rate, providing, to the programmatic search engine, the textual inquiry and the LLM response for storage and use in generating responses to later textual inquiries because doing so would further provide for processing an input query that is syntactically/partially related to a previous query and including the partial answer corresponding to the previous query for generating a consistent answer corresponding to input query.
Claim 4. The method of claim 1, further comprising:
comparing the respective comparison scores to a threshold score; and (SHIRRELL, ¶¶ 15-17, 26, “certain degree of similarity between the received prompt and a stored prompt is detected…prompt engine 106 may set a similarity threshold”);
in response to n two or more of the respective comparison scores meet the threshold score, providing the textual inquiry to the LLM engine comprises: providing the textual inquiry to the LLM engine with the respective related textual responses associated with the two or more of the respective comparison scores meet the threshold score. (SHIRRELL, ¶¶ 15-17, “certain degree of similarity between the received prompt and a stored prompt is detected”; DU, ¶¶ 88-91, wherein “an input question 102 can be: “what is the price of good A and good B on day C?”…while the subset of data corresponding to good A and day C can be determined using the previous query in (1), the subset of data corresponding to good B and day C is not known. Similarly, while the subset of data corresponding to good B can be determined using the previous query in (2), the subset of data corresponding to good A and day C (and good B on day C) is not known. Accordingly, The semantic search can return one or more previous queries that are most similar to the input question 102 as well as the output answers corresponding thereto. The returned previous queries may be input to the query LLM 106 as semantic context for generating the query response 116” suggests that a scoring mechanism is used for identifying similarity between a current question and previous questions and further based on the satisfied similarity, previous questions/answers and current question are provided to an LLM)
Claim 16 is rejected under the same rationale as provided above.
Claim 5. The method of claim 1, further comprising:
comparing the respective comparison scores to a threshold score; and (SHIRRELL, ¶¶ 15-17, 26, “certain degree of similarity between the received prompt and a stored prompt is detected…prompt engine 106 may set a similarity threshold”);
in repone to two or more respective comparison scores meet the threshold score, providing the textual inquiry to the LLM engine comprises: providing the textual inquiry to the LLM engine with the respective related textual responses associated with the two or more respective comparison scores meet the threshold score such that the LLM response comprises an LLM generated combination of the respective related textual responses associated with the two or more respective comparison scores meet the threshold score. (DU, ¶¶ 88-91, wherein “the semantic search can be a cosine similarity search or a KNN search. The previous queries 202 can comprise one or more output answers 132 responsive to one or more previous input question(s) 102…an input question 102 can be: “what is the price of good A and good B on day C?”…while the subset of data corresponding to good A and day C can be determined using the previous query in (1), the subset of data corresponding to good B and day C is not known. Similarly, while the subset of data corresponding to good B can be determined using the previous query in (2), the subset of data corresponding to good A and day C (and good B on day C) is not known. Accordingly, The semantic search can return one or more previous queries that are most similar to the input question 102 as well as the output answers corresponding thereto. The returned previous queries may be input to the query LLM 106 as semantic context for generating the query response 116” suggests that a scoring mechanism is used for identifying similarity between a current question and previous questions and further based on the satisfied similarity, previous questions/answers and current question are provided to an LLM for generating answer)
Claim 17 is rejected under the same rationale as provided above.
Claims 6-9 and 18 are rejected under 35 U.S.C. 103(a) as being unpatentable over SHIRRELL and Avusingi as applied to claims 1 and 13 above, in view of an IDS provided reference: NAHAMOO et al., WO 2021/263138, (NAHAMOO).
Claim 6. SHIRRELL as modified taught the method of claim 1; SHIRRELL as modified did not specifically disclose but NAHAMOO discloses comparing a pair of the textual inquiry and the LLM response to pairs of the previously stored textual inquiries and the respective related textual responses to determine respective pair difference comparison scores between the pair and the pairs, and wherein providing, to the programmatic search engine, the textual inquiry and the LLM response for storage is further based on one or more of the respective pair difference comparison scores. (NAHAMOO, wherein an initial response is unsatisfactory/different compare to a current response and is replaced with the current satisfactory response: ¶ 52, “replacement answer data to replace an initial answer data determined at the local device in response to a particular question, with the initial answer being determined by a user to be an unsatisfactory response to the particular question, and storing in the question-answer cache a data item representative of a pairing of the particular question and the replacement answer, ¶¶ 70, 256, 274, 277)
SHIRRELL, ¶ 63 discloses updating “stored responses and subsequently retain an LLM using the updated stored response” because “data communicated in environment 100 may rapidly change… Therefore, it's desirable to generate new responses after a certain amount of time has passed in order to capture these updates…method 600 may be used to detect a predefined amount of time has passed between receiving the current prompt and when the stored prompt was received, and therefore that a new response should be generated”. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing comparing a pair of the textual inquiry and the LLM response to pairs of the previously stored textual inquiries and the respective related textual responses to determine respective pair difference comparison scores between the pair and the pairs, and wherein providing, to the programmatic search engine, the textual inquiry and the LLM response for storage is further based on one or more of the respective pair difference comparison scores because doing so would further provide an alternative for updating an initial answer)
Claim 18 is rejected under the same rationale as provided above.
Claim 7. The method of claim 6, wherein providing, to the programmatic search engine, the textual inquiry and the LLM response for storage occurs in response to a smallest respective pair difference comparison score is less than a given pair storage threshold score. (NAHAMOO, ¶¶ 52, 70, 256, 274, 277, as noted above with respect to claim 6, an unsatisfactory response is replaced with a satisfactory response, any desired threshold score can be applied as needed)
Claim 8. The method of claim 6, wherein providing, to the programmatic search engine, the textual inquiry and the LLM response for storage occurs in response to an average of the respective pair difference comparison scores being less than a given pair storage threshold score. (NAHAMOO, ¶¶ 52, 70, 256, 274, 277, as noted above with respect to claim 6, an unsatisfactory response is replaced with a satisfactory response, any desired threshold score can be applied as needed)
Claim 9. The method of claim 6, wherein providing, to the programmatic search engine, the textual inquiry and the LLM response for storage occurs in response to a given number of the respective pair difference comparison scores being less than a given pair storage threshold score. (NAHAMOO, ¶¶ 52, 70, 256, 274, 277, as noted above with respect to claim 6, an unsatisfactory response is replaced with a satisfactory response, any desired threshold score can be applied as needed)
Claims 10 and 19 are rejected under 35 U.S.C. 103(a) as being unpatentable over SHIRRELL and Avusingi as applied to claims 1 and 13 above, in view of PerezLeon et al., Pub. No.: US 2024/0362278 A1 (PerezLeon).
Claim 10. SHIRRELL as modified taught the method of claim 1; SHIRRELL as modified did not specifically disclose but PerezLeon discloses prior to providing the textual inquiry and the LLM response for storage: collecting, for a given time period, one or more associated textual inquiries and respective LLM responses, including the textual inquiry and the LLM response, that meet the storage criteria; (PerezLeon, ¶¶ 34- 45, wherein generated results for questions made during a user session/a given time period and successful answers are stored along with user’s rating to be used as hint in future sessions: ¶¶ 34- 45, 95, “A user 102 may initiate a session with the system 100 by opening the web application 104 in one embodiment…The web application 104 may accept a natural language query 130 from the user 102 in the form of typed text or speech electronically converted to text…information when needed…The query responses 136 may be presented to the user 102 by the web application 104. The user 102 may present additional natural language queries 130 to the system 100, and the process may repeat…. The web application 104 may present various options to the user…which may elicit a response from the user 102 indicating the session may be terminated…as part of ending the session…the web application 104 may instruct the user 102 to input a rating 158 indicating how successfully the query responses 136 answered their questions, i.e., indicating their evaluation of the appropriateness and informativeness of the query response 136 they received from the system 100…the session termination signal 160 may instruct the action engine 112 to perform follow-up actions such as instructing storage of data in the context database 200, as well as determining whether to push any of the new historical data 214 through anonymization 210 to create anonymized historical data 220, which may then be used in the hint database 208…the system 100 may include a hint database 208 storing query hints 218, which may be past phrasings for natural language queries 130 that elicited query responses 136 for which the user 102 and/or other users have provided high ratings 158”)
determining one or more of: a number of the one or more associated textual inquiries; a variance between the respective LLM responses; and a total comparison score representing a comparison between the one or more associated textual inquiries and the respective LLM responses, and the associated stored textual inquiries and the respective related textual responses; (PerezLeon, ¶¶ 34- 45, 95, see above, “The user 102 may present additional natural language queries 130 to the system 100”)
comparing one or more of the number, the variance and the total comparison score to respective thresholds; and (PerezLeon, ¶¶ 34- 45, 95, wherein “high ratings” is an indication of comparison: “The second database 204 may include an index that may use the rating to rank historical data by user satisfaction, and to identify successful and unsuccessful query responses from the user's perspective. Natural language queries associated with high ratings, indicating successful query responses, may be stored in a hint database 208 as successful past phrasings 248, ranked according to their associated ratings 246”)
in response to the comparing meets a further storage criteria, providing, to the programmatic search engine, the one or more associated textual inquiries and the respective LLM responses, including the textual inquiry and the LLM response, for storage. (PerezLeon, ¶¶ 34- 45, 95, see above, “the web application 104 may instruct the user 102 to input a rating 158 indicating how successfully the query responses 136 answered their questions, i.e., indicating their evaluation of the appropriateness and informativeness of the query response 136 they received from the system 100…The second database 204 may include an index that may use the rating to rank historical data by user satisfaction, and to identify successful and unsuccessful query responses from the user's perspective. Natural language queries associated with high ratings, indicating successful query responses, may be stored in a hint database 208 as successful past phrasings 248, ranked according to their associated ratings 246”)
SHIRRELL ¶ 14 discloses that a chat bot and LLM can be utilized for questions/answers communication. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing prior to providing the textual inquiry and the LLM response for storage: collecting, for a given time period, one or more associated textual inquiries and respective LLM responses, including the textual inquiry and the LLM response, that meet the storage criteria; determining one or more of: a number of the one or more associated textual inquiries; a variance between the respective LLM responses; and a total comparison score representing a comparison between the one or more associated textual inquiries and the respective LLM responses, and the associated stored textual inquiries and the respective related textual responses; comparing one or more of the number, the variance and the total comparison score to respective thresholds; and when the comparing meets a further storage criteria, providing, to the programmatic search engine, the one or more associated textual inquiries and the respective LLM responses, including the textual inquiry and the LLM response, for storage because doing so would further provide for an explicit disclosure of processing questions and answers related to a user’s session for achieving the same predictable result.
Claim 19 is rejected under the same rationale as provided above.
Response to Amendment and Arguments
With respect to 112(b) rejections, Applicant requested withdrawal of the rejections because of the claims’ amendments. Applicant further noted that “the independent claims do not require the updated response to be approved via user's feedback. Rather, responses by LLM engines are known to drift over time as training of an LLM engine changes and/or is updated, which can occur on an ongoing basis”. Remarks, 11-12.
In response: 112(b) rejections are maintained. As applicant mentioned correctly and as it is well known in the art that an LLM might generate incorrect response and the incorrect response might be replaced with a correct response that is generated later.
However, the 112(b) rejection is not about weather an updated/correct response is approved by the user or not. It is about an LLM response which is correct and approved by the user as correct response. This correct stored response is marked as “deteriorated” when it is compared semantically or textually by a later LLM generated response. It is not clear how a response that is confirmed by the user as correct is determined to be incorrect when it is compared semantically or textually by a later generated response.
With respect to 103 rejections, Applicant argues “the cited passages of Avusingi merely disclose an iterative answer-generation and evaluation process in which multiple candidate answers to a given question are generated and each candidate answer is assigned a confidence score that is generated independently of any previous answers… There is no disclosure of comparing previously generated responses to later generated updated responses as presently claimed… Thus, Avusingi evaluates individual candidate responses based on a quality metric associated with each response… Avusingi merely evaluates the quality of independently generated candidate answers and selects one having an acceptable score, which is fundamentally different from the claimed monitoring of response drift over time through comparison of stored and subsequently generated responses.” Remarks, 12-16.
In response: Avusingi ¶ 7 discloses “the computing system may automatically resubmit the original inquiry to the LLM and obtain a new answer from the LLM for the original inquiry, in an attempt to obtain a better and higher quality answer”. Evidently, “a better and higher quality answer” requires comparing the quality/score of an old answer with quality/score of a new answer for selecting “a better and higher quality answer”.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHSEN ALMANI whose telephone number is (571)270-7722. The examiner can normally be reached on M-F, 9:00 to 5:00.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann J. Lo can be reached on 571-272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHSEN ALMANI/Primary Examiner, Art Unit 2159
Prosecution Insights