DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This action is responsive to the following communication: Original claims filed 09/29/23. This action is made non-final.
3. Claims 1-20 are pending in the case. Claims 1, 4 and 15 are independent claims.
Claim Objections
4. Claims 11-12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
6. Claims 1, 2, 4, 5 and 13-16 XXX are rejected under 35 U.S.C. 102(a)(1) as being rejected by anticipated by Zhang (US 20250077777).
Regarding claim 1, Zhang discloses a computer-implemented method for sampling and cascading large language models (LLMs) with equivalence checking, the method performed by one or more computing devices, the method comprising:
receiving a prompt (see FIG. 1, prompt at 101);
sampling a first large language model (LLM) based on the prompt to obtain a first set of sample answers generated by the first LLM, each sample answer of the first set of sample answers comprising a respective domain-specific text (see FIG. 1, prompt goes to an LLM which generates a response/answer comprising a set of responses);
using an equivalence checker to determine that a first pair of respective domain-specific texts of the first set of sample answers are not equivalent (see FIG. 2, wherein equivalent prompts are sent to a first and second LLM to determine and cross-check its answers to determine equivalence, see also FIG. 4 wherein consistency scores are used to determine equivalence);
sampling a second large language model (LLM) based on the prompt to obtain a second set of sample answers generated by the second LLM, each sample answer of the second set of sample answers comprising a respective domain-specific text (see FIG. 2, wherein equivalent prompts are sent to a first and second LLM to determine and cross-check its answers to determine equivalence, see also FIG. 4 wherein consistency scores are used to determine equivalence);
using an equivalence checker to determine that a second pair of respective domain-specific texts of the second set of sample answers are equivalent (see FIG. 2, wherein equivalent prompts are sent to a first and second LLM to determine and cross-check its answers to determine equivalence, see also FIG. 4 wherein consistency scores are used to determine equivalence);
determining, based on the domain-specific texts of the second pair, a particular domain- specific text for display in a graphical user interface (see FIG. 2, wherein equivalent prompts are sent to a first and second LLM to determine and cross-check its answers to determine equivalence, see also FIG. 4 wherein consistency scores are used to determine equivalence);and
providing the particular domain-specific text for display in a graphical user interface (see paragraph 0039, interface s used to display feedback).
Regarding claim 2, Zhang discloses wherein the prompt is expressed in a natural language (the generative AI models referred to herein are a generative AI model configured to generate a natural language answer to a question provided to the model. Such generative AI models may be referred to as large language models, or LLMs, paragraph 0022).
Regarding claim 5 and 16, the subject matter of the claim is substantially similar to claim 2 and as such the same rationale of rejection applies.
Regarding claim 4, Zhang discloses a computer-implemented method performed by one or more computing devices, the method comprising:
receiving a first prompt (see FIG. 1, prompt at 101);
sampling a large language model (LLM) based on the first prompt to obtain a first set of sample answers generated by the first LLM, each sample answer of the first set of sample answers comprising a respective domain-specific text (see FIG. 1, prompt goes to an LLM which generates a response/answer comprising a set of responses);
using an equivalence checker to determine that a first pair of respective domain-specific texts of the first set of sample answers are not equivalent (see FIG. 2, wherein equivalent prompts are sent to a first and second LLM to determine and cross-check its answers to determine equivalence, see also FIG. 4 wherein consistency scores are used to determine equivalence);
receiving a second prompt, wherein the second prompt is a refinement of the first prompt; sampling the large language model (LLM) based on the second prompt to obtain a second set of sample answers generated by the second LLM, each sample answer of the second set of sample answers comprising a respective domain-specific text (see FIG. 2, wherein equivalent prompts are sent to a first and second LLM to determine and cross-check its answers to determine equivalence, see also FIG. 4 wherein consistency scores are used to determine equivalence);
using an equivalence checker to determine that a second pair of respective domain-specific texts of the second set of sample answers are equivalent (see FIG. 2, wherein equivalent prompts are sent to a first and second LLM to determine and cross-check its answers to determine equivalence, see also FIG. 4 wherein consistency scores are used to determine equivalence);
determining, based on the domain-specific texts of the second pair, a particular domain- specific text for display in a graphical user interface (see FIG. 2, wherein equivalent prompts are sent to a first and second LLM to determine and cross-check its answers to determine equivalence, see also FIG. 4 wherein consistency scores are used to determine equivalence); and
providing the particular domain-specific text for display in a graphical user interface (see paragraph 0039, interface s used to display feedback).
Regarding claim 15, the subject matter of the claim is substantially similar to claim 1 and as such the same rationale of rejection applies.
Regarding claim 13, Zhang discloses wherein the first prompt is received by a service in a provider network from a customer device in a customer network; and wherein the particular domain-specific text is returned by the server in the provider network to the customer device in the customer network (see FIG. 5 wherein the prompt is sent two LLMs to determine if the answers are semantically relevant and if the answers are equivalent responses and in return is displayed to the user as seen in FIG. 3).
Regarding claim 14, Zhang discloses wherein the large language model (LLM) is a first LLM, and whether the method further comprises:
prior to sampling the first large language model based on the first prompt: (receiving a first prompt for submission, see FIG. 6)
sampling a second large language model (LLM) based on the first prompt to obtain a third set of sample answers generated by the second LLM, each sample answer of the third set of sample answers comprising a respective domain-specific text (see FIG. 6, generating a second and third response using the first and second LLM;
using the equivalence checker to determine that a third pair of respective domain-specific texts of the third set of sample answers are not equivalent (see FIG. 6 wherein a semantic consistency score is determined for all prompts and responses); and
determining to sample the first large language model (LLM) based on the first prompt based on determining that the third pair of respective domain-specific texts of the third set of sample answers are not equivalent (see FIG. 6 wherein a semantic consistency score is determined for all prompts and responses).
Claim Rejections - 35 USC § 103
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claim 3, 6, 7, 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Rennie (US 20260195332).
Regarding claim 3, Zhang does not disclose wherein each sample answer of the first set of answers comprises a respective definition of a function in a programming language.
However, Rennie discloses wherein a method for training a collection-specific generative large language model for forming a SQL representation of a natural language query includes processing a number of SQL queries in a generative large language model to determine a corresponding number of natural language representations of the SQL queries, training an intermediate generative large language model to convert natural language representations of SQL queries to SQL queries using the number of SQL queries and corresponding number of natural language representations of the SQL queries, processing a number of collection-specific natural language queries using the intermediate generative large language model to convert the collection-specific natural language queries to collection-specific SQL queries, and training the collection-specific generative large language model to convert natural language queries to collection-specific SQL queries using the number of collection-specific natural language queries and the corresponding number of collection-specific SQL queries (paragraph 0034).
The combination of Zhang and Rennie would have resulted in the LLM utilization of providing equivalence checkers to further utilize Rennie’s teachings of utilizing the response as SQL or the like. One would have been motivated to have combined the teachings because one would have benefited from utilizing programming functional code (such as SQL) to parse out the responses and the queries in order to determine semantic relationships. As such, the combination of references would have been obvious to one of ordinary skill in the art and the resulting invention would have ultimately been predictable to one of ordinary skill.
Regarding claim 6 and 17, the subject matter of the claim is substantially similar to claim 3 and as such the same rationale of rejection applies.
Regarding claim 7, Zhang does not disclose wherein each sample answer of the first set of answers comprises a respective structure query language (SQL) query.
However, Rennie discloses wherein a method for training a collection-specific generative large language model for forming a SQL representation of a natural language query includes processing a number of SQL queries in a generative large language model to determine a corresponding number of natural language representations of the SQL queries, training an intermediate generative large language model to convert natural language representations of SQL queries to SQL queries using the number of SQL queries and corresponding number of natural language representations of the SQL queries, processing a number of collection-specific natural language queries using the intermediate generative large language model to convert the collection-specific natural language queries to collection-specific SQL queries, and training the collection-specific generative large language model to convert natural language queries to collection-specific SQL queries using the number of collection-specific natural language queries and the corresponding number of collection-specific SQL queries (paragraph 0034).
The combination of Zhang and Rennie would have resulted in the LLM utilization of providing equivalence checkers to further utilize Rennie’s teachings of utilizing the response as SQL or the like. One would have been motivated to have combined the teachings because one would have benefited from utilizing programming functional code (such as SQL) to parse out the responses and the queries in order to determine semantic relationships. As such, the combination of references would have been obvious to one of ordinary skill in the art and the resulting invention would have ultimately been predictable to one of ordinary skill.
Regarding claim 18, the subject matter of the claim is substantially similar to claim 7 and as such the same rationale of rejection applies.
9. Claim 8, 9, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Neuhäußer.
Regarding claim 8, Zhang does not disclose wherein the equivalence checker comprises a satisfiability modulo theories (SMT) solver; and wherein using the equivalence checker to determine that the second pair of respective domain-specific texts of the second set of sample answers are equivalent is based on the SMT solver determining that a set of logical constraints representing the first pair of respective domain-specific texts is unsatisfiable.
However, Neuhäußer discloses wherein a satisfiability modulo theories solver can be used to check the satisfiability of a given configuration model. It is possible to derive all possible configurations (also called “variants”) of a configuration model. If no variants can be derived from a configuration model, then the model is “inconsistent.” An SMT solver can be used to compute the reason for the inconsistency (see paragraph 0013).
The combination of Zhang and Neuhäußer would have resulted in the LLM utilization of providing equivalence checkers to further utilize Neuhäußer teachings of SMT solver to check for equivalence. One would have been motivated to have combined the teachings because one would have benefited from utilizing programming functional code (such as SQL) to parse out the responses and the queries in order to determine semantic relationships. As such, the combination of references would have been obvious to one of ordinary skill in the art and the resulting invention would have ultimately been predictable to one of ordinary skill.
Regarding claim 19, the subject matter of the claim is substantially similar to claim 8 and as such the same rationale of rejection applies.
Regarding claim 9, Zhang does not disclose wherein the equivalence checker comprises a satisfiability modulo theories (SMT) solver; and wherein using the equivalence checker to determine that the first pair of respective domain-specific texts of the first set of sample answers are not equivalent is based on the SMT solver determining that a set of logical constraints representing the first pair of respective domain-specific texts is satisfiable.
However, Neuhäußer discloses wherein a satisfiability modulo theories solver can be used to check the satisfiability of a given configuration model. It is possible to derive all possible configurations (also called “variants”) of a configuration model. If no variants can be derived from a configuration model, then the model is “inconsistent.” An SMT solver can be used to compute the reason for the inconsistency (see paragraph 0013).
The combination of Zhang and Neuhäußer would have resulted in the LLM utilization of providing equivalence checkers to further utilize Neuhäußer teachings of SMT solver to check for equivalence. One would have been motivated to have combined the teachings because one would have benefited from utilizing programming functional code (such as SQL) to parse out the responses and the queries in order to determine semantic relationships. As such, the combination of references would have been obvious to one of ordinary skill in the art and the resulting invention would have ultimately been predictable to one of ordinary skill.
Regarding claim 20, the subject matter of the claim is substantially similar to claim 9 and as such the same rationale of rejection applies.
10. Claim 10 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Ivanic (US 20100293530).
Regarding claim 10, Zhang does not disclose further comprising: obtaining a witness from the equivalence checker for the first pair of respective domain- specific texts, the witness comprising a counterexample to equivalence of the first pair of texts; and presenting the counterexample to equivalence of the first pair of texts in a graphical user interface.
However, Ivancic discloses wherein After model checking the generated model 117, counterexamples/witness traces 120 can be reported and provided to the refinement module 122, which can thereafter selectively reduce the value of one or more of the parametric constants based on spurious counterexamples included in report 120. The refinement block 122 can find spurious counterexamples by determining the feasibility of witness traces on model 117. For relatively infeasible paths, the refinement module 116 can decrease the corresponding parametric constant(s) to correct for overly imprecise modeling of floating point operations. After adjusting the value of the parametric constants in the model 117, the refinement block 122 can provide a refined model 117 to the model checker 118, which, in turn, may provide counterexamples/witness traces 120 for the refined model. The refinement module 122 may then receive the updated witness traces 120 and the process can be iterated until optimal values for the parametric constants are obtained. Thereafter, the final counterexample/witness trace report 120 can be output to a user. It should be noted that the refinement process can be combined with other abstract-refinement based techniques, such as predicate abstraction (paragraph 0080).
The combination of Zhang and IVanic would have resulted in the LLM utilization of providing equivalence checkers to further utilize Ivanic’s teachings of witness/counterexamples solver techniques to check for equivalence. One would have been motivated to have combined the teachings because one would have benefited from utilizing programming functional code (such as SQL) to parse out the responses and the queries in order to determine semantic relationships. As such, the combination of references would have been obvious to one of ordinary skill in the art and the resulting invention would have ultimately been predictable to one of ordinary skill.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID E CHOI whose telephone number is (571)270-3780. The examiner can normally be reached on M-F: 7-2, 7-10 (PST). If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bechtold, Michelle T. can be reached on (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID E CHOI/Primary Examiner, Art Unit 2148
Prosecution Insights