DETAILED ACTION
This communication is in response to the Application filed on 07/30/2024. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 13, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/30/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claim 20 objected to because of the following informalities: The claim is listed to be dependent on claim 8 when it presumably should be dependent on claim 15. For the purposes of compact prosecution, it has been examined as if it were dependent on claim 15. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1, 8, and 15 recite A [computer-implemented] method comprising: accessing a set of training examples, wherein each training example of the set of training examples comprises a natural language utterance, a gold logical form corresponding to the natural language utterance, and database schema information; generating a set of error correction training examples based on the set of training examples, wherein generating the set of error correction training examples comprises performing an iterative process for each training example of the set of training examples, and wherein the iterative process comprises: generating, by a text-to-logical form model, an inferred logical form based on the natural language utterance and the database schema information, executing the inferred logical form on a database corresponding to the database schema information, when executing the inferred logical form on the database fails, obtaining an execution error message corresponding to the failure, and recording the natural language utterance, the gold logical form, the database schema information, the inferred logical form, and the execution error message as an execution error example, and populating an error correction prompt template with the execution error example to generate an error correction training example; and training a [machine learning model] with the set of training examples and the set of error correction training examples to generate a trained machine learning model.
The limitations in these claims, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. This is due to the fact that the method primarily revolves around altering/expanding upon data in data set which the human mind is capable of doing. To exhibit this, we’ll look at an example of a librarian training to get better at finding books in the library. They can receive a dataset of natural language utterances (a book name), gold logical forms (a code representing the location of the book), and database schema information (information about the libraries organization system) as a hand written training document. They could then form their own logical representation for each of the natural language utterances by trying to write down the code they think represents the location of the book. They could test this by going to the location and seeing if they were correct, if the code they created doesn’t exist in the library it could be considered an execution error and they could make note of this. Then they could include the code and associated error message on a handwritten dataset with the information form the initial dataset. Finally, the librarian could use this example information to learn from their mistakes and create more accurate location codes in the future. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. The claims recite the additional components of a computer and a machine learning model. The computer is merely used to apply the steps within the method. The computer is described in paragraph 195 of the specification with a generic description of the sub-components in the system. The machine learning model is merely receiving the completed data set created by method steps. The machine learning model is said to be an LLM which is described in paragraph 107 as a general artificial intelligence model. Claims 8 and 15 specifically lists the additional components of a processor and a computer-readable media. The processor is merely used to apply the steps within the method via a computing device. The processor is described in paragraph 197 of the specification with a generic description of the component. The computer-readable media is merely used to apply the steps within the method via a computing device. The computer-readable media is described in paragraph 207 of the specification with a generic description/examples of the component. Accordingly, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claims are not patent eligible.
Claims 2, 9, and 16 recite wherein the iterative process further comprises: when executing the inferred logical form on the database succeeds, obtaining an inferred result, executing the gold logical form on the database corresponding to the database schema information to obtain a gold result, and comparing the inferred result to the gold result; when the inferred result and the gold result are dissimilar, generating an error message corresponding to the dissimilarity, and recording the natural language utterance, the gold logical form, the database schema information, the inferred logical form, and the error message as a semantic error example; and populating the error correction prompt template with the semantic error example to generate an error correction training example.
The limitation(s) in these claims, as drafted, is/are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. The human mind can create their own logic forms and compare them to a known correct (gold) logic form. From the librarian example this would be them guessing the location code for a book before comparing it to the known location written in the training sheet. They could note the location code dissimilarities on a handwritten data set in order to learn from their mistakes in the future. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite any additional elements that were not present in the independent claims. Accordingly, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claims are not patent eligible.
Claims 3, 10, and 17 recite wherein the iterative process further comprises: populating an error correction instruction prompt template with the semantic error example to generate an error correction instruction example; generating, by a [large language model], error correction instructions based on the error correction instruction example; recording the natural language utterance, the gold logical form, the database schema information, the inferred logical form, the semantic error message, and the error correction instructions as an enhanced semantic error example; and populating the error correction prompt template with the enhanced semantic error example to generate an error correction training example.
The limitation(s) in these claims, as drafted, is/are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. The human mind is capable of creating instructions to correct differences in two query results. Using prior knowledge of how the database language works that can see what was different about the results and why the difference happens. The librarian could implement memorization techniques as instructions in order to help do things correctly in the future. They could then write down these instructions with the other information in their training sheet so when they learn from it in the future they have added information. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. The claims recite the additional component(s) of Additional components. (For each component) (Apply it, extra-solution, generally linking to technological environment), (Cite spec description for generality (or state that applicant doesn’t differentiate it from generic example and use prior art). Claim X specifically lists the additional components of (repeat above sentence). Accordingly, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claims are not patent eligible.
Claims 4, 11, and 18 recite wherein the iterative process further comprises: appending a text-to-logical form prompt template with the error correction instructions to generate an enhanced text-to-logical form prompt template; generating, by the text-to-logical form model, random logical forms based on the text-to-logical form prompt template; executing each of the random logical forms on a database corresponding to the database schema information to obtain a result for each of the random logical forms; comparing the result for each of the random logical forms to the gold result; and when a proportion of the results that are similar or the same to the gold result based on the comparing is greater than or equal to a predetermined threshold, the gold logical form, the database schema information, the inferred logical form, the semantic error message, and the error correction instructions are recorded as the enhanced semantic error example.
The limitation(s) in these claims, as drafted, is/are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. The librarian from the previous examples could include the instructions they created in a template for assistance in future conversions. They could then apply that template in many different ways for the same book to see if the instructions allow for many mistakes. They could then continuously compare the results from the instruction-based location codes to the known location code to see if a certain ratio of them match. They could modify the instruction and then repeat this process until an acceptable amount of the logic forms match the gold logic form result. Finally, they could write these instructions down in their handwritten dataset in order to learn from it in the future. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite any additional elements that were not present in the independent claims. Accordingly, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claims are not patent eligible.
Claims 5, 12, and 19 recite wherein: the training examples in the set of training examples are selected randomly from a batch of text-to-logical form examples; and the set of training examples is a subset of the batch of text-to-logical form examples.
The limitation(s) in these claims, as drafted, is/are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human can randomly select which training examples they will use from a larger set of training examples. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite any additional elements that were not present in the independent claims. Accordingly, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claims are not patent eligible.
Claims 6 and 13 recite wherein the machine learning model is a large language model pretrained for a task of converting an input natural language utterance to an output logical form.
The limitation(s) in these claims, as drafted, is/are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. The decision for the MLM to be an LLM is a design decision that the human mind is capable of making. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite any additional elements that were not present in the independent claims. Accordingly, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claims are not patent eligible.
Claims 7, 14, and 20 recite further comprising: receiving an input natural language utterance from a user; converting, using the trained machine learning model, the input natural language utterance to an output logical form based on the input natural language utterance and database schema information; executing the output logical form on a database corresponding to the database schema information to obtain a result; and providing the result to the user.
The limitation(s) in these claims, as drafted, is/are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. From the librarian example, the librarian could receive a book title from someone else, create a location code associated with it, go to that location in the library to retrieve it, and then give the book to the person who requested it. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite any additional elements that were not present in the independent claims. Accordingly, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claims are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 6, 8, 9, 13, 15, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Application Publication US 20250147956 A1 (Payani et al.) in view of “Analyzing the Effectiveness of Large Language Models on Text-to-SQL Synthesis” (Roberson et al.).
Regarding Claims 1, 8, and 15, Payani et al. teaches A computer-implemented method comprising:
(In various implementations, as detailed further below, output correction process 248 may include computer executable instructions that, when executed by processor(s) 220, cause device 200 to perform the techniques described herein.) (Paragraph 25).
Claim 8 presents the alternative limitation of A system comprising: one or more processors; and one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations comprising:
(Device 200 may comprise one or more network interfaces (e.g., network interfaces 210) (e.g., wired, wireless, etc.), at least one processor (e.g., processor(s) 220), and a memory 240 interconnected by a system bus 250, as well as a power supply 260 (e.g., battery, plug-in, etc.).) (Paragraph 20).
Claim 15 presents the alternative limitation of One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause a system to perform operations comprising:
(Alternatively, a tangible, non-transitory, computer-readable medium may have computer-executable instructions stored thereon that, when executed by a processor on a computer, cause the computer to perform a method according to procedure 500.) (Paragraph 63).
accessing a set of training examples, wherein each training example of the set of training examples comprises a natural language utterance, a gold logical form corresponding to the natural language utterance, and database schema information;
(In various implementations, output correction process 248 may employ one or more supervised, unsupervised, or semi-supervised machine learning models. Generally, supervised learning entails the use of a training set of data, as noted above, that is used to train the model to apply labels to the input data.) (Paragraph 26).
(In some embodiments, the FOL program 318 can be converted and compared to a ground truth during a training phase performed by the LLM 322. The LLM 322 can perform one or more actions to fix errors in the conversion of the FOL program 318.) (Paragraph 44).
(Some simplistic examples of the type of edit(s) that could be included in the second prompt 428 include corrections to the SQL computer code 414 that may have been introduced by the LLM 412, such as “wrong table used,” “incorrect order,” etc., although it will be appreciated that other edit(s) and/or suggested corrections that are more (or less) complicated can be provided in the second prompt 428.) (Paragraph 57)
Whilst training the LLM a reward metric is found. In order to find this, training sets are used that include natural language utterances, ground truth data (gold), and schema information such as table names.
generating a set of error correction training examples based on the set of training examples, wherein generating the set of error correction training examples comprises performing an iterative process for each training example of the set of training examples,
(In some embodiments, the LLM 422 can be initially trained using these supervised fine-tuning phases via injection of synthetic perturbations. For example, for each natural language, SQL pair in a data set used to initially train the LLM 422, a set of predefined perturbations can be generated and then used to train the LLM 422 to classify the perturbations.) (Paragraph 60).
(Some non-limiting examples of predefined perturbations that can generated and injected into the LLM 422 during this initial training phase can include simple alterations, such as “change table name,” “change order” (e.g., from ascending to descending or vice versa), “syntax error(s),” and/or “operator reordering,” among others, although it will be appreciated that other alteration(s) that are more (or less) complicated can be provided in to the LLM 422 during the initial training phase.) (Paragraph 61).
The system shows an iterative process finding error information for each training example.
and wherein the iterative process comprises: generating, by a text-to-logical form model, an inferred logical form based on the natural language utterance and the database schema information,
(The approaches described herein transform natural language to SQL based on a multi-pronged or “tiered” processing approach to reducing errors and/or hallucinations in responses from digital assistants.) (Paragraph 36).
(As discussed in more detail below, the LLM 312 can be used to convert facts from an initial prompt 310 (e.g., a “query”) into first order logic (FOL), such as the FOL 314.) (Paragraph 43).
The first LLM converts the initial prompt to a first order logic or more specifically, a SQL query.
(when executing the inferred logical form on the database fails, obtaining an execution error message corresponding to the failure) (taught by Roberson et al.), and recording the natural language utterance, the gold logical form, the database schema information, the inferred logical form, and the execution error message as an execution error example,
(The LLM 322 can perform one or more actions to fix errors in the conversion of the FOL program 318. The modified query (e.g., the FOL 314 and/or the FOL program 318) can be evaluated with a solver 320 (e.g., a Boolean satisfiability solver, a reasoner tool, etc.) to obtain a reward function.) (Paragraph 44).
(The FOL 314 can be parsed using, for example, parser 316 to generate a FOL program 318. In some embodiments, the parser 316 can be configured to parse a query, such as the initial prompt 310 into a string field. Further, the parser 316 may, in some embodiments, ensure that the syntax of the FOL 314 is correct. The FOL program 318 can be processed by a solver 320 to generate a reward function, r.sub.τ. The reward function is then provided to the LLM 322, which uses the reward function to perform reinforcement training.) (Paragraph 48).
(In some embodiments, the second large language model can generate an output and provide the output to the first large language model. This output can be generated as part of processes to correct the error(s) associated with the first large language model, as detailed above. Such an output can include one or more corrections (e.g., edits) to the FOL 314, the FOL program 318, the SQL computer code 414, and/or the SQL query 417, as detailed above.) (Paragraph 69).
The system creates a reward function from the errors it finds when the first LLM converts a prompt to first order logic. An error such as incorrect syntax would result in an execution error that would cause a failure when executing the logic form on a database. While Payani et al. detects these errors prior to execution, the secondary reference from Roberson et al. teaches identifying the errors after executing the logic form on a database.
and populating an error correction prompt template with the execution error example to generate an error correction training example;
(In some embodiments, the FOL program 318 can be converted and compared to a ground truth during a training phase performed by the LLM 322. The LLM 322 can perform one or more actions to fix errors in the conversion of the FOL program 318. The modified query (e.g., the FOL 314 and/or the FOL program 318) can be evaluated with a solver 320 (e.g., a Boolean satisfiability solver, a reasoner tool, etc.) to obtain a reward function. The reward function can then be used to train the LLM 322.) (Paragraph 44).
Individual errors are found to form the reward metrics which train the LLM. In this sense, these errors are being used as a training set for the LLM where the reward metric acts as a template for them.
and training a machine learning model with the set of training examples and the set of error correction training examples to generate a trained machine learning model.
(In some embodiments, the FOL program 318 can be converted and compared to a ground truth during a training phase performed by the LLM 322. The LLM 322 can perform one or more actions to fix errors in the conversion of the FOL program 318. The modified query (e.g., the FOL 314 and/or the FOL program 318) can be evaluated with a solver 320 (e.g., a Boolean satisfiability solver, a reasoner tool, etc.) to obtain a reward function. The reward function can then be used to train the LLM 322.) (Paragraph 44).
The LLM is trained using the generated reward function made from errors found using a natural language query, a ground truth, a generated logic form, schema information, and execution error messages.
Payani et al. does not explicitly teach: executing the inferred logical form on a database corresponding to the database schema information, obtaining an execution error message corresponding to the failure
However, Roberson et al. teaches executing the inferred logical form on a database corresponding to the database schema information,
(The strategy involves executing the previously generated SQL query on the actual database from the spider dataset to detect any execution errors. Should an execution error be identified, it is incorporated into the input provided to gpt-4-turbo.) (Section 3, Sub-section Error Driven Correction).
obtaining an execution error message corresponding to the failure
(The strategy involves executing the previously generated SQL query on the actual database from the spider dataset to detect any execution errors. Should an execution error be identified, it is incorporated into the input provided to gpt-4-turbo.) (Section 3, Sub-section Error Driven Correction).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the WHATS MODIFIED as taught by Payani et al. to execute the generated logic forms in order to receive an error message as taught by Roberson et al. This would have been an obvious improvement as Roberson et al. explicitly states that this method of error correction improves execution accuracy in their tests (Roberson et al. Section 3, Sub-section Error Driven Correction).
Regarding Claims 2, 9, and 16, Payani et al. in view of Roberson et al. teaches the system of claims 1, 8, and 15.
Furthermore, Payani et al. teaches wherein the iterative process further comprises: when executing the inferred logical form on the database succeeds, obtaining an inferred result, executing the gold logical form on the database corresponding to the database schema information to obtain a gold result, and comparing the inferred result to the gold result;
(In some embodiments, the FOL program 318 can be converted and compared to a ground truth during a training phase performed by the LLM 322. The LLM 322 can perform one or more actions to fix errors in the conversion of the FOL program 318. The modified query (e.g., the FOL 314 and/or the FOL program 318) can be evaluated with a solver 320 (e.g., a Boolean satisfiability solver, a reasoner tool, etc.) to obtain a reward function. The reward function can then be used to train the LLM 322.) (Paragraph 44).
Payani et al. compares the generated logic form to a gold form in order to obtain error information that is used to construct a reward function.
Payani et al. does not show this comparison happening to the result of executing the logic forms on a database. Roberson et al. teaches this execution followed by comparison.
(The incorporation of example-driven correction, utilizing output examples to represent the expected result table after executing the queries on a real database… To streamline this approach, we utilized the ground truth (gold) result table in markdown format. Once the fine-tuned gpt-3.5-turbo-16k generates a SQL query, its output is compared with the provided example. If there is a discrepancy, the model is informed that the output of the generated query does not align with expectations.) (Roberson et al. Section 3, Sub-section Example Driven Correction).
when the inferred result and the gold result are dissimilar, generating an error message corresponding to the dissimilarity, and recording the natural language utterance, the gold logical form, the database schema information, the inferred logical form, and the error message as a semantic error example;
(In some embodiments, the FOL program 318 can be converted and compared to a ground truth during a training phase performed by the LLM 322. The LLM 322 can perform one or more actions to fix errors in the conversion of the FOL program 318. The modified query (e.g., the FOL 314 and/or the FOL program 318) can be evaluated with a solver 320 (e.g., a Boolean satisfiability solver, a reasoner tool, etc.) to obtain a reward function. The reward function can then be used to train the LLM 322.) (Paragraph 44).
(The reward metric can be analogous to the reward function (r.sub.τ) described above in connection with FIG. 3 and FIG. 4, above, while the solver can be analogous to the solver 320 of FIG. 3 and/or the solver 420 of FIG. 4. In some embodiments, the solver can be a Boolean satisfiability solver, a reasoner tool, or other solver, as described above.) (Paragraph 66).
Payani et al. creates an error function using various metrics such as generating a logic form from a natural language utterance, comparing it to a ground truth form, and then evaluating it with a solver. The detection of this error represents an error message being received. The actions used to fix the error and the reward function calculated represents the semantic error example. In this instance the natural language, schema information, inferred logic form, gold logic form, and error information are all necessary to create the reward function which is then used to train the LLM. This can be seen in Figs. 3-5 and the reward function is analogous to the error examples used for training.
Once again, Payani et al. does not show this comparison happening to the result of executing the logic forms on a database. Roberson et al. teaches this execution followed by comparison.
(The incorporation of example-driven correction, utilizing output examples to represent the expected result table after executing the queries on a real database… To streamline this approach, we utilized the ground truth (gold) result table in markdown format. Once the fine-tuned gpt-3.5-turbo-16k generates a SQL query, its output is compared with the provided example. If there is a discrepancy, the model is informed that the output of the generated query does not align with expectations.) (Roberson et al. Section 3, Sub-section Example Driven Correction).
and populating the error correction prompt template with the semantic error example to generate an error correction training example.
(At step 520, as detailed above, the device tunes a second large language model configured to correct errors of the first large language model using reinforcement learning. In some embodiments, the device tunes a second large language model configured to correct errors of the first large language model using reinforcement learning based on the reward metric.) (Paragraph 67).
The reward function acts as the error correction training examples and trains the model.
Regarding Claims 6 and 13, Payani et al. in view of Roberson et al. teaches the system of claims 1 and 8.
Furthermore, Payani et al. teaches: wherein the machine learning model is a large language model pretrained for a task of converting an input natural language utterance to an output logical form.
(The approaches described herein transform natural language to SQL based on a multi-pronged or “tiered” processing approach to reducing errors and/or hallucinations in responses from digital assistants. Using a multi-pronged query processing strategy, multiple NL-LF translators and/or large language models (LLMs) can be employed collaboratively to improve responses to queries using less training data.) (Paragraph 36).
Payani et al. utilizes an LLM for their method.
Claims 3, 10, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Application Publication US 20250147956 A1 (Payani et al.) in view of “Analyzing the Effectiveness of Large Language Models on Text-to-SQL Synthesis” (Roberson et al.) and further in view of “EPI-SQL: Enhancing Text-to-SQL Translation with Error-Prevention Instructions” (Liu et al.).
Regarding Claims 3, 10, and 17, Payani et al. teaches the method of claims 1, 8, and 15.
Payani et al. in view of Roberson et al. does not explicitly teach: wherein the iterative process further comprises: populating an error correction instruction prompt template with the semantic error example to generate an error correction instruction example; generating, by a large language model, error correction instructions based on the error correction instruction example; recording the natural language utterance, the gold logical form, the database schema information, the inferred logical form, the semantic error message, and the error correction instructions as an enhanced semantic error example; and populating the error correction prompt template with the enhanced semantic error example to generate an error correction training example.
However, Liu et al. teaches wherein the iterative process further comprises: populating an error correction instruction prompt template with the semantic error example to generate an error correction instruction example;
(In our study, we employed a zero-shot prompting technique on the training set of the Spider dataset (Yu et al., 2018) to produce a response for each instance. Subsequently, the generated responses were compared with the gold-standard answers, and the examples that were not generated accurately were collected.) (Section 2.3).
(After the error-prone instances are collected, they are fed into LLMs. The LLMs are tasked with formulating a set of instructions aimed at circumventing these errors.) (Section 2.4, Paragraph 1).
Liu et al. identifies an error in a text-to-SQL process by comparing it to a gold standard output.
generating, by a large language model, error correction instructions based on the error correction instruction example;
(After the error-prone instances are collected, they are fed into LLMs. The LLMs are tasked with formulating a set of instructions aimed at circumventing these errors.) (Section 2.4, Paragraph 1).
An LLM generates instructions to correct the errors.
recording the natural language utterance, the gold logical form, the database schema information, the inferred logical form, the semantic error message, and the error correction instructions as an enhanced semantic error example;
(If the EPI ES successfully guides the LLM to handle the instance S correctly, ES along with the question QS in S, and the erroneous SQLS generated for S will be recorded in a set QSESet (Question-SQL-EPI set).) (Section 2.4, Paragraph 2).
A training set QSESet is formed from the training set (natural language, gold standard, and schema information from Spider dataset) and the error prevention instructions. This can be seen in Fig. 2 parts 1 and 2.
and populating the error correction prompt template with the enhanced semantic error example to generate an error correction training example.
(In this phase, EPIs are harnessed to facilitate the generation of SQL. To this end, we introduce what we term the SQLGen+EPI prompt, as illustrated in Figure 4 (b). The SQLGen+EPI prompt comprises three constituents: the question, the schema of the database, and the instructions, which notably include the EPI.) (Section 2.6, Paragraph 1).
The QSEset with the error correction templates is combined with future prompts to improve the SQL generated by the LLM. This can be seen in Fig. 2 part 4.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the natural language to logic form method as taught by Payani et al. in view of Roberson et al. to create error correction instructions as taught by Liu et al. This would have been an obvious improvement to provide guidance to the LLM on how to accurately complete the task and avoid potential errors (Liu et al. Section 1, Paragraph 5).
Claims 5, 12, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Application Publication US 20250147956 A1 (Payani et al.) in view of “Analyzing the Effectiveness of Large Language Models on Text-to-SQL Synthesis” (Roberson et al.) and further in view of “Exploring Chain of Thought Style Prompting for Text-to-SQL” (Tai et al.).
Regarding Claims 5, 12, and 19, Payani et al. in view of Roberson et al. teaches the system of claims 1, 8, and 15.
Payani et al. in view of Roberson et al. does not explicitly teach: wherein: the training examples in the set of training examples are selected randomly from a batch of text-to-logical form examples; and the set of training examples is a subset of the batch of text-to-logical form examples.
However, Tai et al. teaches wherein: the training examples in the set of training examples are selected randomly from a batch of text-to-logical form examples;
(Spider is a commonly used benchmark to evaluate text-to-SQL parsing in a cross-database setting, which requires models to generalize to novel database schemas. The dataset consists of 7,000 question-query pairs in the training set and 1,034 pairs in the development set, covering 200 different databases and 138 do mains.) (Section 4.1, Paragraph 1).
(For random selection, we uniformly sample in context examples from the Spider training set at random.) (Section 4.2, Paragraph 2).
Tai et al. teaches a text-to-SQL method which uses the Spider data set and randomly samples it to form a subset of data.
and the set of training examples is a subset of the batch of text-to-logical form examples.
(For random selection, we uniformly sample in context examples from the Spider training set at random.) (Section 4.2, Paragraph 2).
By only including in context examples a subset of the Spider dataset is formed.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the natural language to logic form method as taught by Payani et al. in view of Roberson et al. use a randomized subset of training samples as taught by Tai et al. This would have been an obvious improvement as it is a simple method with an ease of execution (Tai et al. Section 4.2, Paragraph 1).
Claims 7, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Application Publication US 20250147956 A1 (Payani et al.) in view of “Analyzing the Effectiveness of Large Language Models on Text-to-SQL Synthesis” (Roberson et al.) and further in view of US Patent Application Publication US 20220309106 A1 (Ramos et al.)
Regarding Claims 7, 14, and 20, Payani et al. in view of Roberson et al. teaches the system of claims 1, 8, and 15.
Furthermore, Payani et al. teaches: receiving an input natural language utterance from a user;
(Procedure 500 may start at step 505, and continues to step 510, where, as described in greater detail above, a device (e.g., the device 200) inputs and/or receives an input prompt (e.g., the initial prompt 310 of FIG. 3 and/or the initial prompt 410 of FIG. 4) to a first large language model to generate an output.) (Paragraph 64).
converting, using the trained machine learning model, the input natural language utterance to an output logical form based on the input natural language utterance and database schema information;
(NLIDB allows users to access database information using natural language inquiries. This natural language database search capability has become more popular over recent years and, as such, companies are developing deep-learning approaches for accessing specific databases using natural language) (Paragraph 36).
(That is, the device can translate the input prompt to a format chosen from a group consisting of a structured query language format, a first order logic format, or any other suitable format.) (Paragraph 65).
(Accordingly, the initial prompt 410 can be generated by a user in, for example, natural language. The initial prompt 410 is received by an LLM 412.) (Paragraph 51).
(In the non-limiting example illustrated in FIG. 4, the LLM 412 can generate SQL computer code 414 based on the initial prompt 410. This SQL computer code 414 can be parsed by, for example a parser 416.) (Paragraph 52).
The LLM outputs a logical form based on the natural language input provided which includes information about the database.
Payani et al. in view of Roberson et al. does not explicitly teach executing the output logical form on a database corresponding to the database schema information to obtain a result; and providing the result to the user.
However, Ramos et al. teaches executing the output logical form on a database corresponding to the database schema information to obtain a result;
(At operation 308, the formal query is executed to identify data in a data store. For example, the formal query may be executed by a query processor (e.g., query processor 108 in FIG. 1) to identify data in a data store similar to data store 112.) (Paragraph 43).
and providing the result to the user.
(Accordingly, at operation 310, the identified data is provided in response to the natural language input that was received at operation 302.) (Paragraph 43).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the natural language to logic form method as taught by Payani et al. in view of Roberson et al. to execute the generated logic forms and provide the result as an output to a user as taught by Ramos et al. This would have been an obvious improvement to allow a user to identify data from a data store using only a natural language input (Ramos et al. Paragraph 43).
Allowable Subject Matter
Claim 4, 11, and 18 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Furthermore, the claims would need to be rewritten to overcome the current 35 U.S.C. 101 rejections detailed above.
The following is an examiner’s statement of reasons for allowance:
The closest prior art of record for claims 4, 11, and 18 are US Patent Application Publication US 20250147956 A1 (Payani et al.) in view of “Analyzing the Effectiveness of Large Language Models on Text-to-SQL Synthesis” (Roberson et al.) and further in view of “EPI-SQL: Enhancing Text-to-SQL Translation with Error-Prevention Instructions” (Liu et al.).
Payani et al. in view of Roberson et al. and Liu et al. does not explicitly teach wherein the iterative process further comprises: appending a text-to-logical form prompt template with the error correction instructions to generate an enhanced text-to-logical form prompt template; generating, by the text-to-logical form model, random logical forms based on the text-to-logical form prompt template; executing each of the random logical forms on a database corresponding to the database schema information to obtain a result for each of the random logical forms; comparing the result for each of the random logical forms to the gold result; and when a proportion of the results that are similar or the same to the gold result based on the comparing is greater than or equal to a predetermined threshold, the gold logical form, the database schema information, the inferred logical form, the semantic error message, and the error correction instructions are recorded as the enhanced semantic error example.
Payani et al. teaches the overarching method of using error information to train an LLM and improve its ability to convert natural language to a logical form.
Roberson et al. teaches the aspect of checking for errors after the logic form has been executed on a database rather than after it has been generated.
Liu et al. teaches using error information to form error correction templates that improve the LLMs ability to generate logic forms from natural language.
However, none of the prior at, either alone or in combination, overcomes the limitations as presented in claims 4, 11, and 18.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS DANIEL LOWEN whose telephone number is (571)272-5828. The examiner can normally be reached Mon-Fri 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D Shah can be reached at (571) 270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICHOLAS D LOWEN/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
03/19/2026