DETAILED ACTION
Claims 1-20 are presented for examination.
This office action is in response to submission of application on 05/13/2024.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 4 and 15 are objected to because of the following informalities: “…a respective answer for each the question-answer pair…” should be ““…a respective answer for each of the question-answer pair…”. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 2A Prong 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Claim 1 includes the steps of:
A method of automatically generating a dataset of question-answer pairs for domain-specific hallucination testing of generative language processing machine learning models, the method comprising: extracting natural language text from a domain-specific source document as a plurality of blocks of natural language text;
providing a first block of natural language text of the plurality of blocks of natural language text as an input to each of a plurality of question-answer pair generation models,
each of the plurality of question-answer pair generation models configured to generate a plurality of question-answer pairs from the first block of natural language text;
obtaining one or more confidence metrics for each of the plurality of question-answer pairs generated by each of the plurality of question-answer pair generation models;
filtering one or more question-answer pairs included in the plurality of question-answer pairs generated by one or more of the plurality of question-answer pair generation models based, at least in part, on the one or more confidence metrics generated for the one or more question-answer pairs; and
generating a dataset for domain-specific hallucination testing of a generative language processing machine learning model based on the filtering, the dataset comprising question-answer pairs remaining in the plurality of question-answer pairs after the filtering.
The broadest reasonable interpretation of the bolded limitations above are directed to a mental process able to be performed in the human mind through the use of a physical aid, like a pen and paper. A human can:
extract blocks of natural language text from a document,
generate question-answer pairs from blocks of natural language text,
calculate confidence metrics for generated question-answer pairs,
filter question-answer pairs based on confidence metrics,
generating a dataset comprising question-answer pairs remaining after filtering.
Step 2A Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application?
As drafted and under their broadest reasonable interpretation, the following limitations recite additional elements which amount to generic computer components recited at a high level of generality, with merely the words “apply it” or an equivalent with the judicial exception, merely including instructions to implement an abstract idea on the additional elements, or merely using the additional elements as a tool to perform an abstract idea, as discussed in MPEP 2106.05(f).
…a plurality of question-answer pair generation models,
each of the plurality of question-answer pair generation models configured to…,
generative language processing machine learning model.
As drafted and under their broadest reasonable interpretation, the following limitations recite additional elements which amount to mere data gathering and output recited at a high level of generality, and thus are insignificant extra-solution activity. See MPEP 2106.05(g) (“whether the limitation is significant”). In addition, all uses of the recited judicial exceptions require such data gathering and output, and, as such, these limitations do not impose any meaningful limits on the claim. These limitations amount to necessary data gathering and outputting. See MPEP 2106.05.
providing a first block of natural language text of the plurality of blocks of natural language text as an input to each of a plurality of question-answer pair generation models.
The additional elements have been considered both individually and as an ordered combination in order to determine whether they integrates the exception into a practical application. Therefore, no meaningful claim limits are imposed practicing the abstract idea. Accordingly, at Step 2A, prong two, the additional elements do not integrate the judicial exception into a practical application.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The claim limitation(s) reciting generic computer elements amounts to no more than mere instructions to apply the exception using a generic computer.
The claim reciting the additional element(s) of “providing…input”, “obtaining”, “receiving” and/or “transmitting” amount to necessary data gathering and output.
The additional elements have been considered both individually and as an ordered combination in order to determine whether they warrant significantly more consideration. Thus, the claim does not provide an inventive concept.
The claim is ineligible.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
The additional elements have been considered both individually and as an ordered combination in order to determine whether they warrant significantly more consideration. Thus, the claim does not provide an inventive concept.
The claim is ineligible.
Claims 2-11 further recite limitations that encompass mental evaluations that are practically performed in the human mind, but for the recitation of generic computer components (embedding model, entailment model, machine learning model, LLMs). The claims do not integrate the judicial exception into practical application. The claims do not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
Claims 2-11 are ineligible.
Claims 13-16 and 17-19 are substantially similar to claims 2-5 and 7-9 respectively, and are rejected on the same basis. These claims recite additional elements that amount to generic computer components recited at a high level of generality, with merely the words “apply it” or an equivalent with the judicial exception, merely including instructions to implement an abstract idea on the additional elements, or merely using the additional elements as a tool to perform an abstract idea.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 6, 7, 10, 11, 12, 13, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Neerukonda et al. (Pub. No.: US 2025/0156644A1), hereafter Neerukonda, in view of Lee et al. ("LIQUID: A Framework for List Question Answering Dataset Generation"), hereafter Lee.
Regarding claim 1, Neerukonda discloses:
A method of automatically generating a dataset of question-answer pairs for domain-specific hallucination testing of generative language processing machine learning models, the method comprising: extracting natural language text from a domain-specific source document as a plurality of blocks of natural language text (¶[0014] and ¶[0024-0025] teaches extracting units of text, i.e. a plurality of blocks of natural language text, from domain-specific source documents such as scientific articles),
providing a first block of natural language text of the plurality of blocks of natural language text as an input to each of a plurality of question-answer pair generation models, each of the plurality of question-answer pair generation models configured to generate a plurality of question-answer pairs from the first block of natural language text (Fig. 1, Fig. 4, and ¶[0026] teaches generating a plurality of synthetic question answer pairs from one or more models using blocks of natural language text as input),
obtaining one or more … metrics for each of the plurality of question-answer pairs generated by each of the plurality of question-answer pair generation models (Fig. 3 and ¶[0015] teaches obtaining the result of predefined metrics, such as entailment, for filtering of the plurality of question-answer pairs generated by each of the plurality of question-answer pair generation models),
filtering one or more question-answer pairs included in the plurality of question-answer pairs generated by one or more of the plurality of question-answer pair generation models based, at least in part, on the one or more … metrics generated for the one or more question-answer pairs (Fig. 3 and ¶[0015] teaches filtering one or more question-answer pairs included in the plurality of question-answer pairs based on the predefined metrics),
generating a dataset for domain-specific hallucination testing of a generative language processing machine learning model based on the filtering, the dataset comprising question-answer pairs remaining in the plurality of question-answer pairs after the filtering (Fig. 1 and ¶[0016] teaches generating a question answer pair dataset for domain specific hallucination testing of a generative language processing machine learning model based on the filtering).
While Neerukonda discloses obtaining one or more … metrics for each of the plurality of question-answer pairs generated by each of the plurality of question-answer pair generation models, they do not disclose this metric to be a confidence metric.
Lee discloses:
obtaining one or more confidence metrics for each of the plurality of question-answer pairs … (Figure 2 and Figure 2 caption teaches obtaining one or more confidence scores for each question answer pair).
While Neerukonda discloses filtering one or more question-answer pairs included in the plurality of question-answer pairs generated by one or more of the plurality of question-answer pair generation models based, at least in part, on the one or more … metrics generated for the one or more question-answer pairs, they do not disclose this metric to be a confidence metric.
Lee discloses:
filtering one or more question-answer pairs included in the plurality of question-answer pairs … based, at least in part, on the one or more confidence metrics generated for the one or more question-answer pairs (Figure 2 and Figure 2 caption and page 13016, left column, last paragraph “The triplet ⟨c, q,A′⟩ is not used if zero or one answer remains after filtering (i.e., M′ ≤ 1);” teaches filtering question-answer pairs, i.e. triplets, based on the confidence scores).
Neerukonda and Lee are analogous art because they are from the same field of endeavor: question answer pairs and machine learning models.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Neerukonda to include obtaining one or more confidence metrics for each of the plurality of question-answer pairs and filtering one or more question-answer pairs included in the plurality of question-answer pairs … based, at least in part, on the one or more confidence metrics generated for the one or more question-answer pairs, based on the teachings of Lee. One of ordinary skill in the art would have been motivated to make this modification in order to improve the quality of the question-answer pairs, as suggested by Lee (page 13015, left column, paragraph 1, lines 17-18).
Regarding claim 2, Neerukonda, in view of Lee, discloses the method of claim 1 (and thus the rejection of claim 1 is incorporated). Lee further discloses:
wherein the filtering comprises: comparing, for each respective question-answer pair, the one or more confidence metrics to one or more threshold confidence metrics (Algorithm 1, line 11),
determining the one or more confidence metrics for one or more question-answer pairs of the plurality of question-answer pairs generated by one or more question-answer pair generation models of the plurality of question-answer pair generation models do not satisfy the one or more threshold confidence metrics based on the comparing (Algorithm 1, lines 11-12),
removing the one or more question-answer pairs from the plurality of question-answer pairs generated by the one or more question-answer pair generation models (Algorithm 1, lines 11-12 and page 13016, left column, last paragraph “The triplet ⟨c, q,A′⟩ is not used if zero or one answer remains after filtering (i.e., M′ ≤ 1);”).
Regarding claim 6, Neerukonda, in view of Lee, discloses the method of claim 1 (and thus the rejection of claim 1 is incorporated). Neerukonda further discloses:
the plurality of question-answer pair generation models comprise a non-LLM (Examiner’s Note: a non-LLM is interpreted as any machine learning model that cannot accept natural language prompts, as per specification ¶[0026]) (¶[0028-0029] teaches the question-answer pair generation models to be non LLM machine learning models),
the obtaining comprises determining one or more … metrics for each of the plurality of question-answer pairs the non-LLM generated from the first block of natural language text, the filtering comprises filtering one or more question-answer pairs of the plurality of question-answer pairs generated by the non-LLM based, at least in part, on the one or more … metrics determined for the one or more question-answer pairs (Fig. 3 and ¶[0034] teaches determining the result of predefined metrics for each pair generated from the non-LLM and filtering the pairs based on these metrics).
While Neerukonda discloses determining one or more … metrics for each of the plurality of question-answer pairs the non-LLM generated from the first block of natural language text, the filtering comprises filtering one or more question-answer pairs of the plurality of question-answer pairs generated by the non-LLM based, at least in part, on the one or more … metrics determined for the one or more question-answer pairs, they do not disclose this metric to be a confidence metric.
Lee discloses:
determining one or more confidence metrics (Figure 2 and Figure 2 caption teaches obtaining one or more confidence scores for each question answer pair).
filtering one or more question-answer pairs … based, at least in part, on the one or more confidence metrics (Figure 2 and Figure 2 caption and page 13016, left column, last paragraph “The triplet ⟨c, q,A′⟩ is not used if zero or one answer remains after filtering (i.e., M′ ≤ 1);” teaches filtering question-answer pairs, i.e. triplets, based on the confidence scores).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Neerukonda to include determining one or more confidence metrics and filtering one or more question-answer pairs … based, at least in part, on the one or more confidence metrics, based on the teachings of Lee. One of ordinary skill in the art would have been motivated to make this modification in order to improve the quality of the question-answer pairs, as suggested by Lee (page 13015, left column, paragraph 1, lines 17-18).
Regarding claim 7, Neerukonda, in view of Lee, discloses the method of claim 1 (and thus the rejection of claim 1 is incorporated). Neerukonda further discloses:
wherein the domain-specific source document is an unstructured document (¶[0024] teaches news articles, scientific research, and etc. as unstructured document),
wherein extracting the natural language text from the domain-specific source document as a plurality of blocks of natural language text comprises: providing the natural language text in the unstructured document … to dynamically determine different topics within the natural language text and organize the natural language text into the plurality of blocks of natural language text, each respective block of natural text corresponding to a different topic (¶[0014] teaches partitioning scientific articles into units of text that represent different contexts).
While Neerukonda discloses providing the natural language text in the unstructured document … to dynamically determine different topics within the natural language text, they do not disclose providing the unstructured document as an input to a machine learning model.
Lee discloses:
providing the natural language text in the unstructured document as an input to a machine learning model configured to dynamically determine different topics within the natural language text and organize the natural language text into the plurality of blocks of natural language text … (Figure 1-2, and page 13014, figure 1 caption “entities within the summary are usually related by a common topic and fact.” and page 13016, left column, first paragraph, last 3 lines “We used the BARTbase model … trained on the CNN/Daily Mail dataset … as the summarization model.” Teaches providing unstructured document as an input to machine learning BARTbase model to determine different topics within the text and organize by blocks of summarized text).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Neerukonda to include providing the natural language text in the unstructured document as an input to a machine learning model configured to dynamically determine different topics within the natural language text and organize the natural language text into the plurality of blocks of natural language text, based on the teachings of Lee. One of ordinary skill in the art would have been motivated to make this modification in order to improve the quality of the question-answer pairs, as suggested by Lee (page 13015, left column, paragraph 1, lines 17-18).
Regarding claim 10, Neerukonda, in view of Lee, discloses the method of claim 1 (and thus the rejection of claim 1 is incorporated). Neerukonda further discloses:
wherein the plurality of question-answer pair generation models comprise a plurality of large language models (LLMs) (¶[0029]),
wherein the method further comprises: providing one or more prompts to each of the plurality of LLMs, the one or more prompts instructing each of the plurality of LLMs to generate the plurality of question-answer pairs from the first block of natural language text (¶[0027] teaches providing prompts to the LLMs to generate the pairs from text),
the one or more prompts further instructing each of the plurality of LLMs to generate the one or more … metrics for each question-answer pair (Fig. 3 and ¶[0027] teaches further instructing generating metrics for each pair).
While Neerukonda discloses the one or more prompts further instructing each of the plurality of LLMs to generate the one or more…metrics for each question-answer pair, they do not disclose this metric to be a confidence metric.
Lee discloses:
generate the one or more confidence metrics for each question-answer pair … (Figure 2 and Figure 2 caption teaches generating one or more confidence scores for each question answer pair).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Neerukonda to include generate the one or more confidence metrics for each question-answer pair, based on the teachings of Lee. One of ordinary skill in the art would have been motivated to make this modification in order to improve the quality of the question-answer pairs, as suggested by Lee (page 13015, left column, paragraph 1, lines 17-18).
Regarding claim 11, Neerukonda, in view of Lee, discloses the method of claim 1 (and thus the rejection of claim 1 is incorporated). Neerukonda further discloses:
wherein the plurality of question-answer pair generation models comprise a non-LLM model and a plurality of LLMs (¶[0028-0029] teaches the question answer generation models to be comprise combinations of non-LLM models with LLMs).
Claims 12 and 20 are substantially similar to claim 1, and thus are rejected on the same basis as claim 1.
Claims 13 is substantially similar to claim 2, and thus are rejected on the same basis as claim 2.
Claims 17 is substantially similar to claim 7, and thus are rejected on the same basis as claim 7.
Claims 3 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Neerukonda et al. (Pub. No.: US 2025/0156644A1), hereafter Neerukonda, in view of Lee et al. ("LIQUID: A Framework for List Question Answering Dataset Generation"), hereafter Lee, in further view of Chu et al. (Pub. No.: CN 109933661 A), hereafter Chu.
Regarding claim 3, Neerukonda, in view of Lee, discloses the method of claim 1 (and thus the rejection of claim 1 is incorporated). Neerukonda, in view of Lee, do not disclose:
wherein the filtering comprises: providing the plurality of question-answer pairs generated by each of the plurality of question-answer pair generation models to an embedding model configured to generate a plurality of embeddings, each of the plurality of embeddings corresponding to a different question-answer pair ,
identifying a plurality of groups of question-answer pairs based on the plurality of embeddings; and
removing one or more question-answer pairs that are not included in any of the plurality of groups of question-answer pairs.
Chu discloses:
wherein the filtering comprises: providing the plurality of question-answer pairs generated by each of the plurality of question-answer pair generation models to an embedding model configured to generate a plurality of embeddings, each of the plurality of embeddings corresponding to a different question-answer pair (Page 9, last paragraph - page 10, first paragraph “Further, the Sequence-based AC-DC Sequence model reference " all you Attention is need> " used in the method, the parameter is set to the number of multi-head, encoder is set to 8 and the decoder layer is 6, the word vector further, input end of the splicing position of the model using pre-training vector of the words, Further, using the word2vec vector training word, word vector dimension is set to 100. ” teaches providing question answer pairs to an embedding model, i.e. a question-answer pair evaluation model, each of the plurality of embeddings corresponding to a different question-answer pair),
identifying a plurality of groups of question-answer pairs based on the plurality of embeddings; and removing one or more question-answer pairs that are not included in any of the plurality of groups of question-answer pairs (page 10, third paragraph “using DBSCAN algorithm to perform the clustering, filtering the outlier and contains less question-answer pair cluster, by filtering outlier and comprises fewer question and answer pair cluster, can filter the question frequency is low and the quality is not high, and the finally obtained meets the condition of high quality question-answer pair” teaches identifying a plurality of groups and removing outlier pairs not in the groups).
Neerukonda, Lee, and Chu are analogous art because they are from the same field of endeavor: question answer pairs and machine learning models.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Neerukonda, in view of Lee, to include wherein the filtering comprises: providing the plurality of question-answer pairs generated by each of the plurality of question-answer pair generation models to an embedding model configured to generate a plurality of embeddings, each of the plurality of embeddings corresponding to a different question-answer pair, identifying a plurality of groups of question-answer pairs based on the plurality of embeddings; and removing one or more question-answer pairs that are not included in any of the plurality of groups of question-answer pairs, based on the teachings of Chu. One of ordinary skill in the art would have been motivated to make this modification in order to save cost and improve the efficiency, as suggested by Chu (page 13, paragraph 7).
Claims 14 is substantially similar to claim 3, and thus are rejected on the same basis as claim 3.
Claims 8, 9, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Neerukonda et al. (Pub. No.: US 2025/0156644A1), hereafter Neerukonda, in view of Lee et al. ("LIQUID: A Framework for List Question Answering Dataset Generation"), hereafter Lee, in further view of Jia et al. (“Adversarial Examples for Evaluating Reading Comprehension Systems”), hereafter Jia.
Regarding claim 8, Neerukonda, in view of Lee, discloses the method of claim 1 (and thus the rejection of claim 1 is incorporated). Neerukonda further discloses:
…a question in a question-answer pair generated by a question-answer pair generation model of the plurality of question-answer pair generation models to be an unanswerable question for the generative language processing machine learning model (¶[0035] teaches negative samples as unanswerable questions from the generated pairs),
… answer is consistent with an answer the generative language processing machine learning model generates in response to unanswerable questions (¶[0035] teaches the synthetic QA pair to contain an answer for the unanswerable question),
… dataset for the domain-specific hallucination testing… (¶[0034] teaches the dataset for the domain-specific hallucination testing).
While Neerukonda discloses …a question in a question-answer pair generated by a question-answer pair generation model of the plurality of question-answer pair generation models to be an unanswerable question for the generative language processing machine learning model, they do not teach modifying a question to be an unanswerable question.
Jia discloses:
modifying a question … to be an unanswerable question (Figure 2, element “Addsent” teaches modifying a question to be unanswerable in step 1, where the question is unanswerable due to the mutation from “Tesla” to “Tadakatsu” with respect to the given passage).
While Neerukonda discloses … answer is consistent with an answer the generative language processing machine learning model generates in response to unanswerable questions, they do not teach modifying an answer … so that the modified answer is consistent with an answer … in response to unanswerable questions.
Jia discloses:
modifying an answer … so that the modified answer is consistent with an answer … in response to unanswerable questions (Figure 2, element “Addsent” teaches modifying the answer to be consistent with the unanswerable question in step 2).
While Neerukonda discloses … dataset for the domain-specific hallucination testing…, they do not teach dataset … includes … the unanswerable question and the modified answer.
Jia discloses:
dataset … includes … the unanswerable question and the modified answer (Figure 2, element “Addsent” and Table 1-2 teaches the datasets for the models to include the unanswerable question and the modified answer generated by Addsent).
Neerukonda, Lee, and Jia are analogous art because they are from the same field of endeavor: question answer pairs and machine learning models.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Neerukonda, in view of Lee, to include modifying a question … to be an unanswerable question, modifying an answer … so that the modified answer is consistent with an answer … in response to unanswerable questions, and dataset … includes … the unanswerable question and the modified answer, based on the teachings of Jia. One of ordinary skill in the art would have been motivated to make this modification for development of new models that understand language more precisely, as suggested by Jia (page 2, left column, paragraph 2, last 3 lines).
Regarding claim 9, Neerukonda, in view of Lee, in further view of Jia, discloses the method of claim 8 (and thus the rejection of claim 8 is incorporated). Jia further discloses:
wherein modifying the question in the question-answer pair to be an unanswerable question comprises: replacing a word in the question that corresponds to an entity with a word that corresponds to a different entity (Figurer 2 element “Addsent”),
replacing one or more words in the question with antonyms of the one or more words (page 3, right column, paragraph 5, lines 3-5 “We replace nouns and adjectives with antonyms from WordNet”).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Neerukonda, in view of Lee, to include wherein modifying the question in the question-answer pair to be an unanswerable question comprises: replacing a word in the question that corresponds to an entity with a word that corresponds to a different entity, replacing one or more words in the question with antonyms of the one or more words, based on the teachings of Jia. One of ordinary skill in the art would have been motivated to make this modification for development of new models that understand language more precisely, as suggested by Jia (page 2, left column, paragraph 2, last 3 lines).
Claims 18 is substantially similar to claim 8, and thus are rejected on the same basis as claim 8.
Claims 19 is substantially similar to claim 9, and thus are rejected on the same basis as claim 9.
Claims 4, 5, 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Neerukonda et al. (Pub. No.: US 2025/0156644A1), hereafter Neerukonda, in view of Lee et al. ("LIQUID: A Framework for List Question Answering Dataset Generation"), hereafter Lee, in further view of Chu et al. (Pub. No.: CN 109933661 A), hereafter Chu, in further view of Hauer et al. (“UAlberta at SemEval-2020 Task 2: Using Translations to Predict Cross-Lingual Entailment”), hereafter Hauer.
Regarding claim 4, Neerukonda, in view of Lee, in further view of Chu, discloses the method of claim 3 (and thus the rejection of claim 3 is incorporated). Neerukonda further discloses:
providing, for each of the question-answer pairs included a group of the plurality of groups, a respective answer for each the question-answer pairs as an input to an entailment model configured to determine an entailment between two different answers in the group (¶[0036] teaches providing an answer of whether the pair was extracted from a corresponding source to an entailment model),
receiving, from the entailment model, the entailment between two different answers (¶[0036]),
comparing the entailment … and determining whether the two different answers are consistent based on the comparing (¶[0036] teaches the entailment filter model to compare the synthetic answers with others in the dataset, and finds the different answers are consistent based on the comparing if they are from the same source).
Neerukonda teaches comparing the entailment … and determining whether the two different answers are consistent based on the comparing, but does not explicitly disclose comparing to a similarity threshold.
Hauer discloses:
comparing the entailment to a similarity threshold (Figure 1 and section 3.2, paragraph 2 “the cosine similarity threshold for deciding semantic similarity. We tune the threshold on the official trial data set of each language pair, if such a set is provided… we instead adopt a threshold value of 0.25…” teaches comparing the entailment to a similarity threshold during tuning).
Neerukonda, Lee, Chu, and Hauer are analogous art because they are from the same field of endeavor: question answer pairs and machine learning models.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Neerukonda, in view of Lee, in view of Chu, to include comparing the entailment to a similarity threshold, based on the teachings of Hauer. One of ordinary skill in the art would have been motivated to make this modification for improvements in identifying entailment, as suggested by Hauer (page 268, conclusion paragraph, lines 3-4).
Regarding claim 5, Neerukonda, in view of Lee, in further view of Chu, in further view of Hauer, discloses the method of claim 4 (and thus the rejection of claim 4 is incorporated). Neerukonda further discloses:
wherein determining whether the two different answers are consistent based on the comparing comprises: determining the two different answers are not consistent based on the comparing; and removing the group of question-answer pairs (¶[0036] teaches filtering, i.e., removing, the question answer pairs if they are not found to be consistent).
Claims 15 is substantially similar to claim 4, and thus are rejected on the same basis as claim 4.
Claims 16 is substantially similar to claim 5, and thus are rejected on the same basis as claim 5.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's
disclosure.
U.S. Pub No. US 12124932 B1: Poulis et al. teaches question answer pair generation.
U.S. Pub No. US 20140072948 A1: Boguraev et al. teaches question answer pair generation.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUMAIRA ZAHIN MAUNI whose telephone number is (703)756-5654. The examiner can normally be reached Monday - Friday, 9 am - 5 pm (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MATT ELL can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.Z.M./Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights