Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Application 19/229,207 filed 6/5/2025 (with foreign priority to JP2024-093485 with Filing Date 06/10/2024) has been examined.
In this Office Action, Claims 1-5 are currently pending.
Examiner’s Note:
Claim 1 recites:
“a context information retrieving unit configured to retrieve a document database with a characteristic vector of the question and thereby acquire as context information a text
that a similarity level between the characteristic vector and a characteristic vector of the text satisfies a predetermined condition”;
However, it appears it was Applicant’s intent to rather claim:
“a context information retrieving unit configured to retrieve from a document database [[with]] using a characteristic vector of the question and thereby acquire as context information a text from the document database that has a similarity level between the characteristic vector of the question and a characteristic vector of the text satisfies a predetermined condition”;
Additionally, Claim 1 recites:
“a second prompt generating unit configured to generate a second prompt that includes the context information and the first answer and cause to provide a confidence of the first answer;”
However, it appears it was Applicant’s intent to rather claim:
“a second prompt generating unit configured to generate a second prompt that includes the context information and the first answer and causes the second prompt generating unit to provide a confidence of the first answer;”
Appropriate clarification/correction is required.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-5 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-4 of copending Application No. 19/296,257. Although the claims at issue are not identical, they are not patentably distinct from each other because
claim 1 is generic to all that is recited in claim 1 of U.S. Patent App. No. 19/296,257
That is, claim 1 of U.S. Patent App. No. 19/296,257 falls entirely within the scope of claim 1 or, in other words, claim 1 is anticipated by claim 1 of U.S. Patent App. No. 19/296,257.
Specifically, because instant claim 1 recites: " a context information retrieving unit configured to retrieve a document database with a characteristic vector of the question and thereby acquire as context information a text that a similarity level between the characteristic vector and a characteristic vector of the text satisfies a predetermined condition; " this limitation is/are a species of the generic category defined by “a context information retrieving unit configured to retrieve a document database with a characteristic vector of the question and thereby acquire as context information a text group that a similarity level between the characteristic vector and a characteristic vector of the text group satisfies a predetermined condition” (see claim 1, U.S. Patent App. No. 19/296,257) the process of claim 1 reciting “a context information retrieving unit configured to retrieve a document database with a characteristic vector of the question and thereby acquire as context information a text that a similarity level between the characteristic vector and a characteristic vector of the text satisfies a predetermined condition; ” is anticipated by claim 1 of U.S. Patent App. No. 19/296,257 reciting “a context information retrieving unit configured to retrieve a document database with a characteristic vector of the question and thereby acquire as context information a text group that a similarity level between the characteristic vector and a characteristic vector of the text group satisfies a predetermined condition”.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Current Application
App. 19/296,257
1. An information retrieval system that provides an answer corresponding to a question using a large language model, comprising:
a question receiving unit configured to receive the question;
a context information retrieving unit configured to retrieve a document database with a characteristic vector of the question and thereby acquire as context information a text that a similarity level between the characteristic vector and a characteristic vector of the text satisfies a predetermined condition;
a first prompt generating unit configured to generate a first prompt that includes the question and the context information;
an answer acquiring unit configured to acquire as a first answer an answer corresponding to the first prompt using a large language model;
a second prompt generating unit configured to generate a second prompt that includes the context information and the first answer and cause to provide a confidence of the first answer;
and an answer verifying unit configured to acquire as a second answer an answer corresponding to the second prompt using a large language model, and determine the confidence with the second answer.
As to claim 2. The information retrieval system according to claim 1, further comprising an answer outputting unit configured
(a) to determine whether an answer that includes the first answer and the confidence should be outputted as an answer to the question or not on the basis of the confidence,
(b) if it is determined that an answer that includes the first answer and the confidence should be outputted as an answer to the question, to output an answer that includes the first answer and the confidence, and if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, not to output an answer that includes the first answer.
As to claim 3. The information retrieval system according to claim 2, wherein if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, the answer outputting unit outputs an answer that includes a warning message as an answer to the question.
As to claim 4. The information retrieval system according to claim 3, wherein if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, the answer outputting unit outputs an answer that includes the warning message and the context information as an answer to the question.
As to claim 5. The information retrieval system according to claim 1, wherein the confidence is expressed as a confidence level that is a normalized numeral value.
1. An information retrieval system that provides an answer corresponding to a question using a large language model, comprising:
a question receiving unit configured to receive the question;
a context information retrieving unit configured to retrieve a document database with a characteristic vector of the question and thereby acquire as context information a text group that a similarity level between the characteristic vector and a characteristic vector of the text group satisfies a predetermined condition,
the document database in which text groups obtained by dividing a document, character vectors of the text groups and page numbers of the text groups are registered such that the text groups, the characteristic vectors, and page numbers are associated with each other;
a prompt generating unit configured to generate a prompt that includes the question and the context information;
an answer acquiring unit configured to acquire an answer corresponding to the prompt using a large language model;
and an answer outputting unit configured to output as an answer corresponding to the question the context information and a page number associated with the context information together with the answer corresponding to the prompt.
2. The information retrieval system according to claim 1, further comprising a document registering unit configured to (a) divide the document and thereby generate the text group, (b) determine a page number of the text group on the basis of the document, (c) derive a characteristic vector of the text group, and (d) register the generated text group, the determined page number, and the derived characteristic vector to the document database SO as to associate text group, the page number, and the characteristic vector with each other.
3. The information retrieval system according to claim 1, wherein the document database comprises a first database, and a second database, in the first database, text groups obtained by dividing the document page by page, characteristic vectors of these text groups, and page numbers of the text groups are registered SO as to associate these text groups, these characteristic vectors, and these page numbers with each other, respectively, and in the second database, text groups obtained by dividing each page of the document such that each of the text groups has a predetermined number of characters, and characteristic vectors of these text groups are registered so as to associate these text groups and these characteristio vectors with each other, respectively ; the context information retrieving unit retrieves the second database with a characteristio vector of the question and thereby acquires the context information a text group of which a similarity level to this characteristic vector satisfies a predetermined condition; and the page number is determined on the basis of the first database.
4. The information retrieval system according to claim 1, wherein the answer outputting unit displays as an answer corresponding to the question the context information and the page number associated with the context information in a single screen on a predetermined display device so as to associate the context information and the page number with each other.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “predetermined condition” in claim 1, is a relative term which renders the claim indefinite. The term “predetermined condition” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5 are rejected under 35 U.S.C. 101 because the claimed invention is
directed to an abstract idea without significantly more.
Note: the Examiner notes that the claimed “information retrieval system” is defined as a processor/ computer device see generally specification para. [0012] “The information retrieval system 1 shown in Fig. 1 is an information retrieval system that provides an answer corresponding to a question using a large language model 4a, and includes a processor 11 as a computer, a communication device 12, and a storage device 13. Here, the information retrieval system 1 is installed in a single computer device, and alternatively, may be dispersedly installed in plural computer devices”.
Claim 1 recites: (Step 2a, Prong One)
an answer acquiring unit configured to acquire as a first answer an answer corresponding to the first prompt using a large language model.
The limitation of an answer acquiring unit configured to acquire as a first answer an answer corresponding to the first prompt using a large language model, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting a generic information retrieval system, and a generic “large language model” nothing in the claim element precludes the step from practically being performed in the mind. For example, but for the information retrieval system/large language model language, “acquiring” in the context
of this claim encompasses the user manually determining generic “answers” using
generic “acquiring” of generic answers using generic prompts and generic “large language models” steps. Additionally, note the recent and relevant decision Recentive Analytics, Inc. v. Fox Corp. (CAFC Case: 23-2437) explaining that “we hold only that patents that do no more than claim the application of generic machine learning to new data environments, without disclosing improvements to the machine learning models to be applied, are patent ineligible under § 101”). Similarly, the limitation(s) of receiving; retrieving; generating; generating; verifying; and determining, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the
recitation of generic computer components. For example, but for the information retrieval system/large language model language, retrieving; retrieving; generating; generating; verifying; and determining in the context of this claim encompasses the user manually receiving generic “answers” and performing generic “generating”; “verifying” steps. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas (concepts performed in the human mind (including an observation, evaluation, judgment, opinion)).
Further, these concepts also recite “Certain Methods of Organizing Human Activity”; (such as
commercial or legal interactions (including agreements in the form of contracts; legal
obligations; advertising, marketing or sales activities or behaviors; business relations) where
performing generic generating of answers using generic “generating”; “verifying” and “acquiring” steps is a method of human activity in commercial or legal interactions.
Accordingly, the claim recites an abstract idea.
(Step 2a, Prong Two)
This judicial exception is not integrated into a practical application. In particular, the claim only
recites one additional element – using an information retrieval system/large language model to perform both the of receiving; retrieving; generating; generating; verifying; and determining and acquiring steps. The information retrieval system/large language model in both steps is recited at a high level of generality (i.e., as a generic processor performing a generic computer function of “acquiring”) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
(Step 2b)
The claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception. As discussed above with respect to integration of the abstract
idea into a practical application, the additional element of an information retrieval system/large language model to perform both the receiving; retrieving; generating; generating; verifying; and determining and acquiring steps amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim(s) is/are not patent eligible.
Referring to claim 2, (Step 2a, Prong One) this further merely performs an additional abstract
mental step of “further comprising an answer outputting unit configured (a) to determine whether an answer that includes the first answer and the confidence should be outputted as an answer to the question or not on the basis of the confidence, (b) if it is determined that an answer that includes the first answer and the confidence should be outputted as an answer to the question,
to output an answer that includes the first answer and the confidence, and if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, not to output an answer that includes the first answer”.
(Step 2a, Prong Two)
This judicial exception is not integrated into a practical application. In particular, the claim only
recites the additional elements of “further comprising an answer outputting unit configured (a) to determine whether an answer that includes the first answer and the confidence should be outputted as an answer to the question or not on the basis of the confidence, (b) if it is determined that an answer that includes the first answer and the confidence should be outputted as an answer to the question, to output an answer that includes the first answer and the confidence, and if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, not to output an answer that includes the first answer” steps to perform both the aforementioned receiving; retrieving; generating; generating; verifying; and determining and acquiring steps. Accordingly, this
additional element does not integrate the abstract idea into a practical application because it
does not impose any meaningful limits on practicing the abstract idea.
(Step 2b)
The claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception. As discussed above with respect to integration of the abstract
idea into a practical application, the additional element of using “further comprising an answer outputting unit configured (a) to determine whether an answer that includes the first answer and the confidence should be outputted as an answer to the question or not on the basis of the confidence, (b) if it is determined that an answer that includes the first answer and the confidence should be outputted as an answer to the question, to output an answer that includes the first answer and the confidence, and if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, not to output an answer that includes the first answer” steps to perform both the aforementioned receiving; retrieving; generating; generating; verifying; and determining and acquiring steps amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept.
The claim(s) is/are not patent eligible.
Referring to claim 3, (Step 2a, Prong One) this further merely performs an additional abstract
mental step of “wherein if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, the answer outputting unit outputs an answer that includes a warning message as an answer to the question”.
(Step 2a, Prong Two)
This judicial exception is not integrated into a practical application. In particular, the claim only
recites the additional elements of “wherein if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, the answer outputting unit outputs an answer that includes a warning message as an answer to the question” steps to perform both the aforementioned receiving; retrieving; generating; generating; verifying; and determining and acquiring steps. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
(Step 2b)
The claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception. As discussed above with respect to integration of the abstract
idea into a practical application, the additional element of using “wherein if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, the answer outputting unit outputs an answer that includes a warning message as an answer to the question” steps to perform both the aforementioned receiving; retrieving; generating; generating; verifying; and determining and acquiring steps amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept.
The claim(s) is/are not patent eligible.
Referring to claim 4, (Step 2a, Prong One) this further merely performs an additional abstract
mental step of “wherein if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, the answer outputting unit outputs an answer that includes the warning message and the context information as an answer to the question”.
(Step 2a, Prong Two)
This judicial exception is not integrated into a practical application. In particular, the claim only
recites the additional elements of “wherein if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, the answer outputting unit outputs an answer that includes the warning message and the context information as an answer to the question” steps to perform both the aforementioned receiving; retrieving; generating; generating; verifying; and determining and acquiring steps. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
(Step 2b)
The claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception. As discussed above with respect to integration of the abstract
idea into a practical application, the additional element of using “wherein if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, the answer outputting unit outputs an answer that includes the warning message and the context information as an answer to the question” steps to perform both the aforementioned receiving; retrieving; generating; generating; verifying; and determining and acquiring steps amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept.
The claim(s) is/are not patent eligible.
Referring to claim 5, (Step 2a, Prong One) this further merely performs an additional abstract
mental step of “wherein the confidence is expressed as a confidence level that is a normalized numeral value”.
(Step 2a, Prong Two)
This judicial exception is not integrated into a practical application. In particular, the claim only
recites the additional elements of “wherein the confidence is expressed as a confidence level that is a normalized numeral value” steps to perform both the aforementioned receiving; retrieving; generating; generating; verifying; and determining and acquiring steps. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
(Step 2b)
The claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception. As discussed above with respect to integration of the abstract
idea into a practical application, the additional element of using “wherein the confidence is expressed as a confidence level that is a normalized numeral value” steps to perform both the aforementioned receiving; retrieving; generating; generating; verifying; and determining and acquiring steps amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept.
The claim(s) is/are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sasaki et al., JP7325152 B1, in view of Malkiel et al., US Pub. No. 2025/0238629 A1.
As to claim 1, Sasaki discloses
an information retrieval system that provides an answer corresponding to a question using a large language model,
(Sasaki p. 4: “The LLM 1 is connected to a user terminal 200 via a communication network 300 such as the Internet or a mobile phone network, inputs a question from the user terminal 200 , generates an answer to the question, and outputs the answer to the user terminal 200”)
comprising:
a question receiving unit configured to receive the question;
(Sasaki p. 4: “The LLM 1 is connected to a user terminal 200 via a communication network 300 such as the Internet or a mobile phone network, inputs a question from the user terminal 200, generates an answer to the question, and outputs the answer to the user terminal 200”)
a context information retrieving unit configured to retrieve a document database with a characteristic vector of the question and thereby acquire as context information a text that a similarity level between the characteristic vector and a characteristic vector of the text satisfies a predetermined condition;
(Sasaki teaches a vector search unit searches the text DBs using the feature vector calculated by the vector calculation unit for the input question text and similarity comparisons, i.e. “retrieve a document database with a characteristic vector of the question and thereby acquire as context information a text that a similarity level”
See p. 3-4: “In order to solve the above-described problems, in the present invention, another question sentence is generated from an input question sentence, and a sentence using a feature vector calculated from the input question sentence and the generated another question sentence By searching the database, candidates are used to generate additional sentences that add sentences associated with feature vectors that satisfy predetermined conditions regarding similarity to feature vectors as reference information to an input question sentence.”;
See also p. 7: “The vector calculation unit 111c calculates the feature vector of the text from the question text input from the user terminal 200. FIG. For example, the vector calculation unit 111c has a small-scale language model of an embedding model, and uses the embedding model to convert an input question text into a numerical vector.”;
See also p. 7:” In order to apply the second candidate sentence acquisition method, feature vectors are also calculated in advance for the plurality of sentences recorded in the sentence DBs 2a to 2c, and the plurality of sentences are recorded in association with the respective feature vectors in advance. The vector search unit 111d searches the text DBs 2a to 2c using the feature vector calculated by the vector calculation unit 111c for the input question text, and obtains the similarity with the feature vector calculated by the vector calculation unit 111c. Sentences associated with feature vectors that satisfy a predetermined condition are obtained as candidate sentences from the sentence DBs 2a to 2c.”)
a first prompt generating unit configured to generate a first prompt that includes the question and the context information;
(Sasaki teaches a question text generation unit / configures a prompt by adding an example question text to an input question text, i.e. a “prompt generating unit configured to generate a first prompt that includes the question and the context information”
See p. 8: “The question text generation unit 111 e generates another question text from the question text input from the user terminal 200. In addition to the feature vector of the question text input from the user terminal 200, the vector calculation unit 111c calculates the feature vector of another question text generated by the question text generation unit 111e. The vector search unit 111d searches the text DBs2a to 2c using the feature vector calculated from the question text input from the user terminal 200 and the feature vector calculated from another question text generated by the question text generation unit111e. to obtain candidate sentences.
Here, the question text generation unit 111e includes a small-scale language model generated by applying Few-shot learning, for example, and configures a prompt by adding an example question text to an input question text, and generates a small-scale language model. By inputting to the model, another question sentence is output from the small-scale language model. Exemplary question sentences are acquired from the exemplary question sentence DB 2d. FIG. 7 is a diagram showing an example of a prompt and another question sentence generated by the candidate sentence acquisition unit 111 in this case. In the example of FIG. 7, a prompt is configured by adding an exemplary question sentence extracted from the exemplary question sentence DB 2d using "Okinawa trip" included in the question sentence as a keyword to the input question sentence, and It generates another question as indicated by the question.”;
See also p. 8: “FIG. 8 is a diagram showing an example of a prompt and another question sentence generated by the candidate sentence acquisition unit 111 in this case. In the example of FIG. 8, a plurality of exemplary question sentences that match the input question sentence are extracted from the exemplary question sentence DB 2d, and a question generation example that serves as a reference is generated from the se to compose a prompt”)
an answer acquiring unit configured to acquire as a first answer an answer corresponding to the first prompt using a large language model;
(Sasaki p. 4: “The LLM 1 is connected to a user terminal 200 via a communication network 300 such as the Internet or a mobile phone network, inputs a question from the user terminal 200 , generates an answer to the question, and outputs the answer to the user terminal 200”;
See also p. 5: “The text generation unit 10 of the present embodiment generates a prompt for input to the LLM 1 by adding reference information to the question text input from the user terminal 200 . A prompt for input to the LLM 1 is specifically a prompt for input to the answer generator 20 . That is, the text generation unit 10 generates an additional text based on the question text input from the user terminal 200,generates a text with the additional text added as reference information for the question text, and responds with this text as a prompt. Input to the generation unit 20.
The answer generation unit 20 receives the prompt generated by the text generation unit 10, and generates an answer text to the question text included in the prompt by using the additional text included in the prompt as reference information. That is, the answer generation unit 20 generates an answer sentence with a higher degree of conformity to the reference information.”)
Sasaki does not disclose:
a second prompt generating unit configured to generate a second prompt that includes the context information and the first answer and cause to provide a confidence of the first answer;
and an answer verifying unit configured to acquire as a second answer an answer corresponding to the second prompt using a large language model, and determine the confidence with the second answer;
However, Malkiel discloses:
a second prompt generating unit configured to generate a second prompt that includes the context information and the first answer and cause to provide a confidence of the first answer;
(Malkiel teaches a backward prompt, each backward prompt including at least one answer-question pair and the primary answer, and determining similarity measurement/ hallucination extent, i.e. “a second prompt generating unit configured to generate a second prompt that includes the context information and the first answer and cause to provide a confidence of the first answer”
See [0007] In response to the forward prompt, this example embodiment obtains a primary
answer. Then this embodiment submits at least one backward prompt, each backward prompt including at least one answer-question pair and the primary answer, the answer-question pair derived from the question-answer pair by at least a pair component order reversal of the question answer pair question relative to the question-answer pair corresponding answer, the primary answer not accompanied in the backward prompt by the primary question. In response
to the backward prompt, the embodiment acquires at least one candidate question. The embodiment procures a primary question embedding vector from the primary question, and
procures a respective candidate question embedding vector from each candidate question. Then the embodiment calculates a vector similarity measurement between the primary
question embedding vector and at least one candidate question embedding vector, and assigns a hallucination extent to the primary answer based on at least the vector similarity
measurement. One or more computational actions are then taken, based on the hallucination extent.)
and an answer verifying unit configured to acquire as a second answer an answer corresponding to the second prompt using a large language model, and determine the confidence with the second answer
(Malkiel teaches a InterrogateLLM/language model hallucination detection for checking a similarity score to determine a different answer/acceptable answers from the primary answer by checking hallucination extents of answers and repeating the answer determination process, i.e. “answer verifying unit configured to acquire as a second answer an answer corresponding to the second prompt using a large language model, and determine the confidence with the second answer”
See [0028] This language model hallucination detection functionality has several technical benefits. First, it can be used together with model output veracity verification based on
separately retrieved context,
see also [0030] a processor is configured to ascertain, based on at least a comparison of a hallucination extent to a threshold, that a primary answer from a forward language model
interface is unacceptable, and in response to prompt through the forward language model interface for a different answer. This has the technical benefit of improving the accuracy of
language model outputs. Improving outputs accuracy helps increase confidence in the outputs;
see also [0388] 910 computationally prompt a model 412 for a different answer, e.g., by repeating part (steps 802 and 804) or all of method 800 or method 900 when an
obtained 804 answer or its subsequent potential replacement is not acceptable;
see also [0084] An answer produced by a model is acceptable, e.g., when the hallucination extent associated with the answer is less than or equal a specified threshold; otherwise the answer is unacceptable.
See also [0169] Some embodiments which utilize an InterrogateLLM approach check whether the similarity score exceeds a predetermined threshold "t (also referred to her as tau) 456.
In essence, when the reconstructed queries exhibit a significant divergence from the original query, these InterrogateLLM embodiments report a conclusion that there is a potential hallucination in A* without necessarily indicating the strength of that conclusion.
see also Malkiel [0083] In another scenario, the system withholds 908 the unacceptable answer and goes into a loop 1022 that seeks a better answer. One such loop includes prompting 910 for an answer, testing that answer for acceptability via backward traversal 304 to acquire 808 candidate questions, embedding 1002 questions into vectors, measuring vector distance 438,
and threshold comparison 1014, as discussed elsewhere herein. These systems exit the loop 1022 after (a) obtaining an acceptable answer (or in some cases, obtaining an answer that is at least better because the answer's distance 438 is smaller), or (b) reaching a defined limit on looping;
see also [0079-0080] [0079] In some embodiments, the computing system 202 is configured to ascertain 1018, computed from at least a comparison 1014 of the hallucination extent 216 to a threshold 456, that the primary answer 408 is unacceptable ( e.g., acceptability status 458 is "not acceptable"). Depending on the embodiment or configuration or both, additional computational actions then occur in response to the answer's non-acceptability. [0080] In some cases, the system 202 withholds 908 the unacceptable primary answer from the user interface.;
see also [0125] In some embodiments, the method 1000 includes ascertaining 1018, computed from at least the hallucination extent and a threshold, that the primary answer is unacceptable;
and in response to the ascertaining 1018, prompting 910 the first language model for another answer to the primary question, or prompting 910 the first language model for an answer to a substitute question 462, wherein the substitute question is computationally gleaned 1012 from
the primary question or the substitute question is semantically interchangeable 470 with the primary question, or both.).
It would have been obvious to one having ordinary skill in the art at the time of the effective filing date to apply hallucination detection for checking a similarity score to determine a different answer/acceptable answers, since it was known in the art that language model systems provide utilize or provide language model hallucination detection technology wherein a processor is configured to ascertain, based on at least a comparison of a hallucination extent to a threshold,
that a primary answer is unacceptable, and in response to withhold the primary answer from a user interface where this has the technical benefit of improving the accuracy of the group
of language model outputs that are presented via the user interface, where this provides for improving outputs accuracy and helps increase confidence in the outputs and in the underlying model-based technology that creates the outputs, which in turn promotes wider and more varied use of language models in different problem domains, with accompanying productivity and ease-of-use benefits. (Malkiel [0029]).
As to claim 2, Malkiel as modified discloses the information retrieval system according to claim 1, further comprising an answer outputting unit configured
(a) to determine whether an answer that includes the first answer and the confidence should be outputted as an answer to the question or not on the basis of the confidence,
(Malkiel teaches outputting answers/ hallucination extent based on thresholds, i.e. “(a) to determine whether an answer that includes the first answer and the confidence should be outputted as an answer to the question or not on the basis of the confidence”
See [0084] An answer produced by a model is acceptable, e.g., when the hallucination extent associated with the answer is less than or equal a specified threshold; otherwise the answer is unacceptable.;
see also [0386] 906 computationally report a hallucination extent, e.g., via a user interface, a device control interface, or another API;
See also [0029] Some embodiments described herein utilize or provide language model hallucination detection technology wherein a processor is configured to ascertain, based on at
least a comparison of a hallucination extent to a threshold, that a primary answer is unacceptable, and in response to withhold the primary answer from a user interface. This has
the technical benefit of improving the accuracy of the group of language model outputs that are presented via the user interface. Improving outputs accuracy helps increase confidence in the outputs and in the underlying model-based technology that creates the outputs, which in turn promotes wider and more varied use of language models in different problem domains, with accompanying productivity and ease-of-use benefits.;
see also [0080] In some cases, the system 202 withholds 908 the unacceptable primary answer from the user interface.;)
(b) if it is determined that an answer that includes the first answer and the confidence should be outputted as an answer to the question, to output an answer that includes the first answer and the confidence,
(See Malkiel [0084] An answer produced by a model is acceptable, e.g., when the hallucination extent associated with the answer is less than or equal a specified threshold; otherwise the answer is unacceptable.;
see also [0386] 906 computationally report a hallucination extent, e.g., via a user interface, a device control interface, or another API;)
and if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, not to output an answer that includes the first answer.
(Malkiel teaches withholding answers/hallucination extents based on thresholds
i.e. “if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, not to output an answer that includes the first answer”see [0084] An answer produced by a model is acceptable, e.g., when the hallucination extent associated with the answer is less than or equal a specified threshold; otherwise the answer is unacceptable.;
See also [0029] Some embodiments described herein utilize or provide language model hallucination detection technology wherein a processor is configured to ascertain, based on at
least a comparison of a hallucination extent to a threshold, that a primary answer is unacceptable, and in response to withhold the primary answer from a user interface. This has
the technical benefit of improving the accuracy of the group of language model outputs that are presented via the user interface. Improving outputs accuracy helps increase confidence in the outputs and in the underlying model-based technology that creates the outputs, which in turn promotes wider and more varied use of language models in different problem domains, with accompanying productivity and ease-of-use benefits.;
see also [0388] 910 computationally prompt a model 412 for a different answer, e.g., by repeating part (steps 802 and 804) or all of method 800 or method 900 when an
obtained 804 answer or its subsequent potential replacement is not acceptable;
see also (0387] 908 computationally withhold a hallucination extent from a user interface, e.g., by skipping an instruction 116 that passes a hallucination extent to a user interface in a situation where not skipping the instruction would supply the hallucination extent to the
user interface for visual or aural presentation)
As to claim 3, Malkiel as modified discloses the information retrieval system according to claim 2, wherein if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, the answer outputting unit outputs an answer that includes a warning message as an answer to the question
(Malkiel teaches hallucination/correctness warnings see [0082] In one scenario, the system displays the unacceptable answer instead of withholding 908 it, together with a remark such as "Here's an answer, but it may be wrong so another answer is also being generated. Please wait a moment." In some situations, the answer's distance 438, or a scaled version thereof, is also displayed as a confidence level associated with the answer 430.).
As to claim 4, Malkiel as modified discloses the information retrieval system according to claim 3, wherein if it is determined that an answer that includes the first answer and the confidence should not be outputted as an answer to the question, the answer outputting unit outputs an answer that includes the warning message and the context information as an answer to the question.
(Malkiel teaches hallucination/correctness warnings and notifications that another answer is being generated, i.e. “the answer outputting unit outputs an answer that includes the warning message and the context information as an answer to the question” see [0082] In one scenario, the system displays the unacceptable answer instead of withholding 908 it, together with a remark such as "Here's an answer, but it may be wrong so another answer is also being generated. Please wait a moment." In some situations, the answer's distance 438, or a scaled version thereof, is also displayed as a confidence level associated with the answer 430.;
See also [0184] InterrogateLLM can be used together with an extension to retrieval augmented generation (RAG) settings, where a query is provided with a retrieved-context 448, and the task is to generate an answer based at least in part on the information provided in the context).
As to claim 5, Malkiel as modified discloses the information retrieval system according to claim 1, wherein the confidence is expressed as a confidence level that is a normalized numeral value
(Malkiel teaches confidence levels, i.e. “wherein the confidence is expressed as a confidence level that is a normalized numeral value” see [0082] In some situations, the answer's distance 438, or a scaled version thereof, is also displayed as a confidence level associated with the answer 430.; see also (0341] 438 vector similarity measurement, also referred to as vector distance, e.g., a numeric value output by a metric 436 given two or more vectors; some examples include a cosine distance between two vectors, and a distance produced by computation based on Formula (7) or the MAX alternate to Formula (7);).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Onodera et al., US Pub. No. 2007/0168337 A1, teaches an apparatus acquires retrieval history data of information retrieval to be determined, and acquires voice data of the conversation heard about information necessary for the information retrieval. A predetermined keyword dictionary is referred to from the voice data, and an expression matching a keyword indicating necessary information for information retrieval is extracted, and defined as a keyword obtained before retrieval. Then, the keyword obtained before retrieval and the retrieval history data (a retrieval key, a selected document, and an order in the retrieval result of the selected document) are compared with the best case data accumulated in the best case data storage unit, the applicability of the keyword obtained before retrieval and the applicability of the retrieval key are determined, and the applicability of the information retrieving process is obtained based on the determination;
Zhao et al. US Pub. No. 2021/0406329 A1, teaches a question answering processing method and apparatus, and a storage medium is disclosed herein. The detailed implementation solution includes: receiving a speech question inputted by a user; obtaining an answer set corresponding to the speech question, the answer set including a plurality of candidate answers, and each candidate answer corresponding to a region identification; obtaining a current region identification of the user, and determining whether the answer set includes the current region identification; and in a case that the answer set includes the current region identification, obtaining a first candidate answer corresponding to the current region identification, and feeding the plurality of candidate answers sorted with the first candidate answer as a first place back to the user;
Bolcer et al., US Pub. No. 2025/0086211, teaches a system and method, using one or more processors, for grounding large language models using real-time content feeds and reference data. The system is able to receive/generate queries, prompt an AI model and evaluate the answer given by the AI model. A hallucination score is generated according to the evaluation of the answer. According to the hallucination score, the AI model may be iteratively accessed to improve the truthfulness of the answer;
Zhong et al. US Pub. No. 2025/0094464, teaches techniques are disclosed herein for selecting document chunks that are most relevant to a query. The techniques include receiving a query and comparing a plurality of stored text passages to the query using a first similarity metric. Based on the comparison, a subset of the plurality of stored text passages that are most similar to the query are selected. A plurality of sentences from the subset of the plurality of stored text passages are identified. The identified sentences are ranked based on the query and a second similarity metric. A subset of the sentences are selected based on the ranking. The subset of the sentences or a derivative thereof are output in response to the query;
Bierner et al., US Pub. No. 2025/0278423, teaches a genealogical research assistant is provided by receiving a user query at a user interface; classifying the user query using a classification module, refining the classified user query using a refinement module, vectorizing the refined, classified user query using an embeddings module; retrieving, from a vector database, a plurality of results based on the vectorized, refined, classified user query; generating, using a generative machine-learning module, a response to the user query based on the plurality of results; and displaying, at the user interface, the response. The vector database may comprise a plurality of domain-specific content the generative machine-learning module may rely upon to generate the response. The generative machine-learning module may be configured to provide in-line links to the top n results from the vector database in the response;
Agrawal et al., US Pub. No. 2025/0284721, teaches a database system integrates in-database machine learning (ML) models with in-database large language models (LLMs) or other generative artificial intelligence (AI) models that enable new applications. The database system receives one or more inferences from an ML model and provides an inference input to a retrieval agent of an object store. One or more vector stores represent a plurality of reference documents using semantic encodings. The retrieval agent performs a similarity search of the one or more vector stores to retrieve a set of passages from the plurality of reference documents based on similarity of encodings of the inference input and encodings of passages in the plurality of reference documents. The database system generates a linguistic prompt for an LLM having a context including the inferences and passages and applies the LLM to the linguistic prompt to generate a natural language explanation of the one or more inferences
Suba et al., US Pub. No. 2025/0328567, teaches a method for AI-driven natural language search includes receiving a user query for one or more items from a user, processing the user query by searching against at least one relational database associated with the query, where the relational database is generated by extracting features from electronic documents of a plurality of items associated with the one or more items and by identifying specifications or respective values corresponding to the extracted features of the plurality of items, generating one or more query results based on the processing of the user query, where the one or more results include at least one item identified from the plurality of items and a justification for explaining an irrelevance of the at least one item, and transmitting the one or more query results to a user device for presentation to the user;
Berg et al., US Pub. No. 2025/0298721, teaches debug tools comprising a generative AI-based tool configured to provide suggested follow up questions during a debug session. The generative AI-based tool is capable of analyzing the current conversation session and providing suggestions for next steps in the debug process. In one embodiment, the generative AI-based tool can automatically select the next step in the debug process and iteratively continue until the problem has been debugged;
Zhang et al., US Pub. No. 2023/0044106 A1, teaches a method for querying questions. The method includes: acquiring input information of a user; acquiring intention information of the user based on the input information of the user; determining an answer generation rule; and generating, based on the input information and the intention information, a first answer in accordance with the answer generation rule, and providing the first answer to the user.
Jiang et al., US Pub. No. 2019/0371299 A1, teaches a question answering method includes obtaining target question information; determining a candidate question and answer pair based on the target question information; calculating a confidence of answer information in the candidate question and answer pair, where the confidence is used to indicate a probability that question information in the candidate question and answer pair belongs to an answer database or an adversarial database; determining whether the confidence is less than a first preset threshold; and when the confidence is less than the first preset threshold, outputting information indicating incapable of answering;
Subrahmaniam et al., US Pub. No. 2025/0272505, teaches a question and answering system for answering questions on tabular datasets. In certain aspects, answering questions with the question and answering system includes generating a sequence of operations for answering the user question using a large language model wherein each operation of the sequence of operations encodes a data operation. Answering questions further includes generating an output based on performing the sequence of operations on a tabular dataset; determining a confidence score associated with the output; and presenting the output to the user.;
Honke et al., US Pub. No. 2024/0394286 A1, teaches methods includes obtaining a prompt, obtaining a set of documents, generating an input, providing the input to a plurality of language models, generating a distribution from intermediate answers from the language models; and generating an answer to the prompt by performing a probabilistic inference over the distribution.
CONTACT INFORMATIONAny inquiry concerning this communication or earlier communications from the examiner should be directed to EVAN S ASPINWALL whose telephone number is (571)270-7723. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Neveen Abel-Jalil can be reached at 571-270-0474. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Evan Aspinwall/Primary Examiner, Art Unit 2152 4/2/2026
Prosecution Insights