DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is in response to the application 19/218618 filed on 05/27/2025.
Claims 1-20 have been examined and are pending in this application.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. CN 202410714466.1, filed on 06/03/2024.
Claim Objections
Claims 8 and 16 are objected to because of the following informalities:
Regarding claims 8 and 16; claims 8 and 16 recite the limitations “BERT”. The acronym BERT is recited without spelling out in full at its first occurrence. The examiner notes for acronym BERT should be spelled out with its first occurrence. Appropriate correction(s) is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 USC 101 as being directed to an abstract idea without being integrated into a practical application or being significantly more.
Regarding claims 1, 9 and 17, the claims recite the limitations “extract[ing] a named entity …;” and “determine[ing] an authenticity evaluation …;” Broadly interpreted, the aforementioned steps are directed to mental processes as said steps could be performed in the human mind. Therefore, the claims recite an abstract idea.
Said abstract idea and/or judicial exception is not integrated into a practical application as the claim does not recite any other active steps that could be considered that the abstract idea is being integrated into a practical application. It’s noted that the claim recites the operations “obtain[ing] a first question data;” and “input[ting] the second question data.” However, said operations are not sufficient to consider that the abstract idea is being interpreted into a practical application. Said operations are recited at a high level of generality in gathering/processing/storing information, which are a form of insignificant extra-solution activity.
It’s also noted that the claims recite additional limitation/elements (i.e., processor, memory, etc.,). However, said additional elements are recited at a high-level of generality (i.e., as a generic computing device performing a generic computer functions) such that it amounts no more than mere instructions to apply the exception or abstract idea using generic computer components. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements/limitations/embodiments that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. As mentioned above, although the claims recite additional elements, said elements taken individually or as a combination, do not result in the claim amounting to significantly more than the abstract idea because as the additional elements perform generic computer content distributing functions routinely used in information technology field. As discussed above, the additional elements recited at a high-level of generality such that they amount no more than mere instructions to apply the exception using a generic computer component. Therefore, the claim is directed to non-statutory subject matter.
Regarding claims 3, 5-7, 11, 13-15, and 19 are also rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter for the same reasons addressed above as the claims recite an abstract idea and the claims do not positively recite any other operations that could be considered as the abstract idea is being integrated into a practical application or significantly more.
It’s noted that Claim 3, 11, and 19 recite the limitations “determining [] the analysis basis information;” ,” and “determining the authenticity evaluation result;” Claims 5 and 13 recite the limitation: “determining the authenticity evaluation result;” Claims 6 and 14 recite the limitation: “determining the first check result;” Claims 7 and 15 recite the limitation: “determining the second check result;” Said steps are either directed to mental processes and/or in a form of insignificant extra-solution activities. Therefore, claims 3, 5-7, 11, 13-15, and 19 are also rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7, 9-15, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over XU et al. (“XU,” CN 117744802) in view of Bright et al. (“Bright,” US 2025/0139139).
Regarding claim 1: XU discloses a model authenticity evaluation method, wherein the method comprises: obtaining first question data used to perform authenticity evaluation on a target model, and inputting the first question data to the target model to obtain a first response result corresponding to the first question data (XU: page 3 lines 30-34 S1: generating an initial answer based on the initial question input by the large language model. In this step, the initial answer will first be generated using the large language model (this initial answer will be the starting point of the verification chain, i.e., the basis of the subsequent steps). The large language model usually analyzes the given input text according to the pre-trained model parameter to generate the predicted answer);
extracting a named entity comprised in the first question data, and constructing second question data based on the named entity and the first question data, wherein the second question data is used to trigger the target model to output an analysis basis and a result for the first question data (XU: page 3 lines 36-39 S2: the initial question and the initial answer are combined into a prompt word, and then input into the large language model, so as to generate a plurality of verification questions related to the initial question and the initial answer, wherein, by using the corre prompt algorithm, generating several verification questions related to the initial question and the initial answer);
inputting the second question data to the target model to obtain a model prediction result corresponding to the second question data, wherein the model prediction result comprises analysis basis information obtained by analyzing the first question data and a second response result corresponding to the first question data (XU: page 3 lines 40-41 through page 4 lines 1-3 S3: inputting the verification question obtained based on S2 to the large language model to generate the verification answer of the verification question. In this step, in the case where an initial answer and an initial question are provided for a large language model, the model will generate a set of verification questions, the techniques used herein being a lens prompt, the purpose of which is to evaluate the accuracy of the initial answer of the model).
XU does not explicitly disclose determining an authenticity evaluation result of the target model based on the first response result and the model prediction result.
However Bright discloses determining an authenticity evaluation result of the target model based on the first response result and the model prediction result (Bright: par. 0041 one of the validation models 106a through 106n includes an accuracy check. The pre-loaded query context for an accuracy check evaluates the factual correctness and authenticity of the output generated by the AI engine).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to combine the teachings of Bright with the system/method of XU to include determining an authenticity evaluation result of the target model based on the first response result and the model prediction result. One would have been motivated to validate framework seeks to ensure the accuracy, relevance, and reliability of AI-generated content in any system by maintains user trust and creates a consistent experience (Bright: par. 0022).
Regarding claim 2: XU in view of Bright discloses the method according to claim 1.
XU further discloses wherein constructing the second question data based on the named entity and the first question data comprises:
obtaining a preset question template (XU: page 3 lines 33-34 the large language model usually analyzes the given input text according to the pre-trained model parameter); and
replacing corresponding to-be-replaced elements in the question template with the named entity and the first question data, and constructing the second question data based on a replaced question template, wherein the second question data comprises: please introduce each named entity comprised in the first question data, analyze the first question data, provide an analysis basis, and give an answer to the first question data (XU: page 4 lines 33-39 the large language model further answers the set of verification questions. These answers provide further confirmation of the accuracy of the initial answer. If the system detects that the answer of the verification question and the answer of the original question are not in accordance with each other, it is determined that the answer is incorrect or inaccurate, then a correction flow is triggered to ensure that accurate and high-quality output results are generated. The modification or deletion in this step may include, but is not limited to, replacing or re-generating an answer, eliminating misleading information, and adjusting model parameters to improve the accuracy of the answer).
Regarding claim 3: XU in view of Bright discloses the method according to claim 1.
Bright further discloses wherein determining the authenticity evaluation result of the target model based on the first response result and the model prediction result comprises:
performing splitting processing on text data comprised in the model prediction result to obtain one or more pieces of different split text data (Bright: par. 0139 the data layer 1002 is a subset of a larger data set. For example, a data set is split into three mutually exclusive subsets: a training set, a validation (or cross-validation) set, and a testing set. The three subsets of data [] are used sequentially during AI model 1030 training);
inputting each piece of split text data to a pre-trained classification model to obtain a category to which the piece of split text data belongs, wherein the category comprises analysis basis information and a response result corresponding to question data (Bright: par. 0139 the training set is first used to train one or more ML models, each AI model 1030, e.g., having a particular architecture, having a particular training procedure, being describable by a set of model hyperparameters, and/or otherwise being varied from the other of the one or more ML models);
determining, based on the category to which each piece of split text data belongs, the analysis basis information and the second response result that are comprised in the model prediction result (Bright: par. 0139 once such a trained ML model is obtained (e.g., after the hyperparameters have been adjusted to achieve a desired level of performance), a third step of collecting the output generated by the trained ML model applied to the third subset (the testing set) begins); and
determining the authenticity evaluation result of the target model based on the first response result, the analysis basis information, and the second response result (Bright: par. 0139 the validation (or cross-validation) set [] is then used as input data into the trained ML models to, e.g., measure the performance of the trained ML models and/or compare performance between them).
The motivation is the same that of claim 1 above.
Regarding claim 4: XU in view of Bright discloses the method according to claim 3.
Bright further discloses wherein performing the splitting processing on text data comprised in the model prediction result to obtain one or more pieces of different split text data comprises:
performing splitting processing on the text data comprised in the model prediction result based on a position in which a punctuation mark is located and/or a position in which a preset line break is located, to obtain one or more pieces of different split text data (Bright: par. 0043 one of the validation models 106a through 106n includes a format check. The format check is equipped with a pre-loaded query context that establishes specific formatting standards for the generated content (e.g., proper punctuation, correct capitalization, consistent spacing, and other specified formatting standards). The check ensures that the output from the generative AI engine adheres to these predefined formatting guidelines).
The motivation is the same that of claim 1 above.
Regarding claim 5: XU in view of Bright discloses the method according to claim 3.
XU further discloses wherein determining the authenticity evaluation result of the target model based on the first response result, the analysis basis information, and the second response result comprises:
performing consistency check between the first response result and the second response result to obtain a corresponding first check result (XU: page 6 lines 21-22 adding consistency check on the basis of the third mode, making the model focus on the inconsistent content before and after);
performing consistency check between the first response result and the analysis basis information to obtain a corresponding second check result (XU: page 2 lines 22-23 the step S2 is to generate a plurality of verification questions related to the initial question and the initial answer by using a match prompt algorithm).
Bright further discloses determining the authenticity evaluation result of the target model based on the first check result and the second check result (Bright: par. 0041 one of the validation models 106a through 106n includes an accuracy check. The pre-loaded query context for an accuracy check evaluates the factual correctness and authenticity of the output generated by the AI engine).
The motivation is the same that of claim 1 above.
Regarding claim 6: XU in view of Bright discloses the method according to claim 5.
XU further discloses wherein performing the consistency check between the first response result and the second response result to obtain a corresponding first check result comprises:
inputting the first response result and the second response result to a first check model to obtain a consistency score between the first response result and the second response result (XU: page 4 lines 27-30 the search enhancement algorithm further comprises a search result model for optimizing and improving the correlation of the search result; in the search result model, comprising a search result correlation model for performing correlation score on the information obtained from the search content and the knowledge base); and determining the first check result based on the consistency score between the first response result and the second response result (XU: page 5 lines 16-19 the system comprises a search result correlation model for scoring the correlation of the information obtained from the search content and the knowledge base. This helps the system to determine which results are most relevant to meet the user's search requirements).
Regarding claim 7: XU in view of Bright discloses the method according to claim 6.
Bright further discloses wherein performing the consistency check between the first response result and the analysis basis information to obtain a corresponding second check result comprises:
separately constructing to-be-checked data by using the first response result and each piece of split text data that belongs to the analysis basis information (Bright: par. 0041 one of the validation models 106a through 106n includes an accuracy check. The pre-loaded query context for an accuracy check evaluates the factual correctness and authenticity of the output generated by the AI engine [] the check parses the generated output, extracts factual claims, and compares the factual claims against a structured database of verified information);
inputting each piece of to-be-checked data to a second check model to obtain a consistency score between the first response result and each piece of split text data that belongs to the analysis basis information (Bright: par. 0041 a classifier trained on labeled datasets of factual and non-factual statements are used to detect claims using features such as a presence of named entities, specific syntactic patterns, and/or certain keywords or phrases indicative of factual statements); and
determining the second check result based on the consistency score between the first response result and each piece of split text data that belongs to the analysis basis information (Bright: par. 0041 the accuracy check queries the knowledge bases to validate that the generated content aligns with established facts).
The motivation is the same that of claim 1 above.
Regarding claim 9: XU discloses a model authenticity evaluation device, wherein the model authenticity evaluation device comprises:
a processor (XU: page 6 line 30 a processor); and
a memory (XU: page 6 line 30 a memory), configured to store computer-executable instructions, wherein when the executable instructions are executed, the processor is caused to:
obtain first question data used to perform authenticity evaluation on a target model, and input the first question data to the target model to obtain a first response result corresponding to the first question data (XU: page 3 lines 30-34 S1: generating an initial answer based on the initial question input by the large language model. In this step, the initial answer will first be generated using the large language model (this initial answer will be the starting point of the verification chain, i.e., the basis of the subsequent steps). The large language model usually analyzes the given input text according to the pre-trained model parameter to generate the predicted answer);
extract a named entity comprised in the first question data, and construct second question data based on the named entity and the first question data, wherein the second question data is used to trigger the target model to output an analysis basis and a result for the first question data (XU: page 3 lines 36-39 S2: the initial question and the initial answer are combined into a prompt word, and then input into the large language model, so as to generate a plurality of verification questions related to the initial question and the initial answer, wherein, by using the corre prompt algorithm, generating several verification questions related to the initial question and the initial answer);
input the second question data to the target model to obtain a model prediction result corresponding to the second question data, wherein the model prediction result comprises analysis basis information obtained by analyzing the first question data and a second response result corresponding to the first question data (XU: page 3 lines 40-41 through page 4 lines 1-3 S3: inputting the verification question obtained based on S2 to the large language model to generate the verification answer of the verification question. In this step, in the case where an initial answer and an initial question are provided for a large language model, the model will generate a set of verification questions, the techniques used herein being a lens prompt, the purpose of which is to evaluate the accuracy of the initial answer of the model).
XU does not explicitly disclose determine an authenticity evaluation result of the target model based on the first response result and the model prediction result.
However, Bright discloses determine an authenticity evaluation result of the target model based on the first response result and the model prediction result (Bright: par. 0041 one of the validation models 106a through 106n includes an accuracy check. The pre-loaded query context for an accuracy check evaluates the factual correctness and authenticity of the output generated by the AI engine).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to combine the teachings of Bright with the system/method of XU to include determine an authenticity evaluation result of the target model based on the first response result and the model prediction result. One would have been motivated to validate framework seeks to ensure the accuracy, relevance, and reliability of AI-generated content in any system by maintains user trust and creates a consistent experience (Bright: par. 0022).
Regarding claims 10-15: Claims 10-15 are similar in scope to claims 2-7, respectively, and are therefore rejected under similar rationale.
Regarding claim 17: XU discloses a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores a computer program, which when executed by a processor causes the processor to:
obtain first question data used to perform authenticity evaluation on a target model, and input the first question data to the target model to obtain a first response result corresponding to the first question data (XU: page 3 lines 30-34 S1: generating an initial answer based on the initial question input by the large language model. In this step, the initial answer will first be generated using the large language model (this initial answer will be the starting point of the verification chain, i.e., the basis of the subsequent steps). The large language model usually analyzes the given input text according to the pre-trained model parameter to generate the predicted answer);
extract a named entity comprised in the first question data, and construct second question data based on the named entity and the first question data, wherein the second question data is used to trigger the target model to output an analysis basis and a result for the first question data (XU: page 3 lines 36-39 S2: the initial question and the initial answer are combined into a prompt word, and then input into the large language model, so as to generate a plurality of verification questions related to the initial question and the initial answer, wherein, by using the corre prompt algorithm, generating several verification questions related to the initial question and the initial answer);
input the second question data to the target model to obtain a model prediction result corresponding to the second question data, wherein the model prediction result comprises analysis basis information obtained by analyzing the first question data and a second response result corresponding to the first question data (XU: page 3 lines 40-41 through page 4 lines 1-3 S3: inputting the verification question obtained based on S2 to the large language model to generate the verification answer of the verification question. In this step, in the case where an initial answer and an initial question are provided for a large language model, the model will generate a set of verification questions, the techniques used herein being a lens prompt, the purpose of which is to evaluate the accuracy of the initial answer of the model).
XU does not explicitly disclose determine an authenticity evaluation result of the target model based on the first response result and the model prediction result.
However, Bright discloses determine an authenticity evaluation result of the target model based on the first response result and the model prediction result (Bright: par. 0041 one of the validation models 106a through 106n includes an accuracy check. The pre-loaded query context for an accuracy check evaluates the factual correctness and authenticity of the output generated by the AI engine).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to combine the teachings of Bright with the system/method of XU to include determine an authenticity evaluation result of the target model based on the first response result and the model prediction result. One would have been motivated to validate framework seeks to ensure the accuracy, relevance, and reliability of AI-generated content in any system by maintains user trust and creates a consistent experience (Bright: par. 0022).
Regarding claims 18-20: Claims 18-20 are similar in scope to claims 2-4, respectively, and are therefore rejected under similar rationale.
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over XU et al. (“XU,” CN 117744802) in view of Bright et al. (“Bright,” US 2025/0139139) and Khazaeli et al. (“Khazaeli,” US 2025/0139146).
Regarding claim 8: XU in view of Bright discloses the method according to claim 7.
XU in view of Bright does not explicitly disclose wherein the target model is a large language model, the classification model is a model obtained by pre-training a BERT model in through supervised learning, the first check model is a model obtained by pre-training a BERT model in through supervised learning, and the second check model is a model obtained by pre-training a BERT model in through supervised learning.
However, Khazaeli discloses wherein the target model is a large language model, the classification model is a model obtained by pre-training a BERT model in through supervised learning, the first check model is a model obtained by pre-training a BERT model in through supervised learning, and the second check model is a model obtained by pre-training a BERT model in through supervised learning (Khazaeli: par. 0019 leverage attention matrices generated by a trained Bidirectional Encoder Representations from Transformers (BERT) model to calculate attributions for each word token in a resulting query/answer pair; par. 0025 a query/answer pair 102 for a query is provided as input to a BERT model 104 [] the output of the BERT model 104 is provided to a classifier 106, which produces a determination 108 as to whether the answer is a satisfactory answer or an unsatisfactory answer).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to combine the teachings of Khazaeli with the system/method of XU and Bright to include the target model is a large language model, the classification model is a model obtained by pre-training a BERT model in through supervised learning. One would have been motivated to provide leverage attention matrices generated by a trained model to calculate attributions for each word token in a resulting query/answer pair (Khazaeli: par. 0019).
Regarding claim 16: Claim 16 is similar in scope to claim 8, and is therefore rejected under similar rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Fahimeh Mohammadi whose telephone number is (571)270-7857. The examiner can normally be reached Monday - Friday 9:00 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Luu Pham can be reached at 5712705002. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/FAHIMEH MOHAMMADI/ Examiner, Art Unit 2439
/LUU T PHAM/Supervisory Patent Examiner, Art Unit 2439
Prosecution Insights