Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
2.
a. Independent claims 1, 10 and 19
Applicant’s arguments regarding Thompson and Paiement are moot in view of the new grounds of rejection set forth. The present rejection additionally relies on newly cited Parshakova to teach the newly recited probabilistic word sequence.
b. Claims 3 and 12
Applicant’s arguments regarding Thompson and Paiement are moot in view of the new ground of rejection set forth.
c. Claims 4 and 13
Applicant’s arguments regarding Thompson and Paiement are moot in view of the new ground of rejection set forth. The present rejection additionally relies on newly cited Goligorsky to teach the newly recited predetermined prompt portions and ordered prompt content determination limitations.
d. Claims 5 and 14
Applicant’s arguments regarding Thompson and Paiement are moot in view of the new grounds of rejection set forth. The present rejection additionally relies on newly cited Kumar to teach the newly recited validation for domain specific errors and hallucinations.
e. Claims 7 and 16
Applicant’s arguments regarding the patentability of the remaining dependent claims are not moot in view of the new ground of rejection set forth. The present rejection additionally relies on newly cited Wang to teach assigning SHAP values
Claim Rejections - 35 USC § 103
3. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
4. Claims 1-3, 8-12 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Thompson (US 2025/0272577) in view of Paiement (US 2023/0206096) and further in view of Parshakova (US 2022/0083852).
Regarding Claim 1:
Thompson discloses a method for generating customized model explanations via a first model (Thompson: ¶[0025]-[0026] discloses a model for outputting generative output), the method being implemented by at least one processor, the method comprising:
;
modifying, by the at least one processor via the first model, the prompt based on at least one predetermined guideline (Thompson: ¶[0025]-[0026] discloses a predetermined guideline as a predetermined threshold used to govern how the prompt must be modified to produce a valid test output. Modification produces a new prompt (i.e. the test response);
analyzing, by the at least one processor via the first model, patterns from training data(Thompson: ¶[0013]- [0015] discloses an LLM pretrained on data wherein training learns model parameters and semantic relationships among words and wherein the trained model uses contextual embeddings and learned relationships during inference);
computing, by the at least one processor, (Thompson: ¶[0027] and ¶[0029]-[0030] discloses providing the modified second prompt to the LLM and generating a subsequent search result / test response, with repeated prompt refinement causing the LLM to generate a new response);
generating, by the at least one processor, a test response (Thompson: ¶[0025] and ¶[0027] discloses providing the modified second prompt to the LLM and generating a subsequent search result and test response with repeated prompt refinement causing the LLM to generate a new response);
validating, by the at least one processor via the first model, the test response by determining whether at least one error is detected in the test response (Thompson: ¶[0025]-[0026] discloses validating the model’s response and detecting errors when thresholds are not met);
performing, by the at least one processor via the first model when the at least one error is detected, at least one corrective action that resolves each of the at least one detected error by altering the prompt (Thompson: ¶[0026]-[0030] discloses updating and iteratively resolving errors by retraining the model);
tuning, by the at least one processor via the first model, the altered prompt based on at least one response attribute (Thompson: ¶[0026] modifies prompts by varying search scope and restrictions. Teaches adjusting prompts repeatedly based on specific response criteria, where those criteria define how the next prompt must be structured); and
generating, by the at least one processor via the first model and based on the tuned prompt, a model explanation in the natural language format (Thompson: ¶[0040]-[0046] the model explanation corresponds to the structured report generated from the final tuned prompt output) .
Thompson does not explicitly disclose, but Paiement does disclose:
receiving, by the at least one processor, a request to explain an output generated by a second model (Paiement: ¶[0028] teaches generating an explanation text providing an interpretation of a first machine learning model, and obtaining outputs and generating explanation);
generating, by the at least one processor via the first model, a prompt in a natural language format based on the received request, the request including at least one feature attribution query for inquiring how each feature from among a plurality of features contributed to the output generated by the second model and corresponding subject information that relates to a subject of the request (Paiement: ¶[0038] discloses global feature importance values, local feature importance values and local feature modification impact data, ¶[0060] discloses identifying which features are most important/impactful for output. This directly corresponds to how each feature contributed to the output), wherein the prompt includes instructions for enabling the first model to explain how the second model generated the output (Paiement: ¶[0077] clearly indicates generating via a second machine learning model , an explanation of a first machine learning model, this is explicitly teaching a two model architecture and one model explaining another’s output, as recited);
a model explanation in the natural language format that explains how the second model generated the output (Paiement: ¶[0010]-[0011] discloses textual explanations of MLM performance and attributes and factors impacting decisions).
Thompson and Paiement are combinable because they are from the same field of endeavor, generative machine learning text model deployment. Paiement discloses a method for generating, via a second machine learning model, an explanation text providing an interpretation of a first machine learning model in accordance with a set of interpretation criteria and description information of the first machine learning model. Thompson discloses an iterative prompt training apparatus. The iterative prompt training apparatus receives a search request for searching for information and automatically generates a first prompt instructing a large language model to search for the information requested in the search request. It would have been obvious to one of ordinary skill in the art at the time of filing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose Paiement’s teaching into Thompson in order to provide model explanations that include feature-level attribution and explanation of model outputs. The motivation for doing so is disclosed in ¶[0020] of Paiement: “the present disclosure may present an interface for users or developers to provide feedback to improve the explanation model, such as enabling a user to point out portions where the explanation is unclear, not relevant to the user's information needs or preferences.”
The proposed combination of Thompson in view of Paiement does not explicitly disclose:
respective probabilities for a plurality of word sequences;
based on a result of the computing of the respective probabilities.
However, Parshakova discloses:
respective probabilities for a plurality of word sequences (Parshakova: ¶[0007] discloses autoregressive sequential text generation in which each next token is predicted from a locally normalized conditional distribution such as the SoftMax and further explains that the local probabilities establish the probability of the sequence through the chain rule. ¶[0074]-[0076] discloses identifying the target sequence as a text sequence and then defines the model distribution over that same target sequence which is based on the input context. Each next symbol is assigned a normalized conditional probability by the neural network);
based on a result of the computing of the respective probabilities (Parshakova: ¶[0134]-[0135] discloses receiving a new context source such as a token sequence and generating a new target text sequence by sampling the normalized probability distribution supplied by the trained sequential model).
Thompson and Paiement in view of Parshakova are combinable because they are in the same field of endeavor, i.e., all disclose systems or methods using machine learning neural network driven language models to generate natural language output from input. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modifying Thompson to predict an autoregressive probability distribution of word sequences and generation of text from the resulting probability distribution. Parshakova explicitly reasons for this because autoregressive generation permits “exact sampling of full sequences from the model distribution can be directly obtained through a sequence of local sampling decisions.” As discloses in ¶[0007].
Regarding Claim 2:
The proposed combination of Thompson, Paiement further discloses the method of claim 1, wherein each of the at least one predetermined guideline relates to an automated prompt modification procedure that is usable to structure data in the prompt (Thompson: ¶[0022]-[0026] the predetermined guidelines correspond to the predetermined thresholds that govern when and how the prompt must be automatically modified. Further discloses that the prompt is structured to include specified data fields (e.g., content of the request, sources requester information), and wherein the automated prompt modification procedure includes a prompt composition requirement and a prompt modification order requirement (Thompson: ¶[0024], ¶[0041] discloses the reference teaches that prompts have required components (how prompt must be arranged/structured) and a prompt modification order requirement (Thompson: ¶[0021] ¶[0029], ¶[0031], Fig. 3, discloses that a required order of modification is to generate the prompt, evaluate response, modify prompt, re-evaluate and repeat in the same sequence until conditions met).
Regarding Claim 3:
The proposed combination of Thompson and Paiement further discloses the method of claim 2, wherein the prompt composition requirement relates to a predetermined configuration of the data in the prompt (Thompson: ¶[0023] discloses that the prompt must contain specific structured fields, i.e., a predetermined configuration), and wherein the prompt composition requirement includes at least one from among a persona requirement that describes a role for adoption by the at least one model, a task outline requirement that references model inputs (Thompson: ¶[0020]-[0025] discloses the prompt contains the search request which defines the task), a model directive requirement that provides instructions for completing requested tasks (Thompson: ¶[0022]-[0026] the prompt instructs the model to search specified sources), and an input definition requirement that describes the model inputs.
Regarding Claim 8:
The proposed combination of Thompson and Paiement further discloses the method of claim 1, wherein the prompt corresponds to a formulation of natural language text (Thompson: ¶[0021] the prompt is explicitly described as text intended for an large language model (natural language text)) that provides a plurality of instructions to a machine learning model for performance of a task (Thompson: ¶[0020]-[0025] the prompt contains multiple directive e.g., what information to search for, which sources to use. The task is searching and retrieving relevant information which is the explicit purpose of the prompt).
Regarding Claim 9:
The proposed combination of Thompson and Paiement further discloses the method of claim 1, wherein the at least one model includes at least one from among a large language model, a deep learning model, a neural network model, a natural language processing model, a machine learning model, a mathematical model, and a process model (Thompson: ¶[0021] discloses an LLM).
Regarding Claim 10:
Claim 10 has been analyzed with regard to claim 1 (see rejection above) and is rejected for the same reasons of obviousness used above.
It is noted that Thompson discloses a device at ¶[0019].
Regarding Claim 11:
Claim 11 has been analyzed with regard to claim 2 (see rejection above) and is rejected for the same reasons of obviousness used above.
Regarding Claim 12:
Claim 12 has been analyzed with regard to claim 3 (see rejection above) and is rejected for the same reasons of obviousness used above.
Regarding Claim 17:
Claim 17 has been analyzed with regard to claim 8 (see rejection above) and is rejected for the same reasons of obviousness used above.
Regarding Claim 18:
Claim 18 has been analyzed with regard to claim 9 (see rejection above) and
is rejected for the same reasons of obviousness used above.
Regarding Claim 19:
Claim 19 has been analyzed with regard to claim 1 (see rejection above) and is rejected for the same reasons of obviousness used above.
It is noted that Thompson discloses non-transitory computer readable medium ¶[0017].
Regarding Claim 20:
Claim 20 has been analyzed with regard to claim 2 (see rejection above) and is rejected for the same reasons of obviousness used above.
5. Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Thompson in view of Paiement, in view of Goligorsky (US 2024/0265205).
Regarding Claim 4:
The proposed combination of Thompson and Paiement further discloses the method of claim 2, wherein the prompt modification order requirement relates to a predetermined sequence of modification actions that is usable to change the prompt (Thompson: ¶[0020]-[0021] and ¶[0029] teaches a fixed predefined sequence used repeatedly to modify the prompt), and wherein the predetermined sequence includes:
and a fourth step that provides prompt inputs for facilitating the automated prompt modification procedure (Thompson: discloses generating and providing an altered prompt to the LLM as part of the automated prompt modification procedure and repeating the process using the altered prompt and resulting response).
Thompson does not explicitly disclose:
a first step that determines whether first predetermined lines in the prompt contain at least one of a user persona and role information,
a second step that determines whether second predetermined lines in the prompt contain a task outline,
a third step that determines whether third predetermined lines in the prompt contain model directives
However, Goligorsky explicitly discloses:
a first step that determines whether first predetermined lines in the prompt contain at least one of a user persona and role information (Goligorsky: ¶150 discloses predetermined prompt sections separated by whitespace including line breaks, ¶157 discloses prompt templates known in advance and pre parsed into labeled sections and ¶158 discloses how to determine what information is expected where, lastly ¶136 specifically provides a prompt instruction in the voice of a pirate which),
a second step that determines whether second predetermined lines in the prompt contain a task outline (Goligorsky: ¶152 discloses an ordered prompt template and ¶149 discloses that the preamble identifies the goal of the requested task, e.g., writing a blog),
a third step that determines whether third predetermined lines in the prompt contain model directives (Goligorsky: ¶151 explicitly discloses predetermined instruction sections containing direction defining how the LLM should response, including “length, tone of voice, specific style, type of response”).
Thompson and Goligorsky are in the same field of endeavor, i.e., both disclose systems or methods for generating and modifying natural language prompts supplied to large language models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Thompson’s automated prompt modification procedure to organize the prompt into Goligorsky’s predetermined line separated prompts sections and determine whether the designed sections contain the appropriate task, persona, role and model directive information before providing the resulting prompt input. The motivation for doing so is that this would provide a predictable structure for locating and modifying the different types of prompt information and therefore improve the quality of the resulting LLM output because Goligorsky explicitly teaches in ¶[0003] that “The careful pre-construction of a prompt for a particular use case can be valuable in shaping the quality of the output generations resulting from that prompt's execution by an LLM.”
Regarding Claim 13:
Claim 13 has been analyzed with regard to claim 4 (see rejection above) and is rejected for the same reasons of obviousness used above.
6. Claims 5-6 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Thompson in view of Paiement, further in view of Parshakova and further in view of Kumar (US 2024/0330755).
Regarding Claim 5:
The proposed combination of Thompson and Paiement further discloses the method of claim 1, wherein the validating of the test response includes error determination (Thompson: ¶[0025] and ¶[0027] discloses determining whether the model output contains an error by comparing the result to validation criteria) and self-consistency determination that are performed by the at least one model (Thompson: ¶[0023] the model is performing self-consistency evaluation – it selects and compares information sources based on learned relevance and trustworthiness, which requires internal validation against its own knowledge representation), the self-consistency determination relating to a factual accuracy validation of sources utilized by the at least one model (Thompson: ¶[0022]-[0024] discloses the model performing self-consistency evaluation, it selects and compares information sources based on learned relevance and trustworthiness, which requires validation against its own knowledge representation) (Thompson: ¶[0037]-[00380 discloses evaluating generated results and determining errors, cross references sources based on trustworthiness and performing response and prompt validation to determine whether the predetermined requirements are satisfied)
Thompson does not explicitly disclose:
and wherein the validating includes checking for domain-specific errors, prompt validations, and hallucinations.
However, Kumar discloses:
and wherein the validating includes checking for domain-specific errors, prompt validations, and hallucinations (Kumar: ¶18 discloses domain specific errors in the content and detecting the domain specific error material in that domain, ¶20-21 and ¶42-44 further discloses detecting hallucinated LLM output).
Thompson and Kumar are in the same field of endeavor, i.e., both disclose systems or methods for evaluating outputs generated by large language models and taking corrective action when results are erroneous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose the validating includes checking for domain-specific errors, prompt validations, and hallucinations. The motivation for doing so is that checking for hallucinations and domain specific factual errors is a common process in generative language models and would provide the ability to produce user output that is not incorrect or erroneous. Kumar explicitly states in ¶18 that this process “provide fast and accurate detection.”
Regarding Claim 6:
The proposed combination of Thompson, Paiement and Kumar further discloses the method of claim 5, wherein the error determination includes at least one from among technical error validation that relates to identification of domain concept misinterpretations in the test response and input-output validation that substantiates the test response based on the modified prompt (Thompson: ¶[0025] performs a quality check on the search result to determine whether the search result satisfies one or more predetermined thresholds and those threshold include a predetermined character count, word count or phrase or set phrases).
Regarding Claim 14:
Claim 14 has been analyzed with regard to claim 5 (see rejection above) and is rejected for the same reasons of obviousness used above.
Regarding Claim 15:
Claim 15 has been analyzed with regard to claim 6 (see rejection above) and is rejected for the same reasons of obviousness used above.
7. Claims 7 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Thompson in view of Paiement, further in view of Parshakova and further in view of Wang (US 2024/0177071).
Regarding Claim 7:
The proposed combination of Thompson, Paiement and Parshakova further discloses the method of claim 1, wherein the at least one response attribute defines desired output formatting for the model explanation (Thompson: ¶[0041] and ¶[0046] explicitly teaches that the system determines the format of the final output),
wherein the at least one response attribute includes at least one from among a formatting attribute that defines an arrangement of information in the model explanation, a clarity attribute that defines a type of the information for inclusion in the model explanation, and a conciseness attribute that defines an amount of the information for inclusion in the model explanation (Thompson: ¶[0044] and ¶[0046] disclose organizing statements into categories, bullet structures or outline format directly corresponds to formatting attribute defining the arrangement of information), and
.
The proposed combination of Thompson, Paiement and Parshakova does not explicitly disclose:
wherein the generating of the model explanation includes assigning a Shapley Additive Explanation (SHAP) value to each respective feature from among the plurality of features based on a contribution of each respective feature to the generated output, and wherein a magnitude of the assigned SHAP value indicates an impact on model predictions.
However, Wang further discloses:
wherein the generating of the model explanation includes assigning a Shapley Additive Explanation (SHAP) value to each respective feature from among the plurality of features based on a contribution of each respective feature to the generated output, and wherein a magnitude of the assigned SHAP value indicates an impact on model predictions (Wang: ¶211 and ¶219 disclose performing a SHAP based model interpretation and calculation a SHAP value for each feature value of each data instance and ¶214-216 disclose using SHAP value magnitude to identify the importance or impact of features explaining that features are prioritized by mean absolute SHAP value as a metric reflecting the magnitude of the features).
Thompson, Paiement, Parshakova and Wang are combinable because they are from the same field of endeavor, each disclosing systems for interpreting model outputs. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose a SHAP explanation framework to calculate a SHAP value for each respective feature and use the magnitude of the SHAP values to indicate and prioritize the features impact on model predictions as taught by Wang. The motivation for doing so is “ The summary-plot prioritizes features by mean(|SHAP|), which is an effective metric in reflecting the magnitude of features' SHAP values” as disclosed in ¶215 of Wang.
Regarding Claim 16:
Claim 16 has been analyzed with regard to claim 7 (see rejection above) and is rejected for the same reasons of obviousness used above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IAN SCOTT MCLEAN whose telephone number is (703)756-4599. The examiner can normally be reached "Monday - Friday 8:00-5:00 EST, off Every 2nd Friday".
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at (571) 272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IAN SCOTT MCLEAN/Examiner, Art Unit 2654
/HAI PHAN/Supervisory Patent Examiner, Art Unit 2654
Prosecution Insights