DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in Japan on August 10th, 2024. It is noted, however, that applicant has not filed a certified copy of the JP2023-131099 application as required by 37 CFR 1.55.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Edwards et al. (hereninafter Edwards) (US 20240394481 A1) in view of Chan et al. (hereinafter Chan) (US 11386342 B2) (see paragraph numbers from attached copy) further in view of Goyal et al. (hereinafter Goyal) (US 20080120257 A1)
Regarding claim 1, Edwards teaches: A computer system (Edwards, Abstract, P[0024]), wherein
stores a large-scale language model that receives an instruction sentence as an input and outputs an interpretation sentence for interpreting a result of the inference (Edwards: P[0095]: “The prompt-template generating LLM 104b can be implemented using conventional, generally trained LLMs or may be specifically trained to generate candidate prompt-templates. Similarly, the prompt testing LLM 108b can be configured and trained to model the LLMs on which selected prompts will be used for a given application.” (Edwards explicitly discloses LLMs configured to receive instructions (prompts) and output results, directly mapping to the large-scale language model limitation), and first text template information that stores first template data in which a characteristic of a contribution value indicating a magnitude of a contribution to the result of the inference of a feature in a group constituted with one or more features is associated with a template of the instruction sentence (Edwards, P[0072]: “the prompt assessment unit 110b carries out an analysis, ranking the candidate prompt-templates based on their performance.“ and P[0067]: “Once the prompt assessment unit 110b has selected the one or more candidate prompt-templates that have produced the optimum test prompts, this candidate prompt-template, or these can candidate prompt-templates, are then stored in the selected prompt-template database 111b.” (templates with associated performance metrics (analogous to contribution value), are stored which maps closely to “first text template information” storing template data associated with feature contribution characteristics.))
acquires, for each of the plurality of groups, the first template data by referring to the first text template information based on the characteristic of the contribution value of the feature included in the group (Edwards, P[0087]: “Each test prompt is constructed from one of the candidate prompt-templates using input data from a set of pre-labelled input data. This pre-labelled input data comprises a plurality of items of input data and corresponding labels.” (template data (prompt templates) are explicitly acquired based on input characteristics (analogous to feature contribution))
generates, based on the acquired first template data and the feature included in the group, the instruction sentence to be input to the large-scale language model (Edwards, P[0087]: “At a third step S303, for each candidate prompt-template, a plurality of test prompts is generated. Each test prompt is constructed from one of the candidate prompt-templates using input data from a set of pre-labelled input data. T” (the candidate prompt-template corresponds to the first template data, the plurality of items or pre-labelled input data correspond to the features in the group, and constructing the test prompt corresponds to generating the instruction sentence that is input to the LLM)), and
outputs the interpretation sentence obtained by inputting the instruction sentence to the large-scale language model (Edwards, P[0088]: “each test prompt is passed through a further LLM to generate output data.” (an interpretation sentence is explicitly output by passing the instruction prompt (test prompt) through an LLM)).
Edwards does not teach
calculates the contribution value of each of the plurality of features using the input data, the result of the inference, and the inference model
generates a plurality of groups each constituted with one or more of the features
However, Chan teaches
calculates the contribution value of each of the plurality of features using the input data, the result of the inference, and the inference model (Chan, P[44]: “The one or more surrogate non-linear models 115 may include a feature importance model”, Chan, P[45]: “A feature importance model measures the effect that a feature of the set of inputs has on the predictions of the model.”, Chan, P[47]: “The importance feature is accumulated over all trees separately for each feature.” (computing the contribution of each feature to the model output, directly reading on the claimed step of calculating contribution values for each feature))
generates a plurality of groups each constituted with one or more of the features (Chan, P[81]: “At 404, the data associated with a machine learning model is classified into a plurality of clusters. The data may be classified into the plurality of clusters using one or more techniques (e.g., k-means clustering). Each cluster represents a subset of the entries that are similar to each other.” (this teaches grouping multiple features into clusters, reading on the claimed step of generating multiple groups of features)
It would have been prima facie obvious to one of ordinary skill in the art at before the effective filing date of this invention to modify the system of Edwards with the teachings of Chan. Doing so would have provided the feature contribution analysis and presentation of Chan (Chan, P[123]) to the automated prompt-template generation and validation framework of Edwards (Edwards, P[0013]), thus, improving the natural language outputs generated by LLMs through improved accuracy and relevancy of explanations alongside increased trust and transparency by providing the “why” (feature contribution) explanation.
Edwards and Chan does not teach
the computer system is connected to an inference system that receives input data including a plurality of features and performs an inference using an inference model
However, Goyal teaches:
the computer system is connected to an inference system that receives input data including a plurality of features and performs an inference using an inference model (Goyal, Fig. 1 and P[0016], (Fig. 1 illustrated a computer system architecture in which an inference system is connected to other system components and receives processed input data), P[0034]: “Based on the sequence 103 of information (also referred to as an "observation sequence") provided by associative parser 102, semantic inference engine 104 computes the "meaning" of each user input control from online form 101, in order to generate an associated semantic label for each user input control. ” (the inference system receives input data comprising a sequence of multiple characteristics, i.e., a plurality of features, and performs inference based on that data, P[0039]-P[0040]: “an effective machine learning mechanism for accurately labeling form controls is based on conditional random fields (CRF)” (inference is explicitly performed using an inference model)),
It would have been prima facie obvious to one of ordinary skill in the art at before the effective filing date of this invention to modify the system of Edwards with the teachings of Chan and Goyal. Doing so would have provided the explicit machine learning based inference system of Goyal (Goyal, Abstract) with the feature contribution analysis and presentation of Chan (Chan, P[123]) to the automated prompt-template generation and validation framework of Edwards (Edwards, P[0013]), thus, improving the consistency, interpretability, and reliability of natural language outputs generated by LLMs through accurate handling of noisy or incomplete data.
Regarding claim 2, The combination of Edwards, Goyal, and Chan teaches the computer system according to claim 1.
Edwards further teaches:
wherein the computer system
stores second text template information that stores second template data in which a relationship between a value of the feature and the contribution value is associated with a second template for verbalizing the relationship (Edwards, P[0015]: “Optionally, the initial prompt further defines a constraint instruction to be applied by each test prompt which constrains the generated property data generated by each test prompt.” (The constraint instruction links input properties (features) to the expected output (template data), which functions as the second template data that verbalizes the feature relationship)),
analyzes the relationship for the feature for which the relationship needs to be analyzed (Edwards, P[0060]: “In one example, the labelled data database 109b contains text data extracted from a multitude of emails. In this example, each data item comprise a fragment of email text and the label data linked to each data item specifies a property of the fragment of email text.” (This shows the analysis of the relationship between the input (email fragment) and label (feature property))
specifies the second template data by referring to the second text template information based on a result of the analysis (Edwards, P[0066]: “Typically, this enables the prompt assessment unit 110b to select the one or more candidate prompt-templates that produce the optimum results, i.e. optimum test prompts.” (The candidate prompt-templates selected based on assessment results correspond to specifying the second template data based on analysis))),
generates a relationship document that verbalizes the relationship, based on the specified second template data and the feature for which the relationship needs to be analyzed (Edwards, P[0068]: “These prompt templates can then be used in the subsequent generation of prompts” (The selected templates are used to generate outputs (prompts) that verbalize the relationship between features and expected outputs, analogous to a relationship document),
executes language processing for determining whether the interpretation sentence includes a fact inconsistent with the relationship corresponding to the relationship document (Edwards, P[0072]: “Upon the application of these performance metrics, the prompt assessment unit 110b carries out an analysis, ranking the candidate prompt-templates based on their performance.” (The analysis ensures outputs are consistent with the labelled data; effectively, this determines whether a fact is inconsistent)), and
excludes, from the interpretation sentence to be output, the interpretation sentence including the fact inconsistent with the relationship indicated in the relationship document (Edwards, P[0066]: “Typically, this enables the prompt assessment unit 110b to select the one or more candidate prompt-templates that produce the optimum results, i.e. optimum test prompts.” (Templates that produce inconsistent outputs are not selected; this, inconsistent sentences are excluded from final outputs)).
Regarding claim 3, The combination of Edwards, Goyal, and Chan teaches the computer system according to claim 2.
Chan further teaches:
wherein the computer system
selects a plurality of target features based on magnitudes of the contribution values of the plurality of features (Chan, P[124]: “the feature importance graph may be updated to depict the most important features. The most important features may be the most important features associated with a global surrogate model. The most important features may be the most important features associated with the selected observation point.” (A plurality of target features (the “most important features”) are selected based on magnitudes of the contribution values (the importance associated with the global or selected point))), and
generates the group having the plurality of target features as elements (Chan, P[123]-P[124]: “dashboard 1100 includes a K-LIME linear model graph, a feature importance graph, a surrogate model decision tree, and a partial dependence graph… the feature importance graph may be updated to depict the most important features.” (The feature importance graph displayed within dashboard 1100 necessarily aggregates a plurality of selected features and presents them together for analysis, which constitutes generating a group having the plurality of target features as elements).
Regarding claim 4, The combination of Edwards, Goyal, and Chan teaches the computer system according to claim 3.
Edwards further teaches:
comprising: an interface configured to correct the target feature (Edwards, P[0094]: “The terminal 402b provides a means by which an operative can oversee operation of the prompt-template generating system 101b and labelled data generating system 112b.” (An interface (terminal 402b) is provided and configured to correct the target feature by providing a means for an operative to oversee the systems and manage the data generating system, positioned to identify and correct the target feature or property assigned to the data.);
an interface configured to select the instruction sentence to be input to the large-scale language model (Edwards, P[0045]: “prompt-template generating LLM 104b to generate a number of candidate prompt-templates. These candidate prompt-templates can then be used to generate a plurality of test prompt” (Candidate prompt selection is mediated via the interface; operative can select instruction sentences for LM)); and
an interface configured to input or correct the relationship (Edwards, P[0049]: “provides a means by which an operative can oversee operation” (Manual correction/interaction can include modifying relationships between features and template outputs)).
Regarding claim 5, claim 5 recites the method claim corresponding to claim 1 and is rejected for the same reasons as above.
Regarding claim 6, claim 6 recites the method claim corresponding to claim 2 and is rejected for the same reasons as above.
Regarding claim 7, claim 7 recites the method claim corresponding to claim 3 and is rejected for the same reasons as above.
Regarding claim 8, claim 8 recites the method claim corresponding to claim 4 and is rejected for the same reasons as above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHASHIDHAR S MANOHARAN whose telephone number is (571)272-6772. The examiner can normally be reached M-F 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHASHIDHAR SHANKAR MANOHARAN/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655