DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments filed 1/22/2026 have been accepted and considered in this office action. Claims 1-20 have been amended.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot in view of new grounds of rejection necessitated by the applicant’s amendments to the claims.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 6, 9, 13, 16, 20 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 2, 9, and 16, the phrase "the predetermined interval comprises every hour and every week" renders the claims indefinite because there is a logical conflict between this interval comprising both every hour AND every week. Examiner suggests changing this to every hour or every week.
Regarding claims 6, 13, and 20, the phrase “or if it needs to be fine-tuned” renders the claim indefinite because it is unclear whether it is the workflow that needs to be fine-tuned or the response itself. Examiner suggests replace “it” with the specific component that needs to be fine-tuned.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 6, 8, 9, 13, 15, 16, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Amini et al. (hereinafter Amini) (US 2024/0127153), in view of Swope (US 11,900,229), further in view of Silver et al. (hereinafter Silver) (US 2025/0199786), furthest in view of Pathak et al. (hereinafter Pathak) (US 11741302 B1) (see attached copy for paragraph numbers)
Regarding claim 1 Amini teaches:
A computer-implemented method for governing responses generated by a language learning model (LLM) (Abstract, P[0101] Large Language Models) the computer-implemented method comprising:
setting up an initial risk assessment framework for a model of an LLM, wherein setting up further comprises (Amini, Abstract, “developing and implementing framework for quantifying risk in machine learning models”):
defining a reference set of inputs and output pairs for the LLM (e.g. a labeled data set of n inputs can output pairs for risk measurement; [0053]; further note computer system determines various parameters defined by the user’s list of metrics; [0096]) wherein the reference set of inputs and output pairs are actual inputs and reference outputs (e.g. user input includes inputs and ground truth labels [“reference”]; [0106]);
Amani does not explicitly disclose:
defining a set of metadata associated with the reference set of input and output pairs; and
assigning the set of metadata to each pair of the reference set of inputs and output pairs.
passing all inputs from the reference set of inputs and output pairs to the LLM ,and collecting all outputs from the reference set of inputs and output pairs at a predetermined interval
evaluating and producing an evaluation metric of all the inputs and all of the outputs based on one or more policies, further comprising
generating new outputs and comparing the new outputs against the reference set of inputs and output pairs; and
generating scores for the new outputs and adding the scores to a historical data for future analysis
summarizing results of the evaluation metric across all of the reference set of inputs and output pairs
notifying one or more users when the results of the evaluation metric exceed a predetermined metric threshold; and
saving data associated with the result of the evaluation metric and all of the reference set of inputs and output pairs to a data repository;
However, in a related field of endeavor (e.g. supervised learning in machine learning and language systems; see Swope Fig. 1 and col. 9 line 23 – col. 9 line 4; and see Amani [0053]-[0055]) Swope teaches:
defining a set of metadata (e.g. classify elements of training data to type of metadata; col. 20 lines 50 – 53) associated with the reference set of input and output pairs (e.g. supervised machine learning process includes classification algorithms and is used to determine relations between inputs and outputs; col. 21 lines 59 – 65)
assigning the set of metadata to each pair of the reference set of inputs and output pairs (e.g. assigning, generating, classifying, training using metadata; see Column 13, Lines 9-20);
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Amini to incorporate the teachings of Swope. Doing so would have provided the improved data structures and techniques of Swope to the systems and methods of Amini and thus would have address the known difficulty of managing metadata in an efficient manner (col. 1 lines 13 – 47 of Swope) allowing the combination to identify and locate missing data and reducing the need for human intervention allowing the combination to operate more efficiently and autonomously (col. 8 line 65 – col. 9 line 4)
The combination of Amini and Swope further discloses or makes obvious:
defining a set of evaluation criteria (Amini P[0090], Multiple criteria can be used to identify what part of the model need to be transformed);
assigning the set of evaluation criteria to organizational risk framework (Amini Abstract, different automatic risk assessment algorithms, Identifying potential model failures based on risk values, P[0090] Criteria can be used to identify what parts of the model need to be transformed); and
associating the set of metadata to the set of evaluation criteria (Amini’s Fig 1B. Item 116 and 118, Determine, based on output of the risk-aware model, epistemic uncertainty associated with model’s input data, now modified by the metadata features of Swope detailed in the citations above).
evaluating the initial risk assessment framework and the model of the LLM after a predetermined interval, further comprising: (Amini, P[0185], risk assessment at predetermined time):
Amini, in view of Swope, does not disclose:
passing all inputs from the reference set of inputs and output pairs to the LLM, and collecting all outputs from the reference set of inputs and output pairs at a predetermined interval
evaluating and producing an evaluation metric of all the inputs and all of the outputs based on one or more policies, further comprising
generating new outputs and comparing the new outputs against the reference set of inputs and output pairs; and
generating scores for the new outputs and adding the scores to a historical data for future analysis
summarizing results of the evaluation metric across all of the reference set of inputs and output pairs
notifying one or more users when the results of the evaluation metric exceed a predetermined metric threshold; and
saving data associated with the result of the evaluation metric and all of the reference set of inputs and output pairs to a data repository;
However, Silver discloses:
passing all inputs from the reference set of inputs and output pairs to the LLM ,and collecting all outputs from the reference set of inputs and output pairs at a predetermined interval (e.g. receiving inputs from a user for the model... various characteristics used to generate outputs...metadata characteristic relates inputs, outputs to the model... and monitoring the model at regular intervals... Silver, P[0030]; further note applying an LLM to process; Silver, P[0032]; now as applied to the reference data as explained above in the reasoned combination of Amini and Swope);
evaluating and producing an evaluation metric of all the inputs and all of the outputs based on one or more policies, further comprising (P[0030], risk assessment associated with model):
and generating scores for the new outputs and adding the scores to a historical data for future analysis (Silver, P[0030] (versioning/historical record of data maintain for future analysis), P[0064] and P[0066]: (generating output that includes score (which can include accuracy, precision f1 score, etc.)));
summarizing results of the evaluation metric across all of the reference set of inputs and output pairs (Silver, P[0066], “or other validation criteria” evaluation metric; “validation assessment based on generated output that could include a report, score” is a summarization of results of evaluation metric); and
notifying one or more users when the results of the evaluation metric exceed a predetermined metric threshold ((Silver, P[0070] validation assessment may be presented to a user through the user interface, send as notifications; Silver, P[0066] “These filters might include keywords, threshold values” is the predetermined metric threshold); and
saving data associated with the result of the evaluation metric and all of the reference set of inputs and output pairs to a data repository (Silver, P[0058], “system may retrieve necessary data related to first model from project repository or database”, Silver, P[0030], “tracking different versions of the model and changes made over time and maintains a historical record of alterations, updates, or modifications”, refers to saving data and documenting changes, Silver, P[0032], “versioning”, refers to saving data and information on changes/updates).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Amini in view of Swope to incorporate the teachings of Silver. Doing so would have provided the improved data structures and techniques of Swope and the efficient organization of data structures of Silver (P[0002] of Silver) to the systems and methods of Amini and thus would have address the known difficulty of managing metadata in an efficient manner (col. 1 lines 13 – 47 of Swope) allowing the combination to identify and locate missing data and reducing the need for human intervention allowing the combination to operate more efficiently and autonomously (col. 8 line 65 – col. 9 line 4 of Swope).
The combination of Amini, Swope, and Silver does not disclose
generating new outputs and comparing the new outputs against the reference set of inputs and output pairs;
However, Pathak discloses:
generating new outputs and comparing the new outputs against the reference set of inputs and output pairs (Pathak, P(40), “An NLP model may go through multiple iterations of testing that include processing multiple reference inputs and comparing the textual output with reference textual output.” (textual output reads on generated output, and reference input and reference output reads on reference input/output pairs being compared with));
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Amini in view of Swope to incorporate the teachings of Silver and Pathak. Doing so would have provided the reference set comparisons for accuracy of Pathak (Pathak, P(40)) with the improved data structures and techniques of Swope and the efficient organization of data structures of Silver (P[0002] of Silver) to the systems and methods of Amini and thus would have address the known difficulty of managing metadata in an efficient manner (col. 1 lines 13 – 47 of Swope) allowing the combination to identify and locate missing data with even higher accuracy and reducing the need for human intervention allowing the combination to operate more efficiently and autonomously (col. 8 line 65 – col. 9 line 4 of Swope).
Regarding claim 2, Amini in view of Swope, Silver, and Pathak discloses the computer-implemented method according to claim 1.
Silver further discloses:
The computer-implemented method of claim 1, wherein the predetermined interval comprises every hour and every week (Amini, P[0185], “predetermined time intervals”).
Regarding claim 6, Amini in view of Swope, Silver, and Pathak discloses the computer-implemented method according to claim 1.
Silver further discloses:
wherein notifying one or more users further comprising:
generating one or more workflows from a GRC (governance risk compliance) system (([0006] risk management guidelines, "formats these extracted rules to...software approaches"), wherein the one or more workflows contains details relating the inputs and outputs ([0030], a collaboration and workflow management characteristic (e.g., a characteristic that relates to enabling collaboration among different stakeholders involved in managing the model, allowing for efficient workflow management and communication); and
reviewing the one or more workflows to determine whether the responses is within specification or if it needs to be fine-tuned. ([0030], a collaboration and workflow management characteristic (e.g., a characteristic that relates to enabling collaboration among different stakeholders involved in managing the model, allowing for efficient workflow management and communication, P[0064] “fine-tuning”).
Regarding claim 8, claim 8 recites the computer program product corresponding to the method presented in claim 1 and is rejected under the same grounds stated above.
Additionally, the combination further discloses or makes obvious:
A computer program product (Amini [0200], [0201]) for governing responses generated by a language learning model (LLM) (Amini Abstract, P[0101] Large Language Models) the computer program product comprising:
one or more computer readable storage media and program instructions stored on the one or more computer readable storage media (e.g. Amani’s system can be realized in hardware/software any combination thereof [0200] and further note computer/machine readable medium; [0201]), the program instructions comprising the steps of [the method of claim 1] (see rejection of method claim 1 above).
Regarding claim 9, claim 9 recites the computer program product corresponding to the method presented in claim 2 and is rejected under the same grounds stated above.
Regarding claim 13, claim 13 recites the computer program product corresponding to the method presented in claim 6 and is rejected under the same grounds stated above.
Regarding claim 15, claim 15 recites the computer system corresponding to the method presented in claim 1 and is rejected under the same grounds stated above.
Additionally, the combination further discloses or makes obvious:
A computer system (Amini [0200], [0201]) for governing responses generated by a language learning model (LLM) (Amini Abstract, P[0101] Large Language Models) the computer system comprising:
one or more computer processors (e.g. Amini’s system can be realized in hardware/software any combination thereof [0200] and further note computer/machine readable medium; [0201]));
one or more computer readable storage media and program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors (e.g. Amini’s system can be realized in hardware/software any combination thereof [0200] and further note computer/machine readable medium; [0201]), the program instructions comprising the steps of [the method of claim 1] (see rejection of method claim 1 above).
Regarding claim 16, claim 16 recites the computer system corresponding to the method presented in claim 2 and is rejected under the same grounds stated above.
Regarding claim 20, claim 20 recites the computer system corresponding to the method presented in claim 6 and is rejected under the same grounds stated above.
Claim(s) 3, 7, 10, 14, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Amini et al. (hereinafter Amini) (US 2024/0127153), in view of Swope (US 11,900,229), further in view of Silver et al. (hereinafter Silver) (US 2025/0199786), furthest in view of Pathak et al. (hereinafter Pathak) (US 11741302 B1) (see attached copy for paragraph numbers) and Sampath et al. (hereinafter Sampath) (US 2022/0400094).
Regarding claim 3, Amini in view of Swope, Silver, and Pathak discloses the computer-implemented method according to claim 1.
Amini in view of Swope, Silver, and Pathak does not disclose:
wherein the set of evaluation criteria further comprises, an exact match required, a semantic match required and the response must not contain a specified word and/or phrase.
However, Sampath discloses:
wherein the set of evaluation criteria further comprises, an exact match required, a semantic match required and the response must not contain a specified word and/or phrase (P[0052], calculate scores for: syntagmatic complexity (e.g., based on word length); paradigmatic complexity (e.g., based on variety of grammatical categories); organizational complexity (e.g., based on variety of component arrangement, phonotactic restrictions, and word order); and/or hierarchic complexity (e.g., based on recursion and lexical-semantic hierarchies) of language contained).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Amini in view of Swope to incorporate the teachings of Silver, Pathak, and Sampath. Doing so would have provided the improved data structures and techniques of Swope with efficient similarity score detection and response of Sampath (Abstract, lines 10-15 of Sampath) with the reference set comparisons for accuracy of Pathak (Pathak, P(40)) and the improved data structures and techniques of Swope and the efficient organization of data structures of Silver (P[0002] of Silver) to the systems and methods of Amini and thus would have address the known difficulty of managing metadata in an efficient manner (col. 1 lines 13 – 47 of Swope) allowing the combination to identify and locate missing data with even higher accuracy and reducing the need for human intervention allowing the combination to operate more efficiently and autonomously (col. 8 line 65 – col. 9 line 4 of Swope).
Regarding claim 7, Amini in view of Swope, Silver, and Pathak discloses the computer-implemented method according to claim 1.
Amini in view of Swope, Silver, and Pathak does not disclose:
wherein associating the set of metadata to the set of evaluation criteria can include, a high priority is associated to exact match, a low priority and low sensitivity content are required a semantic match and sensitive contents must not contain a certain letter and/or phrase.
Sampath discloses:
wherein associating the set of metadata to the set of evaluation criteria can include, a high priority is associated to exact match, a low priority and low sensitivity content are required a semantic match and sensitive contents must not contain a certain letter and/or phrase ((P[0052], calculate scores for: syntagmatic complexity (e.g., based on word length); paradigmatic complexity (e.g., based on variety of grammatical categories); organizational complexity (e.g., based on variety of component arrangement, phonotactic restrictions, and word order); and/or hierarchic complexity (e.g., based on recursion and lexical-semantic hierarchies) of language contained), P[0133] In another example implementation, the system 100 can execute Block S210 of the second method S200 by generating both a natural language model and a metadata model for an email user or group of users in an organization, (e.g., for high priority personnel or personnel in sensitive positions such as executive leadership, finance, engineering, and human resources).
Regarding Claim 10, claim 10 recites the computer program product corresponding to the method presented in claim 3 and is rejected under the same grounds stated above.
Regarding Claim 14, claim 14 recites the computer program product corresponding to the method presented in claim 7 and is rejected under the same grounds stated above.
Regarding Claim 17, claim 17 recites the computer system corresponding to the method presented in claim 3 and is rejected under the same grounds stated above.
Claim(s) 4, 11, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Amini et al. (hereinafter Amini) (US 2024/0127153), in view of Swope (US 11,900,229), further in view of Silver et al. (hereinafter Silver) (US 2025/0199786), furthest in view of Pathak et al. (hereinafter Pathak) (US 11741302 B1) (see attached copy for paragraph numbers) and Lantzman (US 2023/0281482).
Regarding claim 4, Amini in view of Swope, Silver, and Pathak discloses the computer-implemented method according to claim 1.
Amini in view of Swope, Silver, and Pathak does not disclose:
wherein assigning the set of metadata to each pair of the reference set of inputs and output pairs further comprises:
assessing the set of metadata by using a rules engine or classifier.
However, Lantzman discloses:
wherein assigning the set of metadata to each pair of the reference set of inputs and output pairs further comprises:
assessing the set of metadata by using a rules engine or classifier (P[0003] “metadata” “providing a rules engine therein”, “parsing, by the rules engine, the model metadata file to determine that the parameter value is associated with the rule criteria”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Amini in view of Swope to incorporate the teachings of Silver, Pathak, and Lantzman. Doing so would have provided the improved data structures and techniques of Swope with the efficient rules evaluation (P[0002] lines 11-14 of Lantzman) of Lantzman with the reference set comparisons for accuracy of Pathak (Pathak, P(40)) and the improved data structures and techniques of Swope and the efficient organization of data structures of Silver (P[0002] of Silver) to the systems and methods of Amini and thus would have address the known difficulty of managing metadata in an efficient manner (col. 1 lines 13 – 47 of Swope) allowing the combination to identify and locate missing data with even higher accuracy and reducing the need for human intervention allowing the combination to operate more efficiently and autonomously (col. 8 line 65 – col. 9 line 4 of Swope).
Regarding Claim 11, claim 11 recites the computer program product corresponding to the method presented in claim 4 and is rejected under the same grounds stated above.
Regarding Claim 18, claim 18 recites the computer system corresponding to the method presented in claim 4 and is rejected under the same grounds stated above.
Claim(s) 5, 12, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Amini et al. (hereinafter Amini) (US 2024/0127153), in view of Swope (US 11,900,229), further in view of Silver et al. (hereinafter Silver) (US 2025/0199786), furthest in view of Pathak et al. (hereinafter Pathak) (US 11741302 B1) (see attached copy for paragraph numbers) and Sharma (US 11,714,842).
Regarding claim 5, Amini in view of Swope, Silver, and Pathak discloses the computer-implemented method according to claim 1.
Amini in view of Swope, Silver, and Pathak does not disclose:
Wherein the set of metadata comprises, prioritization, sensitive and topic areas.
Sharma discloses:
Wherein the set of metadata comprises, prioritization, sensitive and topic areas (Abstract, Metadata associated with file objects is analyzed to estimate, for each file object, likelihood that the file object includes sensitive data. The estimates are used to prioritize the file objects for analysis of the file objects’ content to determine the objects include data deemed to be of a sensitive nature.).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Amini in view of Swope to incorporate the teachings of Silver, Pathak, and Sharma. Doing so would have provided the improved data structures and techniques of Swope with the efficient word similarity detection of Sharma (Abstract, lines 10-12 of Sharma) with the reference set comparisons for accuracy of Pathak (Pathak, P(40)) and the improved data structures and techniques of Swope and the efficient organization of data structures of Silver (P[0002] of Silver) to the systems and methods of Amini and thus would have address the known difficulty of managing metadata in an efficient manner (col. 1 lines 13 – 47 of Swope) allowing the combination to identify and locate missing data with even higher accuracy and reducing the need for human intervention allowing the combination to operate more efficiently and autonomously (col. 8 line 65 – col. 9 line 4 of Swope).
Regarding Claim 12, claim 12 recites the computer program product corresponding to the method presented in claim 5 and is rejected under the same grounds stated above.
Regarding Claim 19, claim 19 recites the computer system corresponding to the method presented in claim 5 and is rejected under the same grounds stated above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHASHIDHAR S MANOHARAN whose telephone number is (571)272-6772. The examiner can normally be reached M-F 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHASHIDHAR SHANKAR MANOHARAN/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655