Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
2. This communication is responsive to the amendment filed on 06/25/2026.
3. Claims 1-10, 16-18 and 52-53 are currently pending in this Office action. This action is made Final.
Claim Rejections - 35 USC § 112
4. The 35 U.S.C. 112 rejections made in the prior Office action are withdrawn.
Claim Rejections - 35 USC § 101
5. 35 U.S.C. 101 rejections made in the prior Office action are withdrawn.
Claim Rejections - 35 USC § 103
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claims 1-6, 8, 16-18 and 52-53 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. 2025/0173363 (hereinafter Madisetti) in view of U.S. 2020/0034665 (hereinafter Ghanta), and further in view of U.S. 2021/0390455 (hereinafter Schierz).
Regarding claims 1 and 52-53, Madisetti discloses a generative AI system development method, the method comprising:
constructing, by one or more processors, a plurality of generative AI systems, wherein constructing the plurality of generative AI systems includes executing at least one modeling blueprint ([0022]; “…A family of generative models, LLM1, trained with a data set T1, can be represented as h-LLM1, while a family of models, LLM2, trained with data set T2, can be represented as h-LLM12. Further, a family of models, LLM1, trained with a data set T3, can be represented as h-LLM35. The combination of models and their training sets (T1 could be a subset of T3, for example, or they can be different) may be used in our proposed invention and they are referred to as h-LLMs, throughout…”);
providing, by the one or more processors, a plurality of queries to each generative AI system in the plurality of generative AI systems, the plurality of queries being part of an evaluation dataset ([0030 and 0143-0144]; “…The relevant context allows the LLM to respond to the user query (including through use of the derived queries) even though the LLM may not be trained with the data from the knowledge base. The retrievers 2702 are used to retrieve the relevant context from the knowledge base 2700. The retrievers 2702 may use different search and similarity matching strategies to search and retrieve the documents…”);
during processing of the plurality of queries by each generative AI system, monitoring values of one or more quantitative metrics ([0023 and 0171-0172]; “…An evaluation and fine-tuning module 3824 includes a performance analyzer which is operable to monitor system-wide metrics and evaluate the response quality, and a model fine-tuner which is operable to adjust the system parameters for optimization”);
providing data indicating the values [i.e., the SCORE-RAG] of the one or more quantitative metrics for each generative AI system ([0023 and 0176]; “…This multi-faceted approach allows CORE-RAG to process a wide range of input types and also generate appropriate and context-aware multi-modal outputs. A User 4026 sends a query 4028 (comprising multi-modal inputs) to the SCORE-RAG system 4002 and received a multi-modal response 4030. The multimodal capability of the SCORE-RAG system 4002 enhances the system's utility across diverse fields such as education, research, software development, multimedia content creation, and data analysis, for instance”).
While Madisetti discloses the features of utilizing the generative AI system as explained above, the reference does not explicitly disclose the features of determining, by the one or more processors and for each generative Ai system, an aggregate value of at least one quantitative metric of the one or more quantitative metrics, the aggregate value aggregated across the plurality of queries of the evaluation dataset; providing, by the one or more processors [for display by a user device], data indicating the aggregate value of the at least one quantitative metric for each [generative] AI system. However, Ghanta discloses that “…The other statistics may include confidence metrics, accuracy metrics, precision metrics, and/or the like. Threshold values may be predefined to determine whether the metrics satisfy a predetermined value to indicate the suitability of the second machine learning algorithm…” ([0089]). Ghanta additionally discloses that “In certain embodiments, the secondary validation module 308 determines the suitability of an ensemble of second machine learning algorithms (e.g., a combination of two or more machine learning algorithms) for predicting the performance or accuracy of the predictions of the first machine learning algorithm for an inference data set. The secondary validation module 308, in one embodiment, may generate ensembles that include different combinations of machine learning algorithms/models to determine which ensemble is the best fit or satisfies a suitability threshold for analyzing the predictive performance of the first machine learning algorithm/model…” ([0090]) and it would have been obvious for one with ordinary skill in the art to utilize the teachings of Ghanta in the system of Madisetti in view of the desire to enhance the generative artificial intelligence process by utilizing the machine learning algorithm validation metrics resulting in improving the efficiency of generating the response contents.
Madisetti in view of Ghanta do not explicitly disclose the features of providing, by the one or more processors for display by a user device, data indicating for each generative AI system; and providing, by the one or more processors for display by the user device, a recommendation regarding use or non-use of at least one generative AI system included in the plurality of generative AI systems. However, Schierz discloses that “…For example, the adaptive drift learner 126 can be trained using the test output to predict a suitable (e.g., optimal) binning strategy and/or drift metric. Once trained, the adaptive drift learner 126 can receive as input one or more characteristics or features for a set of data (e.g., length, distribution, minimum, maximum, mean, skewness, number of unique values, or any combination thereof) and provide as output a recommended binning strategy and/or drift metric…” ([0028, 0115 and 0120]). Schierz further discloses that “…Further, new segmentation strategies can be tried, models can be developed for the new segmentation strategies, and the models can be evaluated for performance (e.g., accuracy and/or efficiency). In some
PNG
media_image1.png
1703
1277
media_image1.png
Greyscale
examples, recommendations for new segmentation strategies can be sent to users for feedback or approval. Additionally or alternatively, the systems and methods may evaluate alternative means
of assigning entities in the time series to segments or clusters, based on signals of drift and performance measured after deployment…” ([0136]) and “… If the classifier (or other AI
model) can successfully tell the two datasets apart, then this can imply that the drift has had a system-wide effect. Once the impact of the drift has been assessed at both an individual and systemic level, a user of the system 100 can be alerted with a recommended course of action, or other corrective action can be taken or facilitated, as described herein” ([0117, 0136 and 0146]; figs. 4A and 4B as shown above) and it would have been obvious for one with ordinary skill in the art to utilize the teachings of Schierz in the modified system of Madisetti in view of the desire to enhance the generative artificial intelligence process by utilizing the monitoring and managing machine learning model schemes resulting in improving the efficiency of generating the outputs. Madisetti additionally discloses a computer-readable storage medium, and a computer-readable medium executed by one or more processors ([0198]).
Regarding claim 2, Madisetti in view of Ghanta and Schierz disclose the method wherein each generative AI system in the plurality of generative AI systems is configured to operate as a chat bot, a natural language interface to a knowledge base, or content generation engine (Madisetti: [0139]; the supervised or labeled learning data).
Regarding claim 3, Madisetti in view of Ghanta and Schierz disclose the method wherein each generative AI system in the plurality of generative AI systems is a retrieval-augmented generation (RAG)-based generative AI system (Madisetti: [0020 and 0176]).
Regarding claim 4, Madisetti in view of Ghanta and Schierz disclose the method wherein each generative AI system in the plurality of generative AI systems includes a knowledge base, a prompt construction facility, and a generative model (Madisetti: [0123-0124]).
Regarding claim 5, Madisetti in view of Ghanta and Schierz disclose the method wherein the constructing of each generative AI system in the plurality of generative AI systems is performed based on a set of values of a set of hyperparameters, and wherein the respective set of hyperparameter values corresponding to each generative AI system determines one or more attributes of the knowledge base, the prompt construction facility, or the generative model included in the generative AI system (Madisetti: [0124 and 0126]) and (Ghanta: [0072]). Therefore, the limitations of claim 5 are rejected also rejected in the analysis of claim 1, and the claim is rejected on that basis.
Regarding claim 6, Madisetti in view of Ghanta and Schierz disclose the method wherein the one or more attributes of the knowledge base include a type of encoder used to create a plurality of embeddings of the knowledge base, the plurality of embeddings representing a plurality of portions of source data (Madisetti: [0006-0010]).
Regarding claim 8, Madisetti in view of Ghanta and Schierz disclose the method wherein the one or more attributes of the generative model include a type of the generative model (Madisetti: [0010-0012]).
Regarding claim 16, Madisetti in view of Ghanta and Schierz disclose the method wherein the evaluation dataset is a synthetic evaluation dataset (Madisetti: [0139]) and (Schierz: [0136]). Therefore, the limitations of claim 16 are rejected also rejected in the analysis of claim 1, and the claim is rejected on that basis.
Regarding claim 17, Madisetti in view of Ghanta and Schierz disclose the method further comprising constructing the synthetic evaluation dataset (Madisetti: [0117]) and (Schierz: [0136]). Therefore, the limitations of claim 17 are rejected also rejected in the analysis of claim 1, and the claim is rejected on that basis.
Regarding claim 18, Madisetti in view of Ghanta and Schierz disclose the method wherein, for each generative AI system in the plurality of generative AI systems, at least one aggregate quantitative metric includes (Ghanta: [0089-0090]):
a factual accuracy metric indicating an extent to which completions generated by the respective generative AI system in response to the plurality of queries are factual;
a faithfulness metric indicating an extent to which the completions generated by the respective generative AI system include hallucinated information;
a grounded-ness metric indicating an extent to which the completions generated by the respective generative AI system are based on context data extracted from the knowledge base of the respective generative AI system;
a toxicity metric indicating an extent to which the completions generated by the respective generative AI system include toxic content;
a latency metric indicating a latency associated with the processing of the queries and/or generation of the completions by the respective generative AI system;
a token count metric derived from a number of tokens included in the completions generated by the respective generative AI system; and/or
a cost metric indicative a cost incurred by using the generative model of the respective generative AI system to generate the completions (Madisetti: [0022, 0146 and 0262]; the token counts and the factual accuracy) and (Schierz: [0136]). Therefore, the limitations of claim 18 are rejected also rejected in the analysis of claim 1, and the claim is rejected on that basis.
9. Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Madisetti in view of Ghanta and Schierz, and further in view of U.S. 2024/0370764 (hereinafter Goyal).
Regarding claim 7, Madisetti in view of Ghanta and Schierz do not explicitly disclose the method wherein the one or more attributes of the prompt construction facility include (i) a process by which the prompt construction facility identifies one or more embeddings in the knowledge base matching an embedding representing a query, (ii) a process by which source data corresponding to the identified one or more embeddings is added to a constructed prompt, and/or (iii) a configuration of a prompt template used to construct the constructed prompt. However, Goyal discloses that “At 505, a prompt is configured. For example, a prompt for interfacing with a selected AI service, such as with a generative AI service utilizing the LLM selected at 501 is configured. In some embodiments, the prompt is configured as a prompt template. Using the configured prompt template, a prompt can be generated using predefined text portions of the prompt and dynamic portions of the prompt where the dynamic portions can change based on the specific AI request and/or context of the AI request…” ([0015 and 0046]) and it would have been obvious for one with ordinary skill in the art to utilize the teachings of Goyal in the modified system of Madisetti in view of the desire to enhance the generative artificial intelligence process by utilizing the generative AI configuration scheme resulting in improving the efficiency of generating the outputs.
10. Claims 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Madisetti in view of Ghanta and Schierz, and further in view of U.S. 2021/0224588 (hereinafter Sikand).
Regarding claim 9, Madisetti in view of Ghanta and Schierz disclose the method wherein the plurality of generative AI systems include a first generative AI system, and wherein the method further comprises: providing, by the one or more processors for display by a user device, a visual representation of the first generative AI system (Madisetti: [0022]).
The references do not explicitly disclose the features of wherein a visual representation of an embedding space of the knowledge base. However, such features are well known in the art as disclosed by Sikand ([0072 and 0075]; figs. 4a-4b) and it would have been obvious for one with ordinary skill in the art to utilize the teachings of Sikand in the modified system of Madisetti in view of the desire to enhance the generative artificial intelligence process by utilizing the specific data type of visualization resulting in improving the efficiency of generating the outputs.
Regarding claim 10, Madisetti in view of Ghanta, Schierz and Sikand disclose the method wherein the visual representation of the embedding space includes a plurality of topic labels indicating the topics of a respective plurality of clusters of embeddings (Madisetti: [0163 and 0172]) and (Sikand: figs. 4a-4b). Therefore, the limitations of claim 10 are rejected also rejected in the analysis of claim 9, and the claim is rejected on that basis.
Response to Arguments
11. Applicant’s arguments have been considered but are deemed to be moot in view of new grounds of rejection presented in this Office action.
Conclusion
12. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MONICA M PYO whose telephone number is (571)272-8192. The examiner can normally be reached Monday-Friday 8am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, APU MOFIZ can be reached at 571-272-4080. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MONICA M PYO/Primary Examiner, Art Unit 2161
Prosecution Insights