DETAILED ACTION
Notice of Pre-AIA or AIA Status
● The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
● This action is responsive to the following communication: US Patent Application filed on 2/13/2024.
● Claims 1-20 are currently pending.
Information Disclosure Statement
● The information disclosure statement (IDS) submitted on 2/26/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Mangalam et al (US 20250217603).
Regarding claim 1, Mangalam discloses a method, comprising:
extracting (extracting, par. 5), by a device, topics from queries (LLM identifies and extracts the concepts from the entire transcript, thereby providing complete extraction of the concepts of interest., par. 5) submitted into a large language model;
determining, by the device, a quality (based on categories scores, pars. 48-49) of outputs from the large language model in response to the queries;
assessing, by the device, per-topic performance (topic of categories/interests, pars. 48-49) of the large language model across the topics queried into the large language model based on the quality of the outputs (based on scores, pars. 48-49);
determining, by the device, one or more underperforming topics (determine which categories has low scores, pars. 48-49) for the large language model based on the per-topic performance of the large language model across the topics [This process continues until the final output layer generates scores for each possible category, sub-category, and/or other class. The final output layer of the network represents the predicted category for the input text. Each neuron in this layer represents a specific category, and its activation level reflects the network's confidence in that category being the correct one. Connections between the last hidden layer and the output layer may determine how the network combines the extracted features and their relationships to form a final prediction. The weights of these connections dictate the relative importance of different features and their interactions in influencing the final category score]; and
performing, by the device, one or more mitigation actions (mitigation actions, pars. 12, 68) based on the one or more underperforming topics (categories/interests, pars. 2, 5) for the large language model.
Regarding claim 2, Magalam further discloses the method of claim 1, wherein performing the one or more mitigation actions comprises: displaying a user interface (user interface, par. 91) that indicates the per-topic performance of the large language model across the topics as a heatmap (pars. 65-69); and delineating (classify topics into categories/subcategories based upon scores, pars. 22, 41) the one or more underperforming topics specifically within the heatmap (maps, pars. 65-69).
Regarding claim 3, Magalam further discloses the method of claim 2, wherein the user interface is organized to correlate the topics queried into the large language model against documents in a knowledge base (knowledge base, par. 28) from which the large language model has been trained for those topics.
Regarding claim 4, Magalam further discloses the method of claim 2, further comprising: indicating, within the user interface, which of the topics queried into the large language model have insufficient information (missing data/information from the input query, pars. 5, 28) to verify performance of the large language model.
Regarding claim 5, Magalam further discloses the method of claim 2, further comprising: indicating, within the user interface, specific information to add to a knowledge base to retrain (retrain, par. 12) the large language model for the one or more underperforming topics.
Regarding claim 6, Magalam further discloses the method of claim 2, further comprising: indicating, within the user interface, three or more tiers (categories classes, fig. 7) of per-topic performance of the large language model across the topics (intent category classes, fig. 7), at least one of the three or more tiers corresponding to the one or more underperforming topics.
Regarding claim 7, Magalam further discloses the method of claim 2, further comprising: indicating, within the user interface, one or both of a) a number of chunks needed to be provided per query from a knowledge base from which the large language model has been trained for the topics to reach a satisfactory output, or b) a number of re-queries needed to reach a satisfactory output.
Regarding claim 8, Magalam further discloses the method of claim 1, wherein performing the one or more mitigation actions comprises: detecting a new query (figs. 6-7) submitted into the large language model that relates to one of the one or more underperforming topics; and providing additional context (figs. 6-7) for the new query.
Regarding claim 9, Magalam further discloses the method of claim 1, wherein performing the one or more mitigation actions comprises: supplying, into a knowledge base used to train the large language model, additional content related to the one or more underperforming topics; and retraining (retrain, par. 12) the large language model with the additional content.
Regarding claim 10, Magalam further discloses the method of claim 1, wherein performing the one or more mitigation actions comprises: detecting a new query (figs. 6-7) submitted into the large language model (figs. 6-7) that relates to one of the one or more underperforming topics; and tagging an output from the large language model for the new query with a confidence threshold (par. 34) indicative of comparatively low confidence.
Regarding claim 11, Magalam further discloses the method of claim 1, wherein determining the quality of the outputs from the large language model in response to the queries comprises: obtaining one of either explicit user feedback, implicit user feedback (user feedback, par. 11), or both.
Regarding claim 12, Magalam further discloses the method of claim 1, wherein determining the quality of the outputs from the large language model in response to the queries comprises: obtaining automated verification (pars. 34-35) of the outputs based on an evaluation of the outputs against one more quality assessment criteria.
Regarding claim 13, Magalam further discloses the method of claim 1, wherein determining the quality of the outputs from the large language model in response to the queries comprises: detecting usage of Retrieval-Augmented Generation for one or more particular outputs (pars.34-35).
Regarding claim 14, Magalam further discloses the method of claim 1, wherein determining the quality of the outputs from the large language model in response to the queries comprises: determining performance drift indicative of a loss (par. 51) of knowledge within the large language model based on an increase over time in either uncertainty of the outputs or inconsistencies (par. 34) of the outputs or both.
Regarding claims 15-20 recite limitations that are similar and in the same scope of invention as to those in claims 1-14 above and/or combination thereof; therefore, claims 15-20 are rejected for the same rejection rationale/basis as described in claims 1-14.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THIERRY L PHAM whose telephone number is (571)272-7439. The examiner can normally be reached M-F, 11-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at 571-272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THIERRY L PHAM/ Primary Examiner, Art Unit 2654
Prosecution Insights