DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Applicant’s amendment dated 09/26/2024 has been received and made of record.
Application 18/768,955 has an effective filing date of filed 07/10/2024.
Claims 1-20 are currently pending in Application 18/768,955.
Double Patenting
Claims 1-20 of this application are patentably indistinct from claims 1-20 of Application No. 19/380,832. Pursuant to 37 CFR 1.78(f), when two or more applications filed by the same applicant or assignee contain patentably indistinct claims, elimination of such claims from all but one application may be required in the absence of good and sufficient reason for their retention during pendency in more than one application. Applicant is required to either cancel the patentably indistinct claims from all but one application or maintain a clear line of demarcation between the applications. See MPEP § 822.
A rejection based on double patenting of the “same invention” type finds its support in the language of 35 U.S.C. 101 which states that “whoever invents or discovers any new and useful process... may obtain a patent therefor...” (Emphasis added). Thus, the term “same invention,” in this context, means an invention drawn to identical subject matter. See Miller v. Eagle Mfg. Co., 151 U.S. 186 (1894); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Ockert, 245 F.2d 467, 114 USPQ 330 (CCPA 1957).
A statutory type (35 U.S.C. 101) double patenting rejection can be overcome by canceling or amending the claims that are directed to the same invention so they are no longer coextensive in scope. The filing of a terminal disclaimer cannot overcome a double patenting rejection based upon 35 U.S.C. 101.
Claims 1-20 are provisionally rejected under 35 U.S.C. 101 as claiming the same invention as that of claims 1-20 of copending Application No. 19/380,832 (reference application). This is a provisional statutory double patenting rejection since the claims directed to the same invention have not in fact been patented.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-5, 7-15, and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over D’Agostino (US 2005/0307931 A1) in view of Walters (US 2020/0012900 A1).
Regarding claims 1 and 11, D’Agostino discloses A computer-implemented method/system (D’Agostino: Claim 1, “apparatus”, and Claim 8, “method”) comprising:
a non-transitory storage medium storing computer program instructions (D’Agostino: Claim 15, “computer-readable storage medium comprising instructions stored therein which when executed by a processor…”);
and at least one processor configured to execute the computer program instructions to perform operations (D’Agostino: Claim 1, “a processor coupled to the memory…”, and Claim 15, “computer-readable storage medium comprising instructions stored therein which when executed by a processor…”) comprising:
receiving via a user interface a user (D’Agostino: Paragraph [0111], “In the example of FIG. 5C, the retriever identifies a vector 562, a vector 564 and a vector 566 as being related to the chat content provided from the customer chat window 552 of the chatbot application 550. According to various embodiments, the retriever 554 may augment or otherwise modify a ML model 580, such as a third LLM, to generate a chatbot response based on the vector 562, the vector 564, and the vector 566. Here, the retriever 554 may generate a prompt 570 which includes the vector 562, the vector 564, and the vector 566 with additional content that frames the activity to be performed by the ML model 580. For example, the additional text 572 may include a statement that reads, “generate a response to the question asked by the user based on a geographic location””);
wrapping the user question with AI model statistics(D’Agostino: Paragraph [0111], “In the example of FIG. 5C, the retriever identifies a vector 562, a vector 564 and a vector 566 as being related to the chat content provided from the customer chat window 552 of the chatbot application 550. According to various embodiments, the retriever 554 may augment or otherwise modify a ML model 580, such as a third LLM, to generate a chatbot response based on the vector 562, the vector 564, and the vector 566. Here, the retriever 554 may generate a prompt 570 which includes the vector 562, the vector 564, and the vector 566 with additional content that frames the activity to be performed by the ML model 580. For example, the additional text 572 may include a statement that reads, “generate a response to the question asked by the user based on a geographic location””);
receiving a response from the large language model, the response comprising an (D’Agostino: Paragraph [0111], “For example, the additional text 572 may include a statement that reads, “generate a response to the question asked by the user based on a geographic location”. The prompt 570 may be fed into the ML model 580 which generates a chatbot response that is then output to the customer chat window 552 of the chatbot application 550 via a chatbot, or other means”);
and displaying the response on the user interface (D’Agostino: Paragraph [0111], “The prompt 570 may be fed into the ML model 580 which generates a chatbot response that is then output to the customer chat window 552 of the chatbot application 550 via a chatbot, or other means”).
D’Agostino does not explicitly disclose the use of this system for the purposes of insight into an AI model being developed on a development platform or the use of development platform statistics.
However, Walters discloses this feature (Walters: Paragraph [0065], “development environment 405 can be configured to train data models”, Paragraph [0130], “ the performance criteria can include a similarity metric (e.g., a statistical correlation score, data similarity score, or data quality score, as described herein)”, and Paragraph [0214], “ the notification is sent to the device associated with the request for a model (step 2002). The notification may state that data drift has been detected and/or that the model has been corrected”).
D’Agostino and Walters are analogous art in the same field of endeavor as the instant invention as all are drawn to AI systems. The differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains; that is, it would have been obvious to incorporate Walter’s data drift detection for development models into the system of D’Agostino to allow for use in the machine learning development domain.
D'Agostino-Walter teaches 2/12. The computer-implemented method of claim 1/system of claim 11, receiving a user question comprising:
generating on the user interface a suggestion to explore an issue associated with the AI model (D’Agostino: Paragraph [0098], “The flagged conversations are accompanied by detailed analyses highlighting the specific areas of concern and providing recommendations for corrective action”);
and responsive to receiving a user selection of the suggestion, converting the suggestion to the user question (D’Agostino: Paragraph [0098], “The flagged conversations are accompanied by detailed analyses highlighting the specific areas of concern and providing recommendations for corrective action”).
D'Agostino-Walter teaches 3/13. The computer-implemented method of claim 1/system of claim 11, displaying the response further comprising:
displaying a first level response (D’Agostino: Paragraph [0098], “The flagged conversations are accompanied by detailed analyses highlighting the specific areas of concern and providing recommendations for corrective action”).
D'Agostino-Walter teaches 4/14. The computer-implemented method of claim 1/system of claim 11, displaying the response further comprising:
displaying a second level response (D’Agostino: Paragraph [0098], “The flagged conversations are accompanied by detailed analyses highlighting the specific areas of concern and providing recommendations for corrective action”; Walters: Paragraph [007], “ detect data drift, correct synthetic data models or predictive models for drift, and notify downstream users”).
D'Agostino-Walter teaches 5/15. The computer-implemented method of claim 4/system of claim 14, displaying the second level response further comprising:
displaying at least one of a performance based response, an optimized user's prompt template, or a large language model evaluation template (Walters: Paragraph [0121], “suggest response templates”).
D'Agostino-Walter teaches 7/17. The computer-implemented method of claim 1/system of claim 11, wrapping the user question with the AI model statistics further comprising:
wrapping the user question with at least one of AI model drift metrics, AI model daily volume, AI model actuals (D’Agostino: Paragraph [0098], “The flagged conversations are accompanied by detailed analyses highlighting the specific areas of concern and providing recommendations for corrective action”; Walters: Paragraph [007], “ detect data drift, correct synthetic data models or predictive models for drift, and notify downstream users”).
D'Agostino-Walter teaches 8/18. The computer-implemented method of claim 1/system of claim 11, wrapping the user question further comprising:
wrapping the user question with the user's previous conversation with the large language model comprising first level responses (D’Agostino: Paragraph [0149], “ FIGS. 7A-7B illustrate a process of augmenting a machine learning model based on context of a chat conversation and generating a response based on the augmented machine learning model according to example embodiments. For example, FIG. 7A illustrates a process 700A of augmenting an ML model 729 and generating a chatbot response based on execution of the augmented ML model 729 after augmentation. Referring to FIG. 7A, a source device 710 conducts a chat conversation with a chatbot via a chat application 721 which is hosted by a host platform 720. A user of the source device 710 may enter chat messages via a user interface 712 of the source device 710. Here, the host platform 720 may be associated with a service provider/organization that provides chatbots as a way of answering customer questions. A user of the source device 710 may enter queries, questions, requests, and other text-based input via a user interface 712 of the source device 710 which is then input to a chat window 722 of the chat application 721 over a network such as the Internet”).
D'Agostino-Walter teaches 9/19. The computer-implemented method of claim 1/system of claim 11, wrapping the user question further comprising:
wrapping the user question with the user's previous conversation with the large language model comprising second level responses (D’Agostino: Paragraph [0149], “ FIGS. 7A-7B illustrate a process of augmenting a machine learning model based on context of a chat conversation and generating a response based on the augmented machine learning model according to example embodiments. For example, FIG. 7A illustrates a process 700A of augmenting an ML model 729 and generating a chatbot response based on execution of the augmented ML model 729 after augmentation. Referring to FIG. 7A, a source device 710 conducts a chat conversation with a chatbot via a chat application 721 which is hosted by a host platform 720. A user of the source device 710 may enter chat messages via a user interface 712 of the source device 710. Here, the host platform 720 may be associated with a service provider/organization that provides chatbots as a way of answering customer questions. A user of the source device 710 may enter queries, questions, requests, and other text-based input via a user interface 712 of the source device 710 which is then input to a chat window 722 of the chat application 721 over a network such as the Internet”).
D'Agostino-Walter teaches 10/20. The computer-implemented method of claim 1/system of claim 11, wrapping the user question with the guidelines further comprising:
wrapping the user question with answer guidelines and formatting guidelines (D’Agostino: Paragraph [0111], “In the example of FIG. 5C, the retriever identifies a vector 562, a vector 564 and a vector 566 as being related to the chat content provided from the customer chat window 552 of the chatbot application 550. According to various embodiments, the retriever 554 may augment or otherwise modify a ML model 580, such as a third LLM, to generate a chatbot response based on the vector 562, the vector 564, and the vector 566. Here, the retriever 554 may generate a prompt 570 which includes the vector 562, the vector 564, and the vector 566 with additional content that frames the activity to be performed by the ML model 580. For example, the additional text 572 may include a statement that reads, “generate a response to the question asked by the user based on a geographic location””, and Paragraph [0099], “response templates”).
Claim(s) 6 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over D’Agostino (US 2005/0307931 A1) and Walters (US 2020/0012900 A1) as applied to claims 1 and 11 above, further in view of Conway (US 2025/0190459 A1).
D'Agostino-Walter teaches 6/16. The computer-implemented method of claim 1/system of claim 11.
D'Agostino-Walter does not explicitly disclose that displaying the response further compris[es]:
displaying the response in a dashboard view.
However, Conway teaches this feature (Conway: Paragraph [0138], “the user interface may be configured to provide output (e.g., a dashboard) relating to the development and/or performance of the constructed RAG-based generative AI systems”, and Paragraph [0221], “Visualizations of the monitored metric values and/or the corresponding explanations can be provided to the user in real time (e.g., in a system monitoring dashboard). In some embodiments, the AI monitoring system may be configured with thresholds or ranges related to the metric values, and the AI monitoring system may alert the user when the value of a metric exceeds a corresponding threshold or departs from a specified range”).
D’Agostino-Walters and Conway are analogous art in the same field of endeavor as the instant invention as all are drawn to AI systems. The differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains; that is, it would have been obvious to incorporate Conway’s dashboard into the system of D’Agostino-Walters to allow for use a unified interface for user interaction.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Liu (“G-EVAL: NLGEvaluation using GPT-4 with Better Human Alignment”) describes a system for evaluating the quality of machine learning systems under development. Shinn (“Reflexion: Language Agents with Verbal Reinforcement Learning”) describes a method for providing feedback to machine learning systems under development.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IMAD HUSSAIN whose telephone number is (571)270-3628. The examiner can normally be reached Monday-Friday 0900-1700 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamal Divecha can be reached at (571) 272-5863. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IMAD HUSSAIN/Primary Examiner, Art Unit 2453
Prosecution Insights