DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 5/28/2024 in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Status of Claims
Claims 1-20 are pending in this application.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 4, 6-8, 10-12, 14, 16-18 and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Horesh et al. (U.S. Patent Application Publication 2025/0278630).
As per claims 1 and 11, Horesh et al. discloses:
A system (Figure 5 and paragraphs [0030] & [0057-0071]) comprising:
at least one device including a hardware processor (Figure 5, item 502 and paragraphs [0030] & [0057-0071]);
the system being configured to perform operations comprising:
accessing a first training data set, wherein the first training data set comprises:
a set of system-generated computational metrics corresponding to a first output of a first ML model;
a human evaluation of the first output of the first ML model;
training a second ML model, based at least in part on the first training data set, to estimate human evaluations of output from the first ML model (Figure 3, item 320 and Paragraphs [0035] & [0037] – The Score component is an ML model that is trained with examples of human evaluations of prompts (output of an LLM) to estimate human evaluations of the prompts);
receiving a target output generated by the first ML model (Paragraphs [0034-0038] – candidate prompts are generated);
executing a machine-evaluation of the target output to generate a first set of system-generated computational metrics corresponding to the target output (Paragraphs [0034-0038] – the prompts are scored by various metrics); and
applying the second ML model to the first set of system-generated computational metrics to generate a first estimated human evaluation of the target output (Paragraphs [0034-0038] – the metrics are used by the score component ML model to estimate a human assigned score).
Claim 1 is directed to a one or more computer readable media containing instructions to cause a processor to act as the system of claim 11, so is rejected for similar reasons.
As per claims 2 and 12, Horesh et al. discloses all of the limitations of claims 1 and 11 above. Horesh et al. further discloses:
the first training data set further comprises an application context for the first ML model, wherein the operations further comprise associating the second ML model with the application context (Paragraphs [0033] & [0035-0036]).
As per claims 4 and 14, Horesh et al. discloses all of the limitations of claims 1 and 11 above. Horesh et al. further discloses:
the first ML model comprises a generative large language model (Paragraph [0035]).
As per claims 6 and 16, Horesh et al. discloses:
A system (Figure 5 and paragraphs [0030] & [0057-0071]) comprising:
at least one device including a hardware processor (Figure 5, item 502 and paragraphs [0030] & [0057-0071]);
the system being configured to perform operations comprising:
accessing a first training data set, wherein the first training data set comprises:
a set of system-generated deterministic computational metrics corresponding to a first output of a first ML model;
a qualitative evaluation of the first output of the first ML model generated by a large language model;
training a second ML model, based at least in part on the first training data set, to estimate large language model qualitative evaluations of output of the first ML model (Figure 3, item 320 and Paragraphs [0035] & [0037] – The Score component is an ML model that is trained with examples of human evaluations (qualitative evaluations) of prompts (output of an LLM) to estimate human evaluations of the prompts);
receiving a target output generated by the first ML model (Paragraphs [0034-0038] – candidate prompts are generated);
executing a machine-evaluation of the target output to generate a first set of system-generated computational metrics corresponding to the target output (Paragraphs [0034-0038] – the prompts are scored by various metrics); and
applying the second ML model to the first set of system-generated deterministic computational metrics to generate a first estimated large language model qualitative evaluation of the target output (Paragraphs [0034-0038] – the metrics are used by the score component ML model to estimate a human assigned score).
Claim 6 is directed to a one or more computer readable media containing instructions to cause a processor to act as the system of claim 16, so is rejected for similar reasons.
As per claims 7 and 17, Horesh et al. discloses all of the limitations of claims 6 and 16 above. Horesh et al. further discloses:
the first training set further comprises a human evaluation of the first output of the first ML model, wherein training the second ML model comprises: applying a first influence weight to the human evaluation of the first output and; applying a second influence weight to the qualitative evaluation of the first output of the first ML model generated by a large language model; wherein the first influence weight and the second influence weight are different influence weights (Paragraphs [0036-0037] – training a model to produce a score based on multiple metrics inherently requires multiple different weights).
As per claims 8 and 18, Horesh et al. discloses all of the limitations of claims 6 and 16 above. Horesh et al. further discloses:
the first training data set further comprises an application context for the first ML model, wherein the operations further comprise associating the second ML model with the application context (Paragraphs [0033] & [0035-0036]).
As per claims 10 and 20, Horesh et al. discloses all of the limitations of claims 6 and 16 above. Horesh et al. further discloses:
the first ML model comprises a generative large language model (Paragraph [0035]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Horesh et al. (U.S. Patent Application Publication 2025/0278630).
As per claims 5 and 15, Horesh et al. discloses all of the limitations of claims 1 and 11 above.
Horesh et al. discloses the claimed invention except that the first estimated human evaluation of the target output comprises one qualitative metric instead of two or more qualitative metrics. It would have been obvious to one having ordinary skill in the art at the effective filing date of the invention to have the model output two metrics instead of one, since it has been held that mere duplication of the essential working part of a device involves only routine skill in the art. St. Regis Paper Co. v. Bemis Co., 193 USPQ 8.
Allowable Subject Matter
Claims 3, 9, 13 and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Examiner Notes
The Examiner cites particular columns and line numbers in the references as applied to the claims above for the convenience of the Applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the Applicant fully considers the references in its entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or as disclosed by the Examiner.
Communications via Internet e-mail are at the discretion of the applicant and require written authorization. Should the Applicant wish to communicate via e-mail, including the following paragraph in their response will allow the Examiner to do so:
“Recognizing that Internet communications are not secure, I hereby authorize the USPTO to communicate with me concerning any subject matter of this application by electronic mail. I understand that a copy of these communications will be made of record in the application file.”
Should e-mail communication be desired, the Examiner can be reached at Edwin.Leland@USPTO.gov
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EDWIN S LELAND III whose telephone number is (571)270-5678. The examiner can normally be reached 8:00 - 5:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at 571-272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EDWIN S LELAND III/Primary Examiner, Art Unit 2654