DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the original application filed on 12/28/2023.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”).
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A). The Step 2A analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application. If it is determined at step 2A, Prong 2 that the claims do not integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself.
Claim 1
Step 1: The claim recites a method; therefore, it is directed to the statutory category of a process.
Step 2A Prong 1: The claim recites, inter alia:
generating, …, a question prompt based on the topic and the AI role: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating a question prompt based on a topic and occupational role, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally generate a question prompt asking about birds from a bird expert’s perspective.
generating, …, one or more statements as a statement set corresponding to the question prompt: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating a a statement corresponding to a prompt, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally generate a statement or information that is true or false corresponding to a prompt or question.
masking key terms included in the statements of the statement set to form sets of masked statements in which each statement of the statement set includes a respective key term and each masked statement included in a particular set includes at least one masked key term: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of masking key terms in a statement, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally identify key terms in a statement and mask or hide the term.
determining, …, a set of unmasked statements in which a respective unmasked statement is based on a corresponding masked statement: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of determining unmasked statements, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally determine an unmasked statement or deduce a term in a statement.
evaluating performance of the second generative AI model based on comparing the sets of unmasked statements to respective statements included in the statement set: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of comparing statements or words, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally compare the performance of a model by looking at words and comparing them.
Step 2A Prong 2: The claim does not recite any additional limitations which integrate the abstract idea into a practical application. Specifically, the additional elements consist of “obtaining a topic and an artificial intelligence (AI) role relating to the topic, the topic relating to a field of study and the AI role representing an occupational role in the field of study in which the topic and the AI role are specified by a human user”, “by a first generative AI model”, and “by a second generative AI model”.
The additional elements of “by a first generative AI model”, and “by a second generative AI model” amount to generic AI models used as a tool to perform an existing process. Thus, the additional elements amount to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
The additional element “obtaining a topic and an artificial intelligence (AI) role relating to the topic, the topic relating to a field of study and the AI role representing an occupational role in the field of study in which the topic and the AI role are specified by a human user” is insignificant extra-solution activity required for any uses of the abstract ideas (see MPEP § 2106.05(g)).
Thus, even when viewed individually and as an ordered combination, these additional elements do not integrate the abstract idea into a practical application, and the claim is thus directed to the abstract idea.
Step 2B: Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea.
The additional elements of “by a first generative AI model”, and “by a second generative AI model” amount to generic AI models used as a tool to perform an existing process. Thus, the additional elements amount to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
The additional element “obtaining a topic and an artificial intelligence (AI) role relating to the topic, the topic relating to a field of study and the AI role representing an occupational role in the field of study in which the topic and the AI role are specified by a human user” is insignificant extra-solution activity required for any uses of the abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity(see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network”).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 2
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional element of “retraining or fine-tuning the second generative AI model using a second training dataset different from a first training dataset initially used to train the second generative AI model based on evaluation of the performance of the second generative AI model indicating that the second generative AI model provides inaccurate outcomes according to the comparing the sets of unmasked statements to respective statements included in the statement set” amounts to reciting only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is not clear how the generic second generative AI model is broadly retrained or fine-tuned with a particular training dataset. Thus, the additional elements amount to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 3
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional element of “wherein one or more of the statements included in the statement set are provided by a human user and the one or more statements generated by the first generative AI model and provided by the human user are true statements or false statements about the question prompt” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claim 4
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
identifying one or more stop words that represent common words involved in natural language processing of the statement set: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of identifying stop words or common words in a sentence, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
excluding the one or more stop words from each statement of the statement set: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of excluding words from a statement set, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
identifying the key terms included in each respective statement based on words remaining in each statement after excluding the one or more stop words: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of identifying key terms, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
masking one of the identified key terms: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of masking or hiding or removing key terms, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2, Step 2B: The claim does not recite any additional elements that are sufficient to integrate the judicial exceptions into a practical application or amount to significantly more than the judicial exception. As such, the claim is ineligible.
Claim 5
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
computing an evaluation score that is based on a probability that the second generative AI model returns a correct masked key term to replace a particular masked key term included in a particular statement and a total number of masked key terms included in the particular statement: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mathematical concept of computing an evaluation score based on a probability, which is performed by mathematical calculation as evidenced by paragraph [0040] and equation 1 of the originally filed specification.
Step 2A Prong 2, Step 2B: The claim does not recite any additional elements that are sufficient to integrate the judicial exceptions into a practical application or amount to significantly more than the judicial exception. As such, the claim is ineligible.
Claim 6
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
computing an evaluation score that is based on a total number of masked key terms included in a particular statement and a ranking of how frequently a correct unmasked key term used to replace a particular masked term is returned by the second generative AI model relative to how frequently incorrect unmasked key terms used to replace the particular masked term are returned by the second generative AI model: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mathematical concept of computing an evaluation score based on a total number and a ranking, which is performed by mathematical calculation as evidenced by paragraph [0041] and equation 2 of the originally filed specification.
Step 2A Prong 2, Step 2B: The claim does not recite any additional elements that are sufficient to integrate the judicial exceptions into a practical application or amount to significantly more than the judicial exception. As such, the claim is ineligible.
Claim 7
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2, Step 2B: The additional element of “determining the set of unmasked statements is performed by the second generative AI model and a third generative AI model” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). The additional element “evaluating the performance of the second generative AI model includes visually representing a fairness of the second generative AI model in comparison to a fairness of the third generative AI model” is insignificant extra-solution activity required for any uses of the abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity(see MPEP § 2106.05(d)(II)(i); “Presenting offers and gathering statistics”). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible.
Claims 8-14
Claims 8-14 recite one or more non-transitory computer-readable storage media (step 1: a manufacture) using a system to perform the steps of claims 1-7, respectively, which by MPEP 2106.05(f) (“apply it”) cannot integrate an abstract idea into a practical application or provide significantly more than the abstract idea by itself, and are thus rejected for the same reasons set forth in the rejection of claims 1-7, respectively.
Claims 15-20
Claims 15-20 recite a system (step 1: a machine) using one or more processors and one or more non-transitory computer-readable storage media to perform the steps of claims 1-3 and 5-7, respectively, which by MPEP 2106.05(f) (“apply it”) cannot integrate an abstract idea into a practical application or provide significantly more than the abstract idea by itself, and are thus rejected for the same reasons set forth in the rejection of claims 1-3 and 5-7, respectively.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5, 6, 8, 10, 12, 13, 15, 17, 18, and 19 are rejected under 35 USC § 103 as being obvious over Petroni et al. (Petroni et al., “Language Models as Knowledge Bases?”, Sep. 4, 2019, arXiv:1909.01066v2, pp. 1-11, hereinafter “Petroni”) in view of White et al. (White et al., “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT”, Feb. 21, 2023, arXiv:2302.11382, pp. 1-19, hereinafter “White”).
Regarding claim 1, Petroni discloses [a] method, comprising: (Abstract; “We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the art pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remark ably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches”, which discloses a method for analyzing relation knowledge using language models and §4)
generating,…, one or more statements as a statement set corresponding to the question prompt; (§4; “Each fact is converted into a cloze statement which is used to query the language model for a missing token”, which discloses generating one or more statements or cloze statements corresponding to a question prompt; and §4.1.4; “We manually create cloze-style questions from these questions, e.g., rewriting ‘Who developed the theory of relativity?” as “The theory of relativity was developed by’”)
masking key terms included in the statements of the statement set to form sets of masked statements in which each statement of the statement set includes a respective key term and each masked statement included in a particular set includes at least one masked key term; (§1; “Instead, we could attempt to query neural language models for relational data by asking them to fill in masked tokens in sequences like “Dante was born in [Mask]”, as illustrated in Figure 1 … We define that a pretrained language model knows a fact (subject, relation, object) such as (Dante, born-in, Florence) if it can successfully predict masked objects in cloze sentences such as “Dante was born in ” expressing that fact.”, which discloses that each statement (“Dante was born in Florence”) had a corresponding key term (“Florence”) that is masked to form the masked statement (“Dante was born in [mask]”); and §4)
determining, by a second generative AI model, a set of unmasked statements in which a respective unmasked statement is based on a corresponding masked statement; and (§4; “We evaluate each model based on how highly it ranks the ground truth token against every other word in a fixed candidate vocabulary”; and Table 3; the table discloses BERT (the second generative AI model) and its output or predicted tokens for masked queries)
evaluating performance of the second generative AI model based on comparing the sets of unmasked statements to respective statements included in the statement set (§4.4; “We use the mean precision at k (P@k). For a given fact, this value is 1 if the object is ranked among the top k results, and 0 otherwise”; and §5; and Table 2; “Mean precision at one(P@1) … across the set of evaluation corpora”).
Petroni fails to explicitly disclose but White discloses obtaining a topic and an artificial intelligence (AI) role relating to the topic, the topic relating to a field of study and the AI role representing an occupational role in the field of study in which the topic and the AI role are specified by a human user; (§III.E.; “Act as persona X / Provide outputs that persona X would create … This persona can be expressed in a number of ways, ranging from a job description, title, fictional char acter, historical figure, etc.”, wherein the variable “X” is a human-user specified persona inserted into the prompt by the user. This also further discloses the AI role as an occupational role (“job description”) specified by the user; and §III.M; “Create a game for me around X … The first statement, instructs the LLM to create a game and provides the important scoping of the game to a topic area … We are going to play a cybersecurity game. You are going to pretend to be a Linux terminal for a computer that has been compromised by an attacker. When I type in a command, you are going to output the corresponding text that the Linux terminal would produce”, which discloses the variable “X” that is a human-specified topic, and obtaining this topic and an AI role related to the topic (“cybersecurity”))
generating, by a first generative AI model, a question prompt based on the topic and the AI role; (§I; “With the right prompt, it is possible to create entirely new interaction paradigms, such as having an LLM generate and give a quiz associated with a software engineering concept or tool, or even simulate a Linux terminal window”, which discloses a first generative AI model generating q question prompt or quiz conditioned on a topic such as a software engineering concept; and Abstract; “large language models (LLMs), such as ChatGPT”, which discloses the first generative AI model like ChatGPT; and §III.M; “The more specific the topic, typically the more novel and interesting the game play … This cybersecurity game prompt combines a number of patterns, including Game Play and Persona”, which discloses generating content conditioned jointly on topic and AI role; and §III.E; “Act as persona X / Provide outputs that persona X would create”, which teaches that the LLM’s generated output is separately conditioned on the role or persona variable “X”)
by the first generative AI model (§III.M; “To start the game, print a scenario of what happened that led to my investigation and make the description have clues that I can use to get started”, which discloses that the LLM generates a descriptive statement in response to the role or topic-conditioned game prompt).
Petroni and White are analogous art because both are concerned with prompt engineering and large language models. Before the effective filing date of the claimed invention, it would have been obvious to one skilled in large language models to combine the topic and AI role and first generative AI model of White with the masking and method of Petroni to yield to the predictable result of obtaining a topic and an artificial intelligence (AI) role relating to the topic, the topic relating to a field of study and the AI role representing an occupational role in the field of study in which the topic and the AI role are specified by a human user; generating, by a first generative AI model, a question prompt based on the topic and the AI role; generating, by the first generative AI model, one or more statements as a statement set corresponding to the question prompt. The motivation for doing so would be to solve common problems when conversing with LLMs (White; Abstract).
Regarding claim 8, it is a non-transitory computer-readable storage media claim corresponding to the steps of claim 1 and is rejected for the same reasons as claim 1.
Regarding claim 15, it is a system claim corresponding to the steps of claim 1 and is rejected for the same reasons as claim 1.
Regarding claims 3, 10, and 17, the rejection of claims 1, 8, and 15 are incorporated and Petroni discloses wherein one or more of the statements included in the statement set are provided by a human user and the one or more statements generated by the first generative AI model and provided by the human user are true statements or false statements about the question prompt (§4.1.1; “TheGoogle-RE corpus3 contains ∼60K facts manually extracted from Wikipedia”, wherein these human-provided true facts or statements are combined into the same statement set along with the cloze statements corresponding to the query or prompt; and §4.1.3; “we find the OMCS sentence that contains both the subject and the object).
Regarding claims 5, 12, and 18, the rejection of claims 1, 8, and 15 are incorporated and Petroni discloses wherein evaluating the performance of the second generative AI model includes computing an evaluation score that is based on a probability that the second generative AI model returns a correct masked key term to replace a particular masked key term included in a particular statement and a total number of masked key terms included in the particular statement (§4.4; “We use the mean precision at k (P@k). For a given fact, this value is 1 if the object is ranked among the top k results, and 0 otherwise”; and §5; and Figure 3 Description; “LPFP is the log probability score associated with the first prediction”; and Table 3; “The last column reports the top five tokens generated together with the associated log probability (in square brackets)”).
Regarding claims 6, 13, and 19, the rejection of claims 1, 8, and 15 are incorporated and Petroni discloses wherein evaluating the performance of the second generative AI model includes computing an evaluation score that is based on a total number of masked key terms included in a particular statement and a ranking of how frequently a correct unmasked key term used to replace a particular masked term is returned by the second generative AI model relative to how frequently incorrect unmasked key terms used to replace the particular masked term are returned by the second generative AI model (§4.4; “We use the mean precision at k (P@k). For a given fact, this value is 1 if the object is ranked among the top k results, and 0 otherwise”, which discloses a ranking based score that compares the frequency or rank position of the correct token relative to all candidate tokens, aggregated over the total number of masked-term instances in the statements; and §4; “We evaluate each model based on how highly it ranks the ground truth token against every other word in a fixed candidate vocabulary. This is similar to ranking-based metrics from the knowledge base completion literature”).
Claims 2, 9, and 16 are rejected under 35 USC § 103 as being obvious over Petroni in view of White and further in view of Fichtel et al. (Fichtel et al., “Prompt Tuning or Fine-Tuning- Investigating Relational Knowledge in Pre-Trained Language Models”, Aug. 31, 2021, Automated Knowledge Base Construction (2021), pp. 1-15, hereinafter “Fichtel”).
Regarding claims 2, 9, and 16, the rejection of claims 1, 8, and 15 are incorporated and Petroni fails to explicitly disclose but Fichtel discloses retraining or fine-tuning the second generative AI model using a second training dataset different from a first training dataset initially used to train the second generative AI model based on evaluation of the performance of the second generative AI model indicating that the second generative AI model provides inaccurate outcomes according to the comparing the sets of unmasked statements to respective statements included in the statement set (Abstract; “The performance of the relational fact extraction task depends significantly on the query sentence … we propose using a completely different approach: Instead of spending resources on training an additional model, we simply perform an adaptive fine-tuning of the pre-trained language model on the standard fill-mask task using a small training dataset of existing facts from a knowledge graph. We investigate the differences between complex prompting techniques and adaptive fine-tuning in an extensive evaluation. Remarkably, adaptive fine-tuning outperforms all baselines”; and §1; “instead of this complex additional prompt tuning, a simple adaptive fine-tuning of the pre-trained language model using few training triples from a knowledge graph already does the trick”).
Petroni, White, and Fechter are analogous art because all are concerned with prompt engineering and large language models. Before the effective filing date of the claimed invention, it would have been obvious to one skilled in large language models to combine the fine tuning of Fechter with the masking and method of Petroni and White to yield to the predictable result of retraining or fine-tuning the second generative AI model using a second training dataset different from a first training dataset initially used to train the second generative AI model based on evaluation of the performance of the second generative AI model indicating that the second generative AI model provides inaccurate outcomes according to the comparing the sets of unmasked statements to respective statements included in the statement set. The motivation for doing so would be to show that even fewer training relations are needed to achieve high knowledge extraction quality (Fechter; Abstract).
Claims 4 and 11 are rejected under 35 USC § 103 as being obvious over Petroni in view of White and further in view of Rose et al. (US 20110060747 A1, hereinafter “Rose”).
Regarding claims 4 and 11, the rejection of claims 1 and 8 are incorporated and Petroni fails to explicitly disclose but Rose discloses identifying one or more stop words that represent common words involved in natural language processing of the statement set; excluding the one or more stop words from each statement of the statement set; identifying the key terms included in each respective statement based on words remaining in each statement after excluding the one or more stop words; and masking one of the identified key terms ([0006-0007]; “keywords frequently contain multiple words but rarely contain standard punctuation or stop words, such as the function words and, the, and of or other words with minimal lexical meaning … According to one embodiment of the present invention, rapid, automatic, keyword extraction (RAKE) methods and systems can include parsing words in an individual document by delimiters, stop words, or both in order to identify candidate keywords”; and [0037]; “a keyword score is calculated for each candidate keyword. In this embodiment, the keyword score is defined as the sum of its member word score”).
Petroni, White, and Rose are analogous art because all are concerned with prompt engineering and large language models. Before the effective filing date of the claimed invention, it would have been obvious to one skilled in large language models to combine the fine tuning of Rose with the masking and method of Petroni and White to yield to the predictable result of identifying one or more stop words that represent common words involved in natural language processing of the statement set; excluding the one or more stop words from each statement of the statement set; identifying the key terms included in each respective statement based on words remaining in each statement after excluding the one or more stop words; and masking one of the identified key terms. The motivation for doing so would be to parse words in an individual document by delimiters, stop words, or both in order to identify candidate keywords (Rose; Abstract).
Claims 7, 14, and 20 are rejected under 35 USC § 103 as being obvious over Petroni in view of White and further in view of Liang et al. (Liang et al., “Holistic Evaluation of Language Models”, Oct. 1, 2023, arXiv:2211.09110v2, pp. 1-162, hereinafter “Liang”).
Regarding claims 7, 14, and 20, the rejection of claims 1, 8, and 15 are incorporated and Petroni further discloses determining the set of unmasked statements is performed by the second generative AI model and a third generative AI model; and (§4.2; and Table 1; the table discloses evaluation multiple distinct language models against the same masked cloze statements).
Petroni fails to explicitly disclose but Liang discloses evaluating the performance of the second generative AI model includes visually representing a fairness of the second generative AI model in comparison to a fairness of the third generative AI model (Abstract; “We measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios to the extent possible (87.5% of the time), … we conduct a large-scale evaluation of 30 prominent language models … densely benchmarked on a set of core scenarios and metrics under standardized conditions” (emphasis added); and §8; “To understand these results, we provide a web interface.62 This interface not only provides the quantitative results, as is customary for other benchmarks and leaderboards, but also the underlying model predictions and the exact inputs and prompts that yielded these predictions”).
Petroni, White, and Liang are analogous art because all are concerned with prompt engineering and large language models. Before the effective filing date of the claimed invention, it would have been obvious to one skilled in large language models to combine the fairness representation of Liang with the masking and method of Petroni and White to yield to the predictable result of evaluating the performance of the second generative AI model includes visually representing a fairness of the second generative AI model in comparison to a fairness of the third generative AI model. The motivation for doing so would be to improve the transparency of language models (Liang; Abstract).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Xu et al., “ExpertPrompting: Instructing Large Language Models to be Distinguished Experts”, May 24, 2023, arXiv:2305.14688v1, pp. 1-8.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Brent Hoover whose telephone number is (303)297-4403. The examiner can normally be reached Monday - Friday 9-5 MST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Kawsar can be reached at 571-270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRENT JOHNSTON HOOVER/Primary Examiner, Art Unit 2127
Prosecution Insights