DETAILED ACTION
This office action is in response to Applicant’s Amendment/Request for Reconsideration, received on 05/26/2026. Claims 1, 7, 11, 16, and 20 have been amended. Claims 1-20 are pending and have been considered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) submitted on 03/03/2026 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement(s) is/are being considered by the examiner.
Response to Arguments
Applicant’s arguments, see pg. 13, filed 05/26/2026, with respect to the “Formal Matters” have been fully considered and are persuasive. The objection of the specification and claims 7, 16 have been withdrawn.
Applicant’s arguments, see pgs. 13-14, filed 05/26/2026, with respect to “Rejections under 35 U.S.C. 112” have been fully considered and are persuasive. The rejection of claim 20 has been withdrawn.
Applicant’s arguments, see pgs. 14-15, filed 05/26/2026, with respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 102(a)(2) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Qadrud-Din which was previously relied upon for the rejections of claims 8 and 17. The criteria evaluation prompts and prompt testing of Qadrud-Din will be taken in view of the prompts and evaluation of Jia. See updated rejections below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 8, 10-11, 13, 17, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jia et al. (US-20250371356-A1), hereinafter Jia in view of Qadrud-Din et al. (US-20240289561-A1), hereinafter Qadrud-Din.
Regarding claim 1, Jia discloses: a computing system (Abstract, Methods, systems, and computer-readable storage media) comprising: one or more processing devices ([Fig. 6, Processor 610]) configured to:
receive prompt generation instructions that specify an initial prompt ([0017] providing an initial version of a prompt template, the prompt template including dynamic input and first static input, generating a prompt using the initial version of the prompt template) and a prompt evaluation criterion ([0017] receiving, from a large language model (LLM), an output that is responsive to the prompt, providing an evaluation at least partially based on the output, [The instructions would be similar to “use an LLM to generate prompt evaluations”]); and
based at least in part on the prompt generation instructions, programmatically assemble a prompt iteration module ([The examiner notes that the structure of Fig. 2 suggests a looping structure between prompt generation module 202, prompting module 204, evaluation module 206, and prompt template update module 208 indicating prompt iterations in view of the looping of Fig. 4 ([0169])]) from a candidate prompt generator module ([Fig. 2, Prompt Generation Module 202]) and an evaluator module ([Fig. 2, Evaluation Module 206]).
Jia does not disclose:
wherein:
the candidate prompt generator module and the evaluator module are stored in a prompt generation module library that includes a plurality of candidate prompt generator modules and a plurality of evaluator modules;
the prompt evaluation criterion is a natural language input; and
the one or more processing devices are configured to select the evaluator module at least in part by converting the prompt evaluation criterion into a module selection of the evaluator module.
Qadrud-Din discloses:
wherein:
the candidate prompt generator module and the evaluator module are stored in a prompt generation module library that includes a plurality of candidate prompt generator modules ([Fig. 2, Skills 232, 234, wherein skill 234 contains multiple prompt templates 236, 238], [Containing multiple skills, wherein each skill can contain multiple prompt templates, indicates the orchestrator to be a prompt generation module library comprised of a plurality of candidate prompt generator modules, i.e. skills]) and a plurality of evaluator modules ([0057] perform operations such as testing prompts created based on prompt templates against tests stored in the test repository 224, [Testing prompts based on tests (plurality emphasized) stored in a repository indicates a plurality of evaluator modules in the form of tests]);
the prompt evaluation criterion is a natural language input ([0041] the criteria evaluation prompt includes an instruction to evaluate some or all of the subset of the text portions against the natural language criterion, [The examiner asserts that a criteria evaluation prompt will necessarily contain the criteria to be evaluated, which is disclosed to be natural language]); and
the one or more processing devices are configured to select the evaluator module at least in part by converting the prompt evaluation criterion into a module selection of the evaluator module ([0318] one or more prompts may be tested by the prompt testing utility 226 against one or more tests stored in the test repository 224. A test result may be evaluated using the method 1200 to determine whether a prompt constructed from a prompt template being tested resulted in the generation of a hallucination, [Testing prompts against one or more tests for hallucinations indicates each test to be an evaluation module, wherein determining there is a hallucination indicates converting criteria into a module, i.e. test, for hallucination. Applying the evaluation module of prompts of Jia to the various criteria/tests of Qadrud-Din will effectively result in different evaluations for different prompts as Qadrud-Din discloses multiple prompt templates (236, 238) and a test repository 224]).
Jia and Qadrud-Din are considered analogous art within prompt augmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia to incorporate the teachings of Qadrud-Din, because of the novel way to automatically evaluate documents against a policy through use of a document pipeline which identifies text that should be kept together and extraneous text, allowing for segmentation of input text with more accurate responses generated through removal of extraneous input (Qadrud-Din, [0023]-[0028]).
Jia further discloses:
in each of a plurality of iterations of a prompt generation loop executed by the prompt iteration module:
generate a plurality of candidate prompts at least in part at a machine learning model ([0064] the LLM is used to generate an updated version of the prompt template, [0171] In some examples, the prompt template is an initial version for an initial iteration of optimization. In some examples, the prompt template is an updated version for a next iteration of optimization. A batch of prompts is generated (504)),
wherein the candidate prompts are generated based at least in part on a current-iteration prompt that is initialized as the initial prompt in a first iteration of the plurality of iterations ([0083] After ten (10) iterations of batch-based optimization, an updated version of the prompt template can be provided, [Performing batch-generation of prompts ten times indicates the first optimization to be applied to an initial version in view of [0171] cited above, wherein ten iterations indicates a plurality]);
as specified by the prompt evaluation criterion, compute respective evaluation scores associated with the candidate prompts ([0092] each evaluation in the batch of evaluations 306 is provided from the LLM system 220 in response to respective evaluation prompts provided by the evaluation module 206. For example, the evaluation module 206 can generate an evaluation prompt for each output in the batch of outputs 304 and prompts the LLM of the LLM system 220 using the evaluation prompt); and,
based at least in part on the evaluation scores, replace the current-iteration prompt ([Fig. 4, 412 “Update prompt template”], [0169] If the score does not exceed the threshold score, the prompt template is updated (412)); and,
output a final prompt generated in a final iteration of the plurality of iterations ([0169] If the score does exceed the threshold score, the prompt template is stored for production use (414), [storing prompts based on a threshold being met (indicating a final iteration in a plurality of iterations) requires an outputting of the final prompt from the evaluation module to a storage unit]).
Regarding claim 3, Jia in view of Qadrud-Din discloses: the computing system of claim 1.
Jia further discloses:
wherein the one or more processing devices are configured to compute the evaluation scores at least in part at an evaluation machine learning model ([0168] the evaluation module 206 can make an API call to the LLM system 220, the call including the evaluation prompt, where the LLM system 220 returns the evaluation 232).
Regarding claim 8, Jia in view of Qadrud-Din discloses: the computing system of claim 1.
Qadrud-Din further discloses:
wherein:
the prompt generation instructions further specify a structured input format ([Fig. 10, generated Timeline prompt 1008 based on Chunk 1010 of input text], [0225] determines one or more timeline generation prompts 1008 based on the timeline generation request message 1004. In some embodiments, the determination of the one or more timeline prompts may involve processing one or more input documents via the chunker. As discussed herein, for instance with respect to the methods 500 and 600 shown in FIG. 5 and FIG. 6, the chunker may perform one or more operations such as pre-processing, sharding, and/or chunking the documents into manageable text, [Developing prompts for each chunk of received text indicates the input format to be specified to be in chunks]); and
in the prompt generation loop, the one or more processing devices are configured to generate the candidate prompts in the structured input format ([0332] The prompt template may also include one or more additional instructions, such as an instruction to format the text generated by the text generation model as structured text. For instance, the structured text may be implemented as a JSON list).
Regarding claim 10, Jia in view of Qadrud-Din discloses: the computing system of claim 1.
Jia further discloses:
wherein:
the prompt generation instructions indicate a mutable portion of the initial prompt and an immutable portion of the initial prompt ([0020] prompt templates include static input and dynamic input, [Mutable and immutable track to dynamic and static respectively]); and
in the prompt generation loop, the one or more processing devices are configured to modify the mutable portion of the initial prompt while leaving the immutable portion unchanged ([0020] Here, the static input is the same for each prompt and each invocation of the LLM (each time the LLM is prompted), and the dynamic input includes data dictated by user interaction for each invocation of the LLM).
Regarding claim 11, Jia discloses: a method for use with a computing system (Abstract, Methods, systems, and computer-readable storage media), the method comprising:
receiving prompt generation instructions that specify an initial prompt ([0017] providing an initial version of a prompt template, the prompt template including dynamic input and first static input, generating a prompt using the initial version of the prompt template) and a prompt evaluation criterion ([0017] receiving, from a large language model (LLM), an output that is responsive to the prompt, providing an evaluation at least partially based on the output, [The instructions would be similar to “use an LLM to generate prompt evaluations”]); and
based at least in part on the prompt generation instructions, programmatically assemble a prompt iteration module ([The examiner notes that the structure of Fig. 2 suggests a looping structure between prompt generation module 202, prompting module 204, evaluation module 206, and prompt template update module 208 indicating prompt iterations in view of the looping of Fig. 4 ([0169])]) from a candidate prompt generator module ([Fig. 2, Prompt Generation Module 202]) and an evaluator module ([Fig. 2, Evaluation Module 206]).
Jia does not disclose:
wherein:
the candidate prompt generator module and the evaluator module are stored in a prompt generation module library that includes a plurality of candidate prompt generator modules and a plurality of evaluator modules;
the prompt evaluation criterion is a natural language input; and
selecting the evaluator module includes converting the prompt evaluation criterion into a module selection of the evaluator module.
Qadrud-Din discloses:
wherein:
the candidate prompt generator module and the evaluator module are stored in a prompt generation module library that includes a plurality of candidate prompt generator modules ([Fig. 2, Skills 232, 234, wherein skill 234 contains multiple prompt templates 236, 238], [Containing multiple skills, wherein each skill can contain multiple prompt templates, indicates the orchestrator to be a prompt generation module library comprised of a plurality of candidate prompt generator modules, i.e. skills]) and a plurality of evaluator modules ([0057] perform operations such as testing prompts created based on prompt templates against tests stored in the test repository 224, [Testing prompts based on tests (plurality emphasized) stored in a repository indicates a plurality of evaluator modules in the form of tests]);
the prompt evaluation criterion is a natural language input ([0041] the criteria evaluation prompt includes an instruction to evaluate some or all of the subset of the text portions against the natural language criterion, [The examiner asserts that a criteria evaluation prompt will necessarily contain the criteria to be evaluated, which is disclosed to be natural language]); and
selecting the evaluator module includes converting the prompt evaluation criterion into a module selection of the evaluator module ([0318] one or more prompts may be tested by the prompt testing utility 226 against one or more tests stored in the test repository 224. A test result may be evaluated using the method 1200 to determine whether a prompt constructed from a prompt template being tested resulted in the generation of a hallucination, [Testing prompts against one or more tests for hallucinations indicates each test to be an evaluation module, wherein determining there is a hallucination indicates converting criteria into a module, i.e. test, for hallucination. Applying the evaluation module of prompts of Jia to the various criteria/tests of Qadrud-Din will effectively result in different evaluations for different prompts as Qadrud-Din discloses multiple prompt templates (236, 238) and a test repository 224]).
Jia and Qadrud-Din are considered analogous art within prompt augmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia to incorporate the teachings of Qadrud-Din, because of the novel way to automatically evaluate documents against a policy through use of a document pipeline which identifies text that should be kept together and extraneous text, allowing for segmentation of input text with more accurate responses generated through removal of extraneous input (Qadrud-Din, [0023]-[0028]).
Jia further discloses:
in each of a plurality of iterations of a prompt generation loop executed by the prompt iteration module:
generating a plurality of candidate prompts at least in part at a machine learning model ([0064] the LLM is used to generate an updated version of the prompt template, [0171] In some examples, the prompt template is an initial version for an initial iteration of optimization. In some examples, the prompt template is an updated version for a next iteration of optimization. A batch of prompts is generated (504)),
wherein the candidate prompts are generated based at least in part on a current-iteration prompt that is initialized as the initial prompt in a first iteration of the plurality of iterations ([0083] After ten (10) iterations of batch-based optimization, an updated version of the prompt template can be provided, [Performing batch-generation of prompts ten times indicates the first optimization to be applied to an initial version in view of [0171] cited above, wherein ten iterations indicates a plurality]);
as specified by the prompt evaluation criterion, computing respective evaluation scores associated with the candidate prompts ([0092] each evaluation in the batch of evaluations 306 is provided from the LLM system 220 in response to respective evaluation prompts provided by the evaluation module 206. For example, the evaluation module 206 can generate an evaluation prompt for each output in the batch of outputs 304 and prompts the LLM of the LLM system 220 using the evaluation prompt); and,
based at least in part on the evaluation scores, replacing the current-iteration prompt ([Fig. 4, 412 “Update prompt template”], [0169] If the score does not exceed the threshold score, the prompt template is updated (412)); and,
outputting a final prompt generated in a final iteration of the plurality of iterations ([0169] If the score does exceed the threshold score, the prompt template is stored for production use (414), [storing prompts based on a threshold being met (indicating a final iteration in a plurality of iterations) requires an outputting of the final prompt from the evaluation module to a storage unit]).
Regarding claim 13, Jia in view of Qadrud-Din discloses: the method of claim 11.
Jia further discloses:
computing the evaluation scores at least in part at an evaluation machine learning model ([0168] the evaluation module 206 can make an API call to the LLM system 220, the call including the evaluation prompt, where the LLM system 220 returns the evaluation 232).
Regarding claim 17, Jia in view of Qadrud-Din discloses: the method of claim 11.
Qadrud-Din further discloses:
wherein:
the prompt generation instructions further specify a structured input format ([Fig. 10, generated Timeline prompt 1008 based on Chunk 1010 of input text], [0225] determines one or more timeline generation prompts 1008 based on the timeline generation request message 1004. In some embodiments, the determination of the one or more timeline prompts may involve processing one or more input documents via the chunker. As discussed herein, for instance with respect to the methods 500 and 600 shown in FIG. 5 and FIG. 6, the chunker may perform one or more operations such as pre-processing, sharding, and/or chunking the documents into manageable text, [Developing prompts for each chunk of received text indicates the input format to be specified to be in chunks]); and
the method further comprises, in the prompt generation loop, generating the candidate prompts in the structured input format ([0332] The prompt template may also include one or more additional instructions, such as an instruction to format the text generated by the text generation model as structured text. For instance, the structured text may be implemented as a JSON list).
Regarding claim 19, Jia in view of Qadrud-Din discloses: the method of claim 11.
Jia further discloses:
wherein:
the prompt generation instructions indicate a mutable portion of the initial prompt and an immutable portion of the initial prompt ([0020] prompt templates include static input and dynamic input, [Mutable and immutable track to dynamic and static respectively]); and
the method further comprises, in the prompt generation loop, modifying the mutable portion of the initial prompt while leaving the immutable portion unchanged ([0020] Here, the static input is the same for each prompt and each invocation of the LLM (each time the LLM is prompted), and the dynamic input includes data dictated by user interaction for each invocation of the LLM).
Regarding claim 20, Jia discloses: a computing system (Abstract, Methods, systems, and computer-readable storage media) comprising:
one or more processing devices ([Fig. 6, Processor 610]) configured to:
via a graphical user interface (GUI) ([0178] the input/output device 640 includes a display unit for displaying graphical user interfaces), receive prompt generation instructions that specify an initial prompt ([0017] providing an initial version of a prompt template, the prompt template including dynamic input and first static input, generating a prompt using the initial version of the prompt template) and a prompt evaluation criterion ([0017] receiving, from a large language model (LLM), an output that is responsive to the prompt, providing an evaluation at least partially based on the output, [The instructions would be similar to “use an LLM to generate prompt evaluations”]); and
based at least in part on the prompt generation instructions, programmatically assemble a prompt iteration module ([The examiner notes that the structure of Fig. 2 suggests a looping structure between prompt generation module 202, prompting module 204, evaluation module 206, and prompt template update module 208 indicating prompt iterations in view of the looping of Fig. 4 ([0169])]) from a candidate prompt generator module ([Fig. 2, Prompt Generation Module 202]) and an evaluator module ([Fig. 2, Evaluation Module 206]).
Jia does not disclose:
wherein:
the candidate prompt generator module and the evaluator module are stored in a prompt generation module library that includes a plurality of candidate prompt generator modules and a plurality of evaluator modules;
the prompt evaluation criterion is a natural language input; and
the one or more processing devices are configured to select the evaluator module at least in part by converting the prompt evaluation criterion into a module selection of the evaluator module.
Qadrud-Din discloses:
wherein:
the candidate prompt generator module and the evaluator module are stored in a prompt generation module library that includes a plurality of candidate prompt generator modules ([Fig. 2, Skills 232, 234, wherein skill 234 contains multiple prompt templates 236, 238], [Containing multiple skills, wherein each skill can contain multiple prompt templates, indicates the orchestrator to be a prompt generation module library comprised of a plurality of candidate prompt generator modules, i.e. skills]) and a plurality of evaluator modules ([0057] perform operations such as testing prompts created based on prompt templates against tests stored in the test repository 224, [Testing prompts based on tests (plurality emphasized) stored in a repository indicates a plurality of evaluator modules in the form of tests]);
the prompt evaluation criterion is a natural language input ([0041] the criteria evaluation prompt includes an instruction to evaluate some or all of the subset of the text portions against the natural language criterion, [The examiner asserts that a criteria evaluation prompt will necessarily contain the criteria to be evaluated, which is disclosed to be natural language]); and
the one or more processing devices are configured to select the evaluator module at least in part by converting the prompt evaluation criterion into a module selection of the evaluator module ([0318] one or more prompts may be tested by the prompt testing utility 226 against one or more tests stored in the test repository 224. A test result may be evaluated using the method 1200 to determine whether a prompt constructed from a prompt template being tested resulted in the generation of a hallucination, [Testing prompts against one or more tests for hallucinations indicates each test to be an evaluation module, wherein determining there is a hallucination indicates converting criteria into a module, i.e. test, for hallucination. Applying the evaluation module of prompts of Jia to the various criteria/tests of Qadrud-Din will effectively result in different evaluations for different prompts as Qadrud-Din discloses multiple prompt templates (236, 238) and a test repository 224]).
Jia and Qadrud-Din are considered analogous art within prompt augmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia to incorporate the teachings of Qadrud-Din, because of the novel way to automatically evaluate documents against a policy through use of a document pipeline which identifies text that should be kept together and extraneous text, allowing for segmentation of input text with more accurate responses generated through removal of extraneous input (Qadrud-Din, [0023]-[0028]).
Jia further discloses:
in each of a plurality of iterations of a prompt generation loop executed by the prompt iteration module:
generate a plurality of candidate prompts at least in part at a machine learning model ([0064] the LLM is used to generate an updated version of the prompt template, [0171] In some examples, the prompt template is an initial version for an initial iteration of optimization. In some examples, the prompt template is an updated version for a next iteration of optimization. A batch of prompts is generated (504)),
wherein the candidate prompts are generated based at least in part on a current-iteration prompt that is initialized as the initial prompt in a first iteration of the plurality of iterations ([0083] After ten (10) iterations of batch-based optimization, an updated version of the prompt template can be provided, [Performing batch-generation of prompts ten times indicates the first optimization to be applied to an initial version in view of [0171] cited above, wherein ten iterations indicates a plurality]);
as specified by the prompt evaluation criterion, compute respective evaluation scores associated with the candidate prompts ([0092] each evaluation in the batch of evaluations 306 is provided from the LLM system 220 in response to respective evaluation prompts provided by the evaluation module 206. For example, the evaluation module 206 can generate an evaluation prompt for each output in the batch of outputs 304 and prompts the LLM of the LLM system 220 using the evaluation prompt); and,
based at least in part on the evaluation scores, replace the current-iteration prompt ([Fig. 4, 412 “Update prompt template”], [0169] If the score does not exceed the threshold score, the prompt template is updated (412));
compute a compiled prompt that includes the final prompt and further includes prompt input data received via the GUI ([Fig. 5, Update Prompt Template 510], [In view of the dynamic input of Jia ([0020]) which changes for each invocation of the LLM based on data dictated by user interaction, indicating updating a “final” prompt to be further including prompt input received via the GUI for the dynamic input, wherein the updating prompt is necessarily “compiled” to generate validation prompts from (Step 512)]);
at the machine learning model, process the compiled prompt to generate a compiled prompt response ([Fig. 5, Receive validation outputs from LLM 514], [A validation output tracks to a form of prompt response, i.e. is the prompt valid?]); and
output the compiled prompt response to the GUI ([0177] display graphical information for a user interface on the input/output device 640, [0169] If the score does exceed the threshold score, the prompt template is stored for production use (414), [storing prompts based on a threshold being met (indicating a final iteration in a plurality of iterations) requires an outputting of the final prompt from the evaluation module to a storage unit, wherein the prompt could be output to the GUI without a change in functionality to Jia as the output prompt to be stored could be “stored” on the user interface]).
Claim(s) 2, 4-5, 9, 12, 14, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jia in view of Qadrud-Din, further in view of Vandeputte et al. (US-20250307289-A1), hereinafter Vandeputte.
Regarding claim 2, Jia in view of Qadrud-Din discloses: the computing system of claim 1.
Jia in view of Qadrud-Din does not disclose:
wherein the one or more processing devices are further configured to:
store the final prompt as a prompt fragment in a prompt library that includes a plurality of other prompt fragments;
compute a compiled prompt that includes the final prompt and one or more of the other prompt fragments;
at the machine learning model, process the compiled prompt to generate a compiled prompt response; and
output the compiled prompt response.
Vandeputte discloses:
wherein the one or more processing devices are further configured to:
store the final prompt as a prompt fragment in a prompt library that includes a plurality of other prompt fragments ([Fig. 2, Reference Prompt 220 containing segments 222-226 to be adjusted], [0062] provided with a reference prompt 220 as an input, [0064] one or more augmented prompts 230, 240 are obtained by adjusting or mutating one or more input segments with respect to the reference prompt 220, [Obtaining a reference prompt comprising a plurality of segments 222-226 to be augmented indicates the reference prompt to necessarily be stored before being retrieved for augmentation (in any iteration other than a first), wherein each segment of the prompt corresponds to a prompt fragment, indicating a plurality of prompt fragments to form a prompt library within the reference prompt. The reference prompt is a final prompt at the time of augmentation for any iteration beyond a first, i.e. a reference prompt with respect to a previous iteration output, final prompt, to be improved upon. Vandeputte discloses a reference prompt may be any input sequence ([0017]) indicating the final prompt as generated in Jia could be applied as the reference prompt of Vandeputte]);
compute a compiled prompt that includes the final prompt and one or more of the other prompt fragments ([Fig. 2, Augmented Prompt 230], [0065] Adjusting the reference prompt 220 may thus comprise selecting an input segment from the set of possible input segments and subsequently adding it at any position within the ordered reference prompt 220. Adding a sampled input segment can be achieved by appending the selected input segment, e.g. 231, to the reference prompt 220 as illustrated by augmented prompt 230, [Segment 232 represents a prompt fragment not previously part of the reference, i.e. final, prompt]);
at the machine learning model, process the compiled prompt to generate a compiled prompt response ([Fig. 5, Generative AI Model 503], [0089] This augmented prompt may then be provided to the generative AI model 503 as an input, which generates an output sequence in response); and
output the compiled prompt response ([Fig. 5, Output 513], [0092] the generated output sequence is provided as the final output 513).
Jia, Qadrud-Din, and Vandeputte are considered analogous art within prompt augmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia in view of Qadrud-Din to incorporate the teachings of Vandeputte, because of the novel way to supplement prompts with high-impact information (as determined through quantitative metrics of augmentation effectiveness), resulting in fewer required iterations of a generative AI model to arrive at a final output, reducing required computing resources.
Regarding claim 4, Jia in view of Qadrud-Din discloses: the computing system of claim 1.
Jia in view of Qadrud-Din does not disclose:
wherein, during each of the iterations of the prompt generation loop, the one or more processing devices are further configured to:
insert one or more test input portions into each of the candidate prompts to obtain a plurality of test prompts; and
at the machine learning model, process the test prompts to compute a plurality of test outputs; and
compute the evaluation scores based at least in part on the test outputs.
Vandeputte discloses:
wherein, during each of the iterations of the prompt generation loop, the one or more processing devices are further configured to:
insert one or more test input portions into each of the candidate prompts to obtain a plurality of test prompts ([Fig. 2, Augmented Prompt 230], [0065] Adjusting the reference prompt 220 may thus comprise selecting an input segment from the set of possible input segments and subsequently adding it at any position within the ordered reference prompt 220. Adding a sampled input segment can be achieved by appending the selected input segment, e.g. 231, to the reference prompt 220 as illustrated by augmented prompt 230, [In view of the plurality of generated augmented prompts 230, 240, 250 indicating a plurality of test prompts]); and
at the machine learning model, process the test prompts to compute a plurality of test outputs ([Fig. 5, Generative AI Model 503], [0089] This augmented prompt may then be provided to the generative AI model 503 as an input, which generates an output sequence in response, [In view of the plurality of test prompts 230, 240, 250 indicating a plurality of test outputs]); and
compute the evaluation scores based at least in part on the test outputs ([Fig. 5, Prompt Importance Analysis 504], [0090] Module 504 may further be configured to determine if, and how many augmented prompts need to be obtained from respective reference prompts, as well as for how many target output sequences the prompt importance scores are to be determined. At least one prompt input sequence and target output sequence are obtained when determining the scores. Multiple target output sequences may be selected during the same analysis to smoothen the prompt importance scores. The obtained prompt importance scores allow evaluating the effectiveness of the input segments in the prompt 501, 502).
Jia, Qadrud-Din, and Vandeputte are considered analogous art within prompt augmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia in view of Qadrud-Din to incorporate the teachings of Vandeputte, because of the novel way to supplement prompts with high-impact information (as determined through quantitative metrics of augmentation effectiveness), resulting in fewer required iterations of a generative AI model to arrive at a final output, reducing required computing resources.
Regarding claim 5, Jia in view of Qadrud-Din in view of Vandeputte discloses: the computing system of claim 4.
Vandeputte further discloses:
wherein the one or more processing devices are configured to:
generate a respective plurality of the test prompts for each of the candidate prompts ([Fig. 2, Augmented Prompt 230], [0065] Adjusting the reference prompt 220 may thus comprise selecting an input segment from the set of possible input segments and subsequently adding it at any position within the ordered reference prompt 220. Adding a sampled input segment can be achieved by appending the selected input segment, e.g. 231, to the reference prompt 220 as illustrated by augmented prompt 230, [In view of the plurality of generated augmented prompts 230, 240, 250 indicating a plurality of test prompts]).
Jia further discloses:
repeat the prompt generation loop until, for at least one of the candidate prompts, each of the test prompts generated from that candidate prompt exceeds a predefined evaluation score threshold ([0074] if an evaluation metric (e.g., groundness score, conciseness score, coherence scores, custom score) meets a respective threshold score, it can be determined that the prompt template need not be updated (e.g., the prompt template is considered optimized). If an evaluation metric (e.g., groundness score, conciseness score, coherence scores, custom score) does not meet a respective threshold score, it can be determined that the prompt template is to be updated (e.g., the prompt template is considered non-optimized), [Wherein a non-optimized prompt corresponds to a test prompt and/or candidate prompts of Vandeputte depending on the stage of updating (the original candidate prompt is a first test prompt, further updates are additional test prompts corresponding to the original candidate/test). Further, updating a prompt template is indicative of a prompt generation representing the updated template]).
Regarding claim 9, Jia in view of Qadrud-Din discloses: the computing system of claim 1.
Jia in view of Qadrud-Din does not disclose:
wherein:
the initial prompt is structured as a plurality of prompt chunks; and
in the prompt generation loop, the one or more processing devices are configured to generate the candidate prompts as candidate orderings of the prompt chunks.
Vandeputte discloses:
wherein:
the initial prompt is structured as a plurality of prompt chunks ([Fig. 2, Reference Prompt 220 comprising prompt “chunks” 222, 224, 226], [In a first iteration, the reference prompt will be an initial prompt which has not had any augmentations performed]); and
in the prompt generation loop, the one or more processing devices are configured to generate the candidate prompts as candidate orderings of the prompt chunks ([Fig. 2, Augmented Prompt 230], [0065] Adjusting the reference prompt 220 may thus comprise selecting an input segment from the set of possible input segments and subsequently adding it at any position within the ordered reference prompt 220. Adding a sampled input segment can be achieved by appending the selected input segment, e.g. 231, to the reference prompt 220 as illustrated by augmented prompt 230, [In view of the plurality of generated augmented prompts 230, 240, 250 indicating a plurality of candidate prompts with candidate orderings based on the total amount of segments]).
Jia, Qadrud-Din and Vandeputte are considered analogous art within prompt augmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia in view of Qadrud-Din to incorporate the teachings of Vandeputte, because of the novel way to supplement prompts with high-impact information (as determined through quantitative metrics of augmentation effectiveness), resulting in fewer required iterations of a generative AI model to arrive at a final output, reducing required computing resources.
Regarding claim 12, Jia in view of Qadrud-Din discloses: the method of claim 11.
Jia in view of Qadrud-Din does not disclose:
storing the final prompt as a prompt fragment in a prompt library that includes a plurality of other prompt fragments;
computing a compiled prompt that includes the final prompt and one or more of the other prompt fragments;
at the machine learning model, processing the compiled prompt to generate a compiled prompt response; and
outputting the compiled prompt response.
Vandeputte discloses:
storing the final prompt as a prompt fragment in a prompt library that includes a plurality of other prompt fragments ([Fig. 2, Reference Prompt 220 containing segments 222-226 to be adjusted], [0062] provided with a reference prompt 220 as an input, [0064] one or more augmented prompts 230, 240 are obtained by adjusting or mutating one or more input segments with respect to the reference prompt 220, [Obtaining a reference prompt comprising a plurality of segments 222-226 to be augmented indicates the reference prompt to necessarily be stored before being retrieved for augmentation (in any iteration other than a first), wherein each segment of the prompt corresponds to a prompt fragment, indicating a plurality of prompt fragments to form a prompt library within the reference prompt. The reference prompt is a final prompt at the time of augmentation for any iteration beyond a first, i.e. a reference prompt with respect to a previous iteration output, final prompt, to be improved upon. Vandeputte discloses a reference prompt may be any input sequence ([0017]) indicating the final prompt as generated in Jia could be applied as the reference prompt of Vandeputte]);
computing a compiled prompt that includes the final prompt and one or more of the other prompt fragments ([Fig. 2, Augmented Prompt 230], [0065] Adjusting the reference prompt 220 may thus comprise selecting an input segment from the set of possible input segments and subsequently adding it at any position within the ordered reference prompt 220. Adding a sampled input segment can be achieved by appending the selected input segment, e.g. 231, to the reference prompt 220 as illustrated by augmented prompt 230, [Segment 232 represents a prompt fragment not previously part of the reference, i.e. final, prompt]);
at the machine learning model, processing the compiled prompt to generate a compiled prompt response ([Fig. 5, Generative AI Model 503], [0089] This augmented prompt may then be provided to the generative AI model 503 as an input, which generates an output sequence in response); and
outputting the compiled prompt response ([Fig. 5, Output 513], [0092] the generated output sequence is provided as the final output 513).
Jia, Qadrud-Din, and Vandeputte are considered analogous art within prompt augmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia in view of Qadrud-Din to incorporate the teachings of Vandeputte, because of the novel way to supplement prompts with high-impact information (as determined through quantitative metrics of augmentation effectiveness), resulting in fewer required iterations of a generative AI model to arrive at a final output, reducing required computing resources.
Regarding claim 14, Jia in view of Qadrud-Din discloses: the method of claim 11.
Jia in view of Qadrud-Din does not disclose:
during each of the iterations of the prompt generation loop:
inserting one or more test input portions into each of the candidate prompts to obtain a plurality of test prompts; and
at the machine learning model, processing the test prompts to compute a plurality of test outputs; and
computing the evaluation scores based at least in part on the test outputs.
Vandeputte discloses:
during each of the iterations of the prompt generation loop:
inserting one or more test input portions into each of the candidate prompts to obtain a plurality of test prompts ([Fig. 2, Augmented Prompt 230], [0065] Adjusting the reference prompt 220 may thus comprise selecting an input segment from the set of possible input segments and subsequently adding it at any position within the ordered reference prompt 220. Adding a sampled input segment can be achieved by appending the selected input segment, e.g. 231, to the reference prompt 220 as illustrated by augmented prompt 230, [In view of the plurality of generated augmented prompts 230, 240, 250 indicating a plurality of test prompts]); and
at the machine learning model, processing the test prompts to compute a plurality of test outputs ([Fig. 5, Generative AI Model 503], [0089] This augmented prompt may then be provided to the generative AI model 503 as an input, which generates an output sequence in response, [In view of the plurality of test prompts 230, 240, 250 indicating a plurality of test outputs]); and
computing the evaluation scores based at least in part on the test outputs ([Fig. 5, Prompt Importance Analysis 504], [0090] Module 504 may further be configured to determine if, and how many augmented prompts need to be obtained from respective reference prompts, as well as for how many target output sequences the prompt importance scores are to be determined. At least one prompt input sequence and target output sequence are obtained when determining the scores. Multiple target output sequences may be selected during the same analysis to smoothen the prompt importance scores. The obtained prompt importance scores allow evaluating the effectiveness of the input segments in the prompt 501, 502).
Jia, Qadrud-Din, and Vandeputte are considered analogous art within prompt augmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia in view of Qadrud-Din to incorporate the teachings of Vandeputte, because of the novel way to supplement prompts with high-impact information (as determined through quantitative metrics of augmentation effectiveness), resulting in fewer required iterations of a generative AI model to arrive at a final output, reducing required computing resources.
Regarding claim 18, Jia in view of Qadrud-Din discloses: the method of claim 11.
Jia in view of Qadrud-Din does not disclose:
wherein:
the initial prompt is structured as a plurality of prompt chunks; and
the method further comprises, in the prompt generation loop, generating the candidate prompts as candidate orderings of the prompt chunks.
Vandeputte discloses:
wherein:
the initial prompt is structured as a plurality of prompt chunks ([Fig. 2, Reference Prompt 220 comprising prompt “chunks” 222, 224, 226], [In a first iteration, the reference prompt will be an initial prompt which has not had any augmentations performed]); and
the method further comprises, in the prompt generation loop, generating the candidate prompts as candidate orderings of the prompt chunks ([Fig. 2, Augmented Prompt 230], [0065] Adjusting the reference prompt 220 may thus comprise selecting an input segment from the set of possible input segments and subsequently adding it at any position within the ordered reference prompt 220. Adding a sampled input segment can be achieved by appending the selected input segment, e.g. 231, to the reference prompt 220 as illustrated by augmented prompt 230, [In view of the plurality of generated augmented prompts 230, 240, 250 indicating a plurality of candidate prompts with candidate orderings based on the total amount of segments]).
Jia, Qadrud-Din, and Vandeputte are considered analogous art within prompt augmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia in view of Qadrud-Din to incorporate the teachings of Vandeputte, because of the novel way to supplement prompts with high-impact information (as determined through quantitative metrics of augmentation effectiveness), resulting in fewer required iterations of a generative AI model to arrive at a final output, reducing required computing resources.
Claim(s) 6, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jia in view of Qadrud-Din, further in view of Shea et al. (US-20250110975-A1), hereinafter Shea.
Regarding claim 6, Jia in view of Qadrud-Din discloses: the computing system of claim 1.
Jia in view of Qadrud-Din does not disclose:
wherein the final prompt includes one or more non-ASCII characters.
Shea discloses:
wherein the final prompt includes one or more non-ASCII characters ([0043] The string prompt may include letters, numbers, whitespace, punctuation, and in some cases formatting. Similarly, the generative output of a generative output engine as described herein can be formatted/encoded according to any suitable encoding (e.g., ISO, Unicode, ASCII as examples), [Wherein Unicode includes encodings for symbols and emojis, non-ASCII characters]).
Jia, Qadrud-Din, and Shea are considered analogous art within prompt engineering. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia in view of Qadrud-Din to incorporate the teachings of Shea, because of the novel way to leverage retained data (wherein the data includes non-ASCII characters) including past prompt results for training and functionality improvement in the context of prompt engineering resulting in further customized prompts for particular users, sessions, or use histories (Shea, [0187]).
Regarding claim 15, Jia in view of Qadrud-Din discloses: the method of claim 11.
Jia in view of Qadrud-Din does not disclose:
wherein the final prompt includes one or more non-ASCII characters.
Shea discloses:
wherein the final prompt includes one or more non-ASCII characters ([0043] The string prompt may include letters, numbers, whitespace, punctuation, and in some cases formatting. Similarly, the generative output of a generative output engine as described herein can be formatted/encoded according to any suitable encoding (e.g., ISO, Unicode, ASCII as examples), [Wherein Unicode includes encodings for symbols and emojis, non-ASCII characters]).
Jia, Qadrud-Din, and Shea are considered analogous art within prompt engineering. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia in view of Qadrud-Din to incorporate the teachings of Shea, because of the novel way to leverage retained data (wherein the data includes non-ASCII characters) including past prompt results for training and functionality improvement in the context of prompt engineering resulting in further customized prompts for particular users, sessions, or use histories (Shea, [0187]).
Claim(s) 7, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jia in view of Qadrud-Din, further in view of Tu et al. (US-20240330603-A1), hereinafter Tu.
Regarding claim 7, Jia in view of Qadrud-Din discloses: the computing system of claim 1.
Jia further discloses:
wherein the prompt generation instructions further specify a machine learning model task ([Col. 3, Table 2] {TASK} Answer: Is the submission concise and to the point? {TASK} Answer: Is the submission coherent, well- structured, and organized? {TASK} Answer: [customized question] {DATA} {LLM output}, [Having LLM output as data indicates a machine learning model performing a task as is necessarily required to produce output]).
Jia in view of Qadrud-Din does not disclose:
wherein in the prompt generation loop, the one or more processing devices are configured to generate the candidate prompts such that the candidate prompts include one or more few-shot examples of the machine learning model task.
Tu discloses:
wherein in the prompt generation loop, the one or more processing devices are configured to generate the candidate prompts such that the candidate prompts include one or more few-shot examples of the machine learning model task ([0073] 5-shots, 15-shots, and all-shots for vanilla classifiers on intent tasks…As illustrated, aligned prompts can further improve performance, with the best results obtained in few-shot settings. Additionally, the variances in task performance across all languages with aligned prompts are significantly smaller than those observed with fine-tuning and prompt tuning only. Although prompt tuning achieves higher accuracies on few-shot settings, [Intent classification, i.e. a machine learning model task, in a few-shot learning setting for aligning prompts indicates generation of candidate, i.e. aligned prompts using few-shot intent classification examples]).
Jia, Qadrud-Din, and Tu are considered analogous art within prompt fine-tuning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia in view of Qadrud-Din to incorporate the teachings of Tu, because of the novel way to adapt prompts for performing tasks which will function well across different languages (modifying the prompt to be compatible with multiple languages), improving prompt tuning methods across different language domains which will result in more accurate cross-lingual tasks with the same prompt (Tu, [0017]-[0018]).
Regarding claim 16, Jia in view of Qadrud-Din discloses: the method of claim 11.
Jia further discloses:
wherein the prompt generation instructions further specify a machine learning model task ([Col. 3, Table 2] {TASK} Answer: Is the submission concise and to the point? {TASK} Answer: Is the submission coherent, well- structured, and organized? {TASK} Answer: [customized question] {DATA} {LLM output}, [Having LLM output as data indicates a machine learning model performing a task as is necessarily required to produce output]).
Jia in view of Qadrud-Din does not disclose:
the method further comprises, in the prompt generation loop, generating the candidate prompts such that the candidate prompts include one or more few-shot examples of the machine learning model task.
Tu discloses:
the method further comprises, in the prompt generation loop, generating the candidate prompts such that the candidate prompts include one or more few-shot examples of the machine learning model task ([0073] 5-shots, 15-shots, and all-shots for vanilla classifiers on intent tasks…As illustrated, aligned prompts can further improve performance, with the best results obtained in few-shot settings. Additionally, the variances in task performance across all languages with aligned prompts are significantly smaller than those observed with fine-tuning and prompt tuning only. Although prompt tuning achieves higher accuracies on few-shot settings, [Intent classification, i.e. a machine learning model task, in a few-shot learning setting for aligning prompts indicates generation of candidate, i.e. aligned prompts using few-shot intent classification examples]).
Jia, Qadrud-Din, and Tu are considered analogous art within prompt fine-tuning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jia in view of Qadrud-Din to incorporate the teachings of Tu, because of the novel way to adapt prompts for performing tasks which will function well across different languages (modifying the prompt to be compatible with multiple languages), improving prompt tuning methods across different language domains which will result in more accurate cross-lingual tasks with the same prompt (Tu, [0017]-[0018]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Miller et al. (US-20250131192-A1) discloses “A computing device and methods of making and using a computing device having machine learning capabilities to analyze course text content based on prompting to generate a list of course learning objectives, and in particular embodiments, having machine learning capabilities to analyze presentation content text against each of the course learning objectives to generate a course competency score with supportive reasoning for each course learning objective and an overall presentation score” (abstract). See entire document.
Srinivasan et al. (US-20250111169-A1) discloses “Methods, systems, and devices for processing prompts by an array of large language models (LLMs). The system may provide the prompt to multiple LLMs. The multiple LLMs are trained on different datasets and have different knowledge and capabilities. The system receives multiple responses from the multiple LLMs, determines a rank for each of the multiple responses, the rank indicating a level of confidence of the corresponding response representing a ground truth, compares the ranks of the multiple responses, and selects the response having the best rank as the most probable ground truth response” (abstract). See entire document.
Heller et al. (US-12067366-B1) discloses “Text generation prompts may be determined based on an input document and a text generation prompt template. The text generation prompts may include text from the input document and questions related to the text. The text generation prompts may be sent to a remote text generation modeling system, which may respond with text generation prompt response messages including novel text portions generated by a text generation model. The text generation prompt response messages may be parsed to generate answers corresponding with the questions” (abstract). See entire document.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THEODORE JOHN WITHEY whose telephone number is (703)756-1754. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571) 272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THEODORE WITHEY/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights