DETAILED ACTION
This communication is in response to the Application filed on 01/02/2026. Claims 1-7, 9-17, and 19-22 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/02/2026 has been entered.
Priority
Applicant claims the benefit of US Provisional Application No. 63/499,045, filed April 28, 2023. Claims 1-7, 9-17, and 19-22 have been afforded the benefit of this filing date.
Response to Arguments
The reply filed on 01/02/2026 has been entered. Applicant’s arguments with respect to claims 1-7, 9-17, and 19-22 have been considered but are not persuasive/moot in view of new ground(s) of rejection caused by the amendments.
With respect to the applicant’s arguments to claim rejections under 35 U.S.C § 101, Applicant has amended each of the independent claims and asserts that “the independent claims of the subject application recite a new model architecture and the prompt rewriting strategy represents a technical improvement to a machine learning model.” The examiner agrees that these newly added limitations overcome the rejection under 35 U.S.C. 101.
With respect to the applicant’s arguments to claim rejections under 35 U.S.C § 103, Applicant has amended each of the independent claims and asserts that “Cai discloses an interface by which a user can update the prompt itself but not the assessment criteria.” The examiner respectfully disagrees with these assertions. Cai et al. discloses the ability for users to select helper nodes which help provide data evaluation functionality (Cai et al. ¶ [0100]) such as toxicity filters (Cai et al., Table 3). Data evaluation is considered analogous to assessment criteria because both Cai et al.’s data evaluation and the instant application’s assessment criteria share the common purpose of evaluating LLM output against certain criteria. Further detail can be found below with respect to claim rejections under 35 USC § 103.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 9, 10-12, 14, and 19-20 are rejected under 35 U.S.C. 103 as obvious over "Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback" (Peng et al.) in view of US Patent Publication 20230112921 A1 (Cai et al.).
Claim 1
Regarding claim 1, Peng et al. disclose receiving, [via the prompt interface,] a prompt from a user including an instruction for the LLM to generate an output (Peng et al. pg. 2, Section 1, paragraph 2, "As illustrated by the example in Figure 1, given a user query (e.g., regarding a 2013 Los Angeles Galaxy player transfer), LLM-AUGMENTER first retrieves evidence from external knowledge ... ");
providing first input including the prompt to the LLM (Peng et al. pg. 2, Section 1, paragraph 2, "Then, LLM-AUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence for ChatGPT to generate a candidate response grounded in external knowledge (evidence).");
generating, in response to the first input, a first response to the prompt via the LLM (Peng et al. pg. 2, Section 1, paragraph 2, "Then, LLM-AUGMENTER queries a fixed LLM (i.e., ChatGPT in our study) using a prompt that contains the consolidated evidence for ChatGPT to generate a candidate response grounded in external knowledge (evidence).");
performing assessment and revision of the prompt, at least in part by:
assessing the first response according to [the] one or more assessment criteria (Peng et al. pg. 2, Section 1, paragraph 2, "LLM-AUGMENTER then verifies the candidate response e.g., by checking whether it hallucinates evidence." Checking for hallucinations is considered analogous to an assessment criteria) to generate an assessment report for the first response, via the LLM (Peng et al. pg. 2, Section 1, paragraph 2, "If so, LLM-AUGMENTER generates a feedback message (e.g., about the team “C.S.D. Municipal”).");
providing second input including the first prompt, the first response, the assessment report, and a prompt revision instruction to revise the prompt in view of the first assessment report to the LLM (Peng et al. pg. 2, Section 2.4, paragraph 4, "In addition, we have developed a utility function to generate informative and actionable feedback to help revise prompts to allow the LLM to generate better responses. ... Such a utility function is a text generation model
Q
parameterized by
ψ
, and can be implemented as a seq2seq or auto-regression language model. It tasks as input user query
q
, evidence
e
, candidate response
o
and dialog history
h
q
, and generates feedback in text
f
as
f
=
Q
(
q
;
e
;
o
;
h
q
)
";
f
is considered analogous to both the claimed assessment report and claimed prompt revision instruction,
q
is considered analogous to the claimed first prompt,
o
is considered analogous to the first response); and
generating a revised prompt in response to the second input, via the LLM (Peng et al. pg. 4, Section 2.3.2, paragraph 1, "The Prompt Engine generates a prompt to query the LLM to generate a (candidate) response
o
for
q
. The prompt is a text string that consists of task instruction, user query
q
, dialog history
h
q
, evidence
e
if it is made available by Knowledge Consolidator, and feedback
f
if it is made available by the Utility module.");
providing final input including the revised prompt to the LLM (Peng et al. pg. 2, Section 1, paragraph 2, "The message is used to revise the prompt to query ChatGPT again.");
in response to the final input, generating a final response to the revised prompt, via the LLM (Peng et al. pg. 2, Section 1, paragraph 2, "The process iterates until a candidate response passes the verification and is sent to the user."); and
outputting the final response to the user (Peng et al. pg. 2, Section 1, paragraph 2).
Peng et al. do not explicitly disclose all of a prompt interface.
However, Cai et al. disclose a computing system for revising large language model (LLM) input prompts, the computing system comprising:
at least one processor configured to (Cai et al. ¶ [0099], "The user computing device 102 includes one or more processors 112 and a memory 114."):
cause a prompt interface for a trained LLM to be presented (Cai et al. ¶ [0041], "the provided user interface can enable the user to ... view and edit the inputs, outputs, and/or prompts for each instantiation within the chain");
receive, via the prompt interface, a prompt from a user including an instruction for the LLM to generate an output (Cai et al. ¶ [0041], "the provided user interface can enable the user to ... view and edit the inputs, outputs, and/or prompts for each instantiation within the chain. In some implementations, a user can be enabled to select a prompt for a given instantiation from a set of primitive operations defined herein which are useful for chain construction and refinement."); and
perform assessment and revision of the prompt, at least in part by:
receiving a user input of one or more assessment criteria via an input pane of the prompt interface (Cai et al. ¶ [0099], "the node library can provide a user one or more available node types from which a user can select at one or more nodes. ... helper nodes [help to] address data transformation and evaluation needs. The computing system can provide commonly-used Processing and Evaluation helpers." Nodes in a model chaining system are considered analogous to assessment criteria. For example, see Toxicity classifier node in Fig. 3A), the user input being a selection of the one or more assessment criteria from [one or more suggested assessment criteria that are suggested by the LLM and/or] one or more predetermined assessment criteria (Cai et al. ¶ [0099], "the node library can provide a user one or more available node types from which a user can select at one or more nodes." See Table 3 for a list of predetermined nodes.); and
assessing the first response according to the one or more assessment criteria (Cai et al. pg. 25, Table 3, "Helper Evaluation nodes: Filter undesired LLM outputs, or Re-ranking multiple outputs based on human-designed criteria, e.g., whether the reply is concise, polite, etc." Evaluating, filtering, or re-ranking LLM outputs is considered analogous to assessing an LLM response according to one or more assessment criteria. See Fig. 3A) [to generate an assessment report for the first response, via the LLM];
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Peng et al. to incorporate Cai et al.’s prompt interface.
The suggestion/motivation for doing so would have been that, “The user interface provides an internal view of the state of the system and enables the user to better operate the system. This additionally provides a better means of user input for operating the system compared to for example, a single monolithic LLM which as discussed above can be difficult for a user to determine the correct input prompt to obtain a desired output,” as noted by the Cai et al. disclosure in paragraph [0050].
Claim 2
Regarding claim 2, the rejection of claim 1 is incorporated.
Peng et al. further disclose wherein the assessment and revision of the prompt is performed iteratively for a plurality of iterations (Peng et al. pg. 2, Section 1, paragraph 2, "The process iterates until a candidate response passes the verification and is sent to the user.").
Claim 4
Regarding claim 4, the rejection of claim 2 is incorporated.
Peng et al. further disclose wherein the at least one processor is further configured to output the final response generated after the plurality of iterations to the user without outputting any intermediate responses to the user (Peng et al. pg. 3, Section 2.1, paragraph 2, "Note that given user query
q
, LLM-AUGMENTER can take multiple iterations to revise its response, with each iteration generating a candidate response based on evidence, feedback and utility, before sending the final response to the user, as illustrated in Figure 1.").
Claim 9
Regarding claim 9, the rejection of claim 1 is incorporated.
Peng et al. further disclose wherein the assessment report includes one or both of a score and a written description of how well the first response met the assessment criteria (Peng et al. pg. 5, Section 3.2, paragraph 5, "To evaluate the degree to which the generated responses are grounded in consolidated evidence, we use the utility score, Knowledge F1 (Shuster et al., 2021), to measure the overlap between a prediction and evidence which is either consolidated by Knowledge Consolidator or provided as golden knowledge. Feedback generation is accomplished using a template-based natural language generator.").
Claim 10
Regarding claim 10, the rejection of claim 1 is incorporated.
Cai et al. further disclose wherein the at least one processor is further configured to:
cause a prompt revision element to be displayed (Cai et al. ¶ [0096], "The interface in FIG. 2A-C, FIGS. 3A-B, 4 and 5A-F includes or is operable in three primary views or modes: the chain view (FIGS. 2A 3A, 4 and 5A-F), the step view (FIGS. 2B/C), and the node view (FIG. 3B)." ¶ [0102], "[The step] view in FIG. 2B allows users to explore each model step by interacting with inputs, outputs, and the underlying prompt structure." The Step View is considered analogous to a prompt revision element); and
in response to the user input selecting the prompt revision element, outputting the revised prompt to the user (Cai et al. ¶ [0104], "The step view also handles the per-step execution. Users can click the small “run” button to execute each running block individually. To improve natural language interaction transparency, this would also trigger a preview of the final prompt text (b2)." See Fig. 2B).
Claim 11
Regarding claim 11, the limitations of claim 11 are similar in scope to the limitations of claim 1 and therefore are rejected for similar reasons as described above.
Claim 12
Regarding claim 12, the rejection of claim 11 is incorporated. The limitations of claim 12 are similar in scope to that of claim 2 and therefore are rejected for similar reasons as described above.
Claim 14
Regarding claim 14, the rejection of claim 12 is incorporated. The limitations of claim 14 are similar in scope to that of claim 4 and therefore are rejected for similar reasons as described above.
Claim 19
Regarding claim 19, the rejection of claim 11 is incorporated. The limitations of claim 19 are similar in scope to that of claim 9 and therefore are rejected for similar reasons as described above.
Claim 20
Regarding claim 20, Cai et al. disclose executing a prompt interface application programming interface (API) for a trained LLM (Cai et al. ¶ [0113], "Additionally or alternatively, one or more machine-learned models 190 can be accessed as a service over the network 180. For example, the calls (e.g., requests for inference) can be made to the models 190 using one or more application programming interfaces (APIs).").
The remaining limitations of claim 20 are similar in scope to the limitations of claim 1 and therefore are mapped similarly as described above.
Claims 3 and 13 are rejected under 35 U.S.C. 103 as obvious over Peng et al. in view of Cai et al. as applied to claims 2 and 12 above, and further in view of US Patent 9454733 B1 (Purpura et al.).
Claim 3
Regarding claim 3, the rejection of claim 2 is incorporated. Peng et al. in view of Cai et al. disclose all the elements of the claimed invention as stated above.
Peng et al. in view of Cai et al. do not explicitly disclose allowing a user to customize the number of iterations for looping.
However, Purpura et al. disclose wherein the plurality of iterations is a number customizable by the user (Purpura et al. ¶ (25), "The node trains the predictive model on the new subset of training data using the updated state of the model as the initial state for the training (step 308). In particular, the node performs iterations of the L-BFGS process using the updated state of the model, i.e., the stored L-BFGS components and the updated values of the model parameters, as the initial state for the first iteration of the L-BFGS process. The number of iterations performed can be specified by a user of the model training system. ").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Peng et al. in view of Cai et al. to include Purpura et al.’s customizable number of iterations because such a modification is the result of combining prior art elements according to known methods to yield predictable results. More specifically, Peng et al. in view of Cai et al. as modified by Purpura et al.’s customizable number of iterations can yield a predictable result of giving the user more granular control over the LLM system since the user would be able to choose a specific number of times that the model should iterate. Thus, a person of ordinary skill would have appreciated including in Peng et al.'s prompt revision method the ability to customize the number of times the model iterates since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Claim 13
Regarding claim 13, the rejection of claim 12 is incorporated. The limitations of claim 13 are similar in scope to that of claim 3 and therefore are rejected for similar reasons as described above.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as obvious over Peng et al. in view of Cai et al. as applied to claims 1 and 10 above, and further in view of "PaLM-E: An Embodied Multimodal Language Model," published March 6, 2023 (Driess et al.).
Claim 5
Regarding claim 5, the rejection of claim 1 is incorporated. Peng et al. in view of Cai et al. disclose all the elements of the claimed invention as stated above.
Peng et al. in view of Cai et al. do not explicitly disclose a multimodal LLM.
However, Driess et al. disclose wherein the LLM is multimodal (Driess et al. Figure 1, "PaLM-E is a single general-purpose multimodal language model for embodied reasoning tasks, visual-language tasks, and language tasks.").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Peng et al. in view of Cai et al. to incorporate the multimodal LLM as taught by Driess et al.
The suggestion/motivation for doing so would have been that, “PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities,” as noted by the Driess et al. publication in the abstract.
Claim 15
Regarding claim 15, the rejection of claim 10 is incorporated. The limitations of claim 15 are similar in scope to that of claim 5 and therefore are rejected for similar reasons as described above.
Claims 6, 16, 21, and 22 are rejected under 35 U.S.C. 103 as obvious over Peng et al. in view of Cai et al. as applied to claims 1 and 10 above, and further in view of US Patent Publication 20240273345 A1 (Bharadwaj et al.).
Claim 6
Regarding claim 6, the rejection of claim 1 is incorporated.
Cai et al. further disclose wherein the assessment criteria [generated by the LLM] are modified by the user (Cai et al. ¶ [0044], "In addition to potentially improving outcomes, chaining opens up new channels for fine-grained human feedback and control. In particular, according to another example aspect, the present disclosure provides interactive user interfaces that expose these additional “knobs” to end users. For example, an interactive user interface can display a live visualization of the model chain structure and can allow users to customize chains at various levels. As examples, the user can be enabled to: iterate on or otherwise modify the local prompts per step; edit intermediate data transformed between steps; and/or reconstruct or modify the architecture of flow of the model chain." Modifying intermediate model steps is considered analogous to modifying assessment criteria).
Peng et al. in view of Cai et al. do not explicitly disclose all of an LLM generating assessment criteria.
However, Bharadwaj et al. disclose wherein the assessment criteria generated by the LLM (Bharadwaj et al. ¶ [0133], "the prompting optimization model may generate, based on the input query “update my product descriptions” and data retrieved by the retrieval model, prompting that recites, in part, “For each of the following product descriptions, create ten separate product description outputs that are each fit to be empathetic with a target audience of Firefighters from Chicago; however, keep the fitting abstractive and nonliteral and only weigh the audience twenty percent.”" The generated segment "empathetic with a target audience of Firefighters from Chicago" is considered analogous to generated assessment criteria) are modified by the user (Bharadwaj et al. ¶ [0183], "Users may add to, subtract from, or otherwise modify a generated response.... The learning and inference engine 708 may be configured to record interactions and determine user preferred metrics in accordance with the manner described above with respect to FIGS. 1 through 6.").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Peng et al. in view of Cai et al. to incorporate Bharadwaj et al.’s assessment criteria generation.
The suggestion/motivation for doing so would have been that, “response modules allow for granular adjustments to various components of response modules from changing prompting and configuration files to finetuning models within the modules, and so on, such that response modules can be tailored down to specific user preferences. As such, two users having otherwise similar characteristics using the same fine-tuned models can experience different outputs that fit their respective desired levels of semantic complexity or tone,” as noted by the Bharadwaj et al. disclosure in paragraph [0179].
Claim 16
Regarding claim 16, the rejection of claim 11 is incorporated. The limitations of claim 16 are similar in scope to that of claim 6 and therefore are rejected for similar reasons as described above.
Claim 21
Regarding claim 21, the rejection of claim 1 is incorporated.
Peng et al. in view of Cai et al. do not explicitly disclose generating assessment criteria
Bharadwaj et al. further disclose wherein performing assessment and revision of the prompt is further accomplished at least in part by generating assessment criteria by the LLM based on at least a target audience of the output (Bharadwaj et al. ¶ [0133], "a prompting optimization model ... is a trained generative model configured to generate prompting based on the input query.... the prompting optimization model may generate, based on the input query “update my product descriptions” and data retrieved by the retrieval model, prompting that recites, in part, “For each of the following product descriptions, create ten separate product description outputs that are each fit to be empathetic with a target audience of Firefighters from Chicago; however, keep the fitting abstractive and nonliteral and only weigh the audience twenty percent.”" The generated segment "empathetic with a target audience of Firefighters from Chicago" is considered analogous to generated assessment criteria), the target audience being provided by the user or inferred by the LLM and being different from the user (Bharadwaj et al. ¶ [0133], see above for citation. The LLM infers the target audience ("Firefighters from Chicago") from the input query and retrieved data.), and the target audience is a category (Bharadwaj et al. ¶ [0133], see above for citation. "Firefighters from Chicago" are considered analogous to a category).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Peng et al. in view of Cai et al. to incorporate Bharadwaj et al.’s assessment criteria generation.
The suggestion/motivation for doing so is similar to the suggestion/motivation described above with respect to claim 6.
Claim 22
Regarding claim 22, the rejection of claim 1 is incorporated.
Bharadwaj et al. further disclose wherein performing assessment and revision of the prompt is further accomplished at least in part by generating assessment criteria by the LLM based on at least a target audience of the output (Bharadwaj et al. ¶ [0133], "a prompting optimization model ... is a trained generative model configured to generate prompting based on the input query.... the prompting optimization model may generate, based on the input query “update my product descriptions” and data retrieved by the retrieval model, prompting that recites, in part, “For each of the following product descriptions, create ten separate product description outputs that are each fit to be empathetic with a target audience of Firefighters from Chicago; however, keep the fitting abstractive and nonliteral and only weigh the audience twenty percent.”" The generated segment "empathetic with a target audience of Firefighters from Chicago" is considered analogous to generated assessment criteria), the target audience being provided by the user or inferred by the LLM and being different from the user, and the target audience is provided by the user (Bharadwaj et al. ¶ [0172], "in response to receiving the above input query “rewrite my product descriptions to sell more hammers to Firefighters…,” a single response module may be used to generate responses for a plurality of user segments/target audiences (e.g., using the prompting model to tailor the input to the generative models to best suit each respective user segment).").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Peng et al. in view of Cai et al. to incorporate Bharadwaj et al.’s assessment criteria generation.
The suggestion/motivation for doing so is similar to the suggestion/motivation described above with respect to claim 6.
Claims 7 and 17 are rejected under 35 U.S.C. 103 as obvious over Peng et al. in view of Cai et al. as applied to claims 1 and 10 above, and further in view of US Patent Publication 20220189474 A1 (Sharifi et al.).
Claim 7
Regarding claim 7, the rejection of claim 1 is incorporated. Peng et al. in view of Cai et al. disclose all the elements of the claimed invention as stated above.
Peng et al. in view of Cai et al. do not explicitly disclose requesting further information specifying the prompt from the user.
However, Sharifi et al. disclose wherein the at least one processor is further configured to request information further specifying the prompt from the user (Sharifi et al. ¶ [0043], At block 256, the system determines, based on processing the recognition that corresponds to the spoken utterance, that the spoken utterance is ambiguous. ... When the system encounters such a situation, it will provide a clarification prompt to the user requesting clarifying details that may be used to disambiguate between the two candidate responsive action options.").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Peng et al. in view of Cai et al. to incorporate further prompting the user as taught by Sharifi et al.
The suggestion/motivation for doing so would have been that, “While natural language analysis and semantic processing enable users to use slight variations in their commands, these variations may only stray so far before natural language analysis and/or other semantic processing are unable to determine which action to perform,” as noted by the Sharifi et al. publication in paragraph [0002].
Claim 17
Regarding claim 17, the rejection of claim 10 is incorporated. The limitations of claim 17 are similar in scope to that of claim 7 and therefore are rejected for similar reasons as described above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JACOB B VOGT whose telephone number is (571)272-7028. The examiner can normally be reached Monday - Friday 9:30am - 7pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D Shah can be reached at (571)270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACOB B VOGT/ Examiner, Art Unit 2653
/Paras D Shah/ Supervisory Patent Examiner, Art Unit 2653
02/17/2026