DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to Amendment filed 12/19/2025.
Claims 1-5, 11-12, 15, 19, and 21-30 have been amended, and no claim has been canceled or added. Currently, claims 1-30 are pending.
Information Disclosure Statement
The Information Disclosure Statement (IDS) filed by Applicant on 01/14/2026 has been considered. The NPL document #5 was not considered because it was not submitted. A copy of the considered IDS is enclosed with this Office action.
Response to Arguments
Applicant's arguments filed 12/19/2025 have been fully considered but they are not fully persuasive.
Regarding Applicant’s argument (see Remarks, pages 26-27) regarding independent claim 1 and similarly applied to independent claims 22 and 27 that Gupta in view of Guttridge fails to disclose receiving, at a graphical user interface, a plurality of user inputs that at least specify one or more candidate language models, a system prompt and a user prompt, Examiner respectfully disagrees.
A user can interact with a system through some type of user interfaces (e.g., graphical user interface (GUI), command-line interface (CLI), voice user interface (VUI), etc.). User inputs received/processed by any module in the system must be through the user interface used by the system. The features (e.g., Fig. 5A and Fig. 5B) suggest that the system disclosed by Gupta et al. interacts with a user through a graphical user interface.
It should be noted that a graphical user interface can broadly interpreted as a set of pages/windows/screens that allow a system to interact with a user (e.g., receiving inputs/requests from the user and displaying information/responses to the user). A graphical user interface as broadly recited is not limited to a particular window/screen/page displayed to a user.
Gupta et al. disclosed that the system receives a plurality of user inputs including selecting one or more LLMs to be evaluated, and a prompt from a user (see [0045]), wherein the prompt may include the task request of the user and additional context information provided by the user (see [0040]), wherein a different piece of information of the prompt as disclosed can be interpreted as either a system prompt or a user prompt as broadly recited.
Thus, Gupta et al. as modified by Guttridge et al. teaches the feature of receiving, at a graphical user interface, a plurality of user inputs that at least specify one or more candidate language models, a system prompt and a user prompt, as discussed above.
Regarding Applicant’s argument (see Remarks, pages 27-29) with respected to newly added limitations specifying a system prompt that sets a response constraint and is received via a first input text field and a user prompt that includes a question is received via a second input text field, these limitations will be addressed though new ground(s) of rejection in view of Hawes et al. (U.S. Publication No. 2024/0403194).
Claim Objections
Claims 1-30 are objected to because of the following informalities:
Regarding claim 1, the term “answers” in line 16 should be “answer” for being compatible with the subject.
Regarding claim 2, the term “answers” in line 5 should be “answer” for being compatible with the subject.
Regarding claim 12, the term “includes” in line 7 should be “include” for being compatible with the subject.
Regarding claim 13, the phrase “a total amount” in line 6 should be “a total amount of time”.
Regarding claim 22, the term “answers” in line 16 should be “answer” for being compatible with the subject.
Regarding claim 23, the term “answers” in line 5 should be “answer” for being compatible with the subject.
Regarding claim 27, the term “answers” in line 23 should be “answer” for being compatible with the subject.
Regarding claim 28, the term “answers” in line 5 should be “answer” for being compatible with the subject.
Other dependent claims are objected for incorporating the informality of the objected independent claims 1, 22 and 27 upon which they depend correspondingly.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 10, 12-14, 16-18, 20, 22-25 and 27-30 (effective filing date 07/21/2025) are rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (U.S. Publication No. 2025/0124236, Publication date 04/17/2025), in view of Guttridge et al. (U.S. Publication No. 2025/0086096, Publication date 03/13/2025), and further in view of Hawes et al. (U.S. Publication No. 2024/0403194, Publication date 12/05/2024).
As to claim 1, Gupta et al. teaches:
“A computer-program product comprising a non-transitory machine-readable storage medium storing computer instructions that, when executed by one or more processors, perform operations comprising” (see Gupta et al., Abstract and [0079]):
“adding, to a model repository, a plurality of language models that are invocable by a standardized programmatic interface” (see Gupta et al., Fig. 1, [0023]-[0025] wherein a storage of a set of machine learning models (e.g., LLMs) managed by the system (e.g., the model serving system) can be interpreted as a model repository as recited; also see [0045] wherein the inference module receiving a prompt from a user of a client device and providing the prompt to one or more selected LLMs deployed in the model serving system to perform inference can be interpreted as a standardized programmatic interface as recited);
“receiving, at a graphical user interface, a plurality of first user inputs for executing a new prompt experiment, wherein the plurality of first user inputs at least specify” (see Gupta et al., [0045] for receiving user inputs or selections though a user interface (UI) provided by the interface module, e.g., inputting a prompt and selecting one or more LLMs to be evaluated with a prompt, wherein a process for evaluation of one or more LLMs with a prompt can be interpreted as a prompt experiment as recited; also see Fig. 5A for an example of a user interface as a graphical user interface):
“one or more candidate language models of the plurality of language models to use in the new prompt experiment” (see Gupta et al., [0045] for selecting one or more LLMs to be evaluated), and
“a system prompt and a user prompt to provide as input to the one or more candidate language models during the new prompt experiment” (see Gupta et al., [0045] for receiving a prompt from a user, wherein a prompt used to input to the one or more selected LLMs as recited can be interpreted as equivalent to both a system prompt and/or a user prompt as broadly recited; also see [0040] wherein a prompt may include the task request of the user and additional context information provided by the user);
“executing the new prompt experiment with the one or more candidate language models based on the plurality of first user inputs” (see Gupta et al., [0045] and [0063] for performing “LLM evaluation” experiment with the one or more selected LLMs and a prompt specified by a user);
“adding, to the graphical user interface, a new experiment run container object comprising one or more prompt experiment run entries that display model execution data associated with the one or more candidate language models executed during the new prompt experiment” (see Gupta et al., Fig. 5A and [0063] for displaying, in a table UI, the contents of the experiment including input prompts to the selected LLMs, the generated responses from each of the selected LLMs corresponding to each of the input prompts, and the evaluation results of the generated responses, wherein a table UI can be interpreted as a new experiment run container object as recited, and the generated responses and/or input prompts can be interpreted as model execution data as recited, wherein each set of an input prompt and generated responses can be interpreted as a prompt experiment run entry as broadly recited);
“receiving, via the graphical user interface, a plurality of second user inputs that designate a user-preferred prompt experiment run entry within the one or more prompt experiment run entries and that save the new prompt experiment” (see Gupta et al., Fig. 5B and [0076] for receiving user feedback assigned to the generated responses from one or more LLMs and the button “Save feedback” on Fig. 5B; also see [0066] for disclosure of displaying recorded runs of the LLMs, which suggests the function of saving the run/execution/experiment of the LLMs).
Thus, Gupta et al. teaches saving contents of the experiences including feedback data (see Figs. 5A-B, [0038] and [0076] for using the interface provided by the interface module may include interface for accessing experiments and saving feedback data associated with generated responses from one or more LLMs).
However, Gupta et al. does not explicitly teach a feature for storing run/execution entries as structured records in a data structure as equivalent recited as follows:
“inserting, into a version-controlled prompt tracking data structure, one or more structured prompt experiment records corresponding to the user-preferred prompt experiment run entry and a remainder of the one or more prompt experiment run entries”.
Also, in case Gupta et al. does not explicitly teach a model repository and a standardized programmatic interface for invoking language models as equivalently recited as follows:
“adding, to a model repository, a plurality of language models that are invocable by a standardized programmatic interface”.
On the other hand, Guttridge et al. explicitly teaches:
“adding, to a model repository, a plurality of language models that are invocable by a standardized programmatic interface” (see Guttridge et al., Fig. 2 and [0049] for Model Repository for storing/adding a plurality of models including language models, wherein the software application may trigger execution of the model (i.e., invoking the model) from the model repository via submission of a call to an API (application programming interface) of the AI engine, wherein the API as disclosed can be interpreted as a standardized programmatic interface as recited), and
“inserting, into a version-controlled prompt tracking data structure, one or more structured prompt experiment records corresponding to the user-preferred prompt experiment run entry and a remainder of the one or more prompt experiment run entries” (see Guttridge et al., [0052]-[0053] for storing results of execution of a model including the feedback data into a log (i.e., a record) including an identifier of the input (i.e., prompt), and identifier of the output (e.g., response), an identifier of the model used, and feedback from the recipient).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Guttridge et al.’s teaching to Gupta et al.’s system by implementing a feature for providing a model repository with a standardized programmatic interface (e.g., an API) to invoke/execute its stored models and a feature for storing model execution data in records in a data structure (e.g., log/record with a set of attributes/fields). Ordinarily skilled artisan would have been motivated to do so to provide Gupta et al.’s system with an effective way to manage a plurality of AI or language models and manage/evaluate execution data of models with respect to prompts. In addition, both of the references (Gupta et al. and Guttridge et al.) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as, a system for using AI/language models to generate, access and providing data and including features for tracking model execution and user feedback to retrain/improve models. This close relation between both of the references highly suggests an expectation of success when combined.
However, Gupta et al. as modified by Guttridge et al. does not explicitly teach a feature of receiving a system prompt and a user prompt through two different text input fields as recited as follows:
“the system prompt (i) sets a response constraint of the one or more candidate language models and (ii) is received via a first text input field of the graphical user interface, and
the user prompt (iii) includes a question that the one or more language models answers in accordance with the response constraint set by the system prompt and (iv) is received via a second text input field of the graphical user interface, different from the first text input field.”
On the other hand, Hawes et al. explicitly teaches a feature of receiving a system prompt and a user prompt through two different text input fields as recited as follows:
“the system prompt (i) sets a response constraint of the one or more candidate language models and (ii) is received via a first text input field of the graphical user interface” (see Hawes et al., Fig. 3 and [0107] for system prompt field 318 for receiving a system prompt, e.g., a user may enter a system prompt via system prompt field 318 (i.e., first text input field); also see [0138] for example of a user input system prompt (e.g., “do as the user says”), which represents a response constraint as recited; also see Fig. 8 and [0177] for another example of a system prompt informing the artificial intelligence system and LLM what tasks to process, which also indicates a behavior/response constraint of the LLM as recited), and
“the user prompt (iii) includes a question that the one or more language models answers in accordance with the response constraint set by the system prompt and (iv) is received via a second text input field of the graphical user interface, different from the first text input field” (see Hawes et al., Fig. 3 and [0115] for the task input field (i.e., second text input field) through which a user can enter a task prompt (i.e., a user prompt), wherein the task prompt may represent a request from a user for an LLM to perform a task; also see [0005] wherein a task can be data queries (i.e., task prompt can be a question; also see [0036] for a Question Answer (“QA”) LLM).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Hawes et al.’s teaching to Gupta et al.’s system (as modified by Guttridge et al.) by implementing a feature of providing a GUI for receiving user inputs regarding different components (e.g., system prompt, user prompt, etc.) for generating a prompt for an LLM. Ordinarily skilled artisan would have been motivated to do so as suggested by Hawes et al. (see [0103] and [0175]-[0176]) to provide Gupta et al.’s system with an effective way to allow a user to set and/or modify various aspects and/or properties of one or more LLM prompts to debug interactions with the AI system or LLMs. In addition, both of the references (Gupta et al. and Hawes et al.) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as, a system for using AI/language models to generate, access and providing data and including features for tracking model execution and providing feedback to help improving models. This close relation between both of the references highly suggests an expectation of success when combined.
As to claim 2, this claim is rejected based on the same arguments as above to reject claim 1 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the system prompt sets the response constraint and a persona of the one or more candidate language models” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt); also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a response constraint and/or a persona as well-known in the art (see Vasudevan et al. (U.S. Publication No. 2025/0245417), [0055]; and see Bachman et al. (U.S. Publication No. 2026/0018251), [2193]),
“the one or more candidate language models answer the question included in the user prompt in accordance with the response constraint and the persona set by the system prompt” (see Gupta et al., [0044] wherein the LLMs generate the response to the prompt based on the contextual information (e.g., instructions or system prompt) included in the prompt; also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a response constraint and/or a persona as well-known in the art (see Vasudevan et al. (U.S. Publication No. 2025/0245417), [0055]; and see Bachman et al. (U.S. Publication No. 2026/0018251), [2193]),
“the plurality of language models are invocable by the standardized programmatic interface when each of the plurality of language models implement an instance of a score model function” (see Gupta et al., [0045] wherein the inference module receiving a prompt from a user of a client device and providing the prompt to one or more selected LLMs deployed in the model serving system to perform inference can be interpreted as a standardized programmatic interface as recited; also see Guttridge et al., Fig. 2 and [0049] for Model Repository for storing/adding a plurality of models including language models, wherein the software application may trigger execution of the model (i.e., invoking the model) from the model repository via submission of a call to an API (application programming interface) of the AI engine, wherein the API as disclosed can be interpreted as a standardized programmatic interface as recited; it should be noted that each language model inherently includes a scoring mechanism, which can be interpreted as a score model function as broadly recited, the scoring mechanism/function is a core component of how the model works (e.g., how the input is processed to generate output)),
“each instance of the score model function includes a plurality of input arguments, model invocation code, and a plurality of return values” (see Gupta et al., [0025] wherein each model (i.e., each instance of the score model function) must including input arguments/variables for managing input tokens, code for processing the input tokens to generate output tokens (i.e., model invocation code), and output arguments/variable for managing output tokens as disclosed),
“the plurality of input arguments and the plurality of return values are common across each instance of the score model function” (see Gupta et al., [0025] wherein each model (i.e., each instance of the score model function) must including input arguments/variables for managing input tokens, code for processing the input tokens to generate output tokens (i.e., model invocation code), and output arguments/variable for managing output tokens as disclosed, wherein input arguments and output argument for each model/function can be the same across models (e.g., input argument for receiving a string input (e.g., a prompt), output argument for returning a string output (e.g., a response/answer), etc. (see [0045])), and
“the model invocation code is different across each instance of the score model function” (see Gupta et al., [0025] and [0045] wherein each LLMs must have a different code/function for processing the prompt to generate a respective response, wherein code/function for processing the input data can be interpreted as model invocation code as recited).
As to claim 3, this claim is rejected based on the same arguments as above to reject claim 2 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the response constraint set by the system prompt causes the one or more candidate language models to return a response to the question in JavaScript Object Notation format” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt) to tell the LLM on how to return a response (e.g., “Please list the identified keywords along with their positions in the text”; also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a instruction to return the response/output in JSON format as well-known in the art (see Boyd et al. (U.S. Publication No. 2025/0378602), [0176]; and see Richard et al. (U.S. Publication No. 2025/0322175), [0078]),
“the persona set by the system prompt instructs the one or more candidate language modes to act as a system for translating the user prompt into searches for data catalog” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt) to the LLM; also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a instruction to define a persona of the LLM and its task as well-known in the art (see Mishra et al. (U.S. Publication No. 2025/0328525), [0127]; and see Bachman et al. (U.S. Publication No. 2026/0018251), [2193]),
“the plurality of input arguments that are common across each instance of the score model function include:
a system prompt input argument that receives the system prompt,
a user prompt input argument that receives the user prompt, and
a model options input argument that receives one or more model configuration parameters” (see Gupta et al., [0045] wherein each LLM must include one or more input arguments for receiving information from a prompt provided by the inference module as disclosed; also see [0040] wherein a prompt may include the task request of the user (i.e., user prompt) and additional context information (e.g., system prompt or other information used to train the LLMs); also see Hawes et al., [0054], [0106], Fig. 3 and Fig. 8 for assigning variables to different types of data (e.g., LLM outputs, LLM input prompts, etc.), wherein a variable as disclosed can be interpreted as an argument as recited), and
“the plurality of return values that are common across each instance of the score model function include:
a response return value that returns a response generated from the system prompt and the user prompt,
a run time return value that returns a total execution time required to generate the response,
a prompt length return value that returns a total number of input tokens included in the system prompt and the user prompt, and
an output length return value that returns a total number of output tokens included in the response” (see Gupta et al., [0045] wherein each LLM generates/returns a response (i.e., a response return value); also see Hawes et al., [0054], [0106], Fig. 3 and Fig. 8 for assigning variables to different types of data (e.g., LLM outputs, LLM input prompts, etc.), wherein a variable as disclosed can be interpreted as an argument as recited).
As to claim 4, this claim is rejected based on the same arguments as above to reject claim 2 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the response constraint set by the system prompt constrains the one or more candidate language models to using bullet point when a response to the question comprises a list” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt); also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM , wherein the instructions can relate to how to format an output (see [0094])),
“the user prompt includes the question, audio, an image, and a video” (see Gupta et al., [0025] wherein the prompt or response may include text, images, audio, etc.; also see Hawes et al., [0069] wherein user inputs (e.g., user prompt) can be in a variety of different formats),
“the instance of the score model function implemented by a first language model of the plurality of language models corresponds to a first instance of the score model function, and the instance of the score model function implemented by a second language model of the plurality of language models corresponds to a second instance of the score model function” (see Gupta et al., [0045] for one or more selected LLMs deployed in the model serving system to perform inference; also see Guttridge et al., Fig. 2 and [0049] for Model Repository for storing/adding a plurality of models including language models; it should be noted that each language model inherently includes a scoring mechanism, which can be interpreted as a score model function as broadly recited, the scoring mechanism/function is a core component of how the model works (e.g., how the input is processed to generate output)),
“the model invocation code in the first instance of the score model function includes a first set of computer-executable code for invoking an external API endpoint of the first language model with the system prompt and the user prompt” (see Gupta et al., [0045] for invoking/executing each LLM to perform inference (i.e., generating a response to a prompt) through the inference module wherein any request/instruction including a prompt from the inference module to each LLM can be interpreted as a set of computer-executable code as recited, wherein invoking models deployed in the model serving system remote from the data processing service can be interpreted as invoking an external API endpoint as recited; also see Guttridge et al., [0049] for triggering execution of the model (i.e., invoking the model) via submission of a call to an API, wherein each API call can be interpreted as a set of computer-executable code; also see Hawes et al., [0038] wherein the LLMs can be locally hosted, cloud managed, accessed via one or more Application Programming Interfaces (“APIs”) and/or any combination of the foregoing), and
“the model invocation code in the second instance of the score model function includes a second set of computer-executable code for invoking a local on-premise interface of the second language model with the system prompt and the user prompt” (see Gupta et al., [0045] for invoking/executing each LLM to perform inference (i.e., generating a response to a prompt) through the inference module wherein any request/instruction including a prompt from the inference module to each LLM can be interpreted as a set of computer-executable code, wherein invoking models deployed in the model serving system local to the data processing service can be interpreted as invoking a local-premise interface as recited; also see Guttridge et al., [0049] for triggering execution of the model (i.e., invoking the model) via submission of a call to an API, wherein each API call can be interpreted as a set of computer-executable code; also see Hawes et al., [0038] wherein the LLMs can be locally hosted, cloud managed, accessed via one or more Application Programming Interfaces (“APIs”) and/or any combination of the foregoing).
As to claim 10, this claim is rejected based on the same arguments as above to reject claim 1 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the system prompt includes one or more of: model-initializing context, model behavior constraints, and response formatting rules” (see Gupta et al., [0040] and [0045] wherein a prompt including a task request of the user (i.e., the user prompt) and additional context information (i.e., system prompt); also see [0049]-[0058] wherein the contextual information to the evaluation LLM, which guide the evaluation LLM in generating evaluation results, is an example of a system prompt as recited; ); also see Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM), and
“the user prompt includes user-specific input content used to generate a response in accordance with the system prompt” (see Gupta et al., [0040] and [0045] wherein a prompt including a task request of the user (i.e., the user prompt) and additional context information (i.e., system prompt)).
As to claim 12, this claim is rejected based on the same arguments as above to reject claim 1 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“receiving, via the graphical user interface, third user input for executing one or more additional prompt experiments using one or more second sets of candidate language models and one or more second sets of system and user prompts” (see Gupta et al., [0045] and Fig. 5A for receiving through a user interface (e.g., a GUI) a selection of one or more LLMs for being evaluation and a prompt to provide to the one or more selected LLMs, wherein the user selection or input as disclosed can be interpreted as third user input as recited, and each evaluation process with a prompt can be interpreted as a prompt experiment),
“wherein the one or more sets of system and user prompts includes the user prompt used during the new prompt experiment and a different system prompt than the system prompt used during the new prompt experiment” (see Gupta et al., [0045] for receiving a prompt from user; also see Hawes et al., Fig. 8 and [0175]-[0176] for the GUI that includes the prompt modification pane 810 that enables a user to provide user inputs that used by the AIS to generate and/or modify LLM prompts, the prompt modification pane includes a prompt system control to receive a system prompt, and a task prompt control to receive a task prompt (i.e., user prompt)),
“executing the one or more additional prompt experiments in accordance with the third user input” (see Gupta et al., [0045] for executing the one or more selected LLMs with the prompt),
“adding, to the graphical user interface, one or more second experiment run container objects corresponding to the one or more additional prompt experiments” (see Gupta et al., Fig. 5A and [0063] for displaying, in a table UI, the contents of the experiment including input prompts to the selected LLMs, the generated responses from each of the selected LLMs corresponding to each of the input prompts, and the evaluation results of the generated responses, wherein a table UI can be interpreted as a new experiment run container object as recited)
“displaying, via the one or more second experiment run container objects, one or more second prompt experiment run entries comprising second model execution data associated with the one or more second sets of candidate language models” (see Gupta et al., Fig. 5A and [0063] for displaying, in a table UI, the contents of the experiment, wherein corresponding the generated responses and/or input prompts can be interpreted as model execution data as recited, wherein each set of an input prompt and generated responses can be interpreted as a prompt experiment run entry as broadly recited),
“receiving, via the graphical user interface, the plurality of second user inputs that designate the user-preferred prompt experiment run entry and that save the new prompt experiment and the one or more additional prompt experiments” (see Gupta et al., Fig. 5B and [0076] for receiving user feedback assigned to the generated responses from one or more LLMs and the button “Save feedback” on Fig. 5B; also see [0066] for disclosure of displaying recorded runs of the LLMs, which suggests the function of saving the run/execution/experiment of the LLMs), and
“inserting, into the version-controlled prompt tracking data structure, the one or more structured prompt experiment records and one or more second structured prompt experiment records corresponding to the one or more second prompt experiment run entries” (see Guttridge et al., [0052]-[0053] for storing results of execution of a model including the feedback data into a log (i.e., a record) including an identifier of the input (i.e., prompt), and identifier of the output (e.g., response), an identifier of the model used, and feedback from the recipient, wherein each log including a set of attribute data as disclosed can be interpreted as a structured record or a run entry as recited).
As to claim 13, this claim is rejected based on the same arguments as above to reject claim 1 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“wherein a respective prompt experiment run entry of the one or more prompt experiment run entries corresponds to a respective candidate language model of the one or more candidate language models and includes:
a response generated by the respective candidate language model,
a total amount spent by the respective candidate language model to generate the response,
a total number of input tokens included in the system prompt and the user prompt, and
a total number of output tokens included in the response” (see Gupta et al., Fig. 5A wherein result (i.e., run entry) of each LLM including its response to the prompt; also see Guttridge et al., [0053] wherein each result of execution of each model (i.e., run entry) includes an identifier of the input, an identifier of the output (i.e., a response), an identifier of the model, and feedback from the user/recipient; also see Hawes et al., Fig. 4 and Fig. 9 for displaying the amount of tokens used associated with inputs/arguments and output/response; It should be noted that the limitation “a total amount spent” can be interpreted as a number/amount of any resource (e.g., a number of tokens used)).
As to claim 14, this claim is rejected based on the same arguments as above to reject claim 1 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“wherein the respective prompt experiment run entry further includes a selectable option that is selectable to designate the respective prompt experiment run entry as the user-preferred prompt experiment run entry” (see Gupta et al., Fig. 5B and [0076] for providing user feedback to each generated response, wherein each generated response represents each run entry as recited; also see Guttridge et al., [0052]-[0053] for displaying a user interface that enables a user to provide feedback from the output/response provided by the model and storing the feedback data in the log (i.e., run entry)).
As to claim 16, this claim is rejected based on the same arguments as above to reject claim 1 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“in response to receiving the plurality of second user inputs” (see Gupta et al., Fig. 5B and [0076] for receiving user feedback (i.e., second user inputs) for the generated responses):
“creating a callable endpoint based on the user-preferred prompt experiment run entry” (see Gupta et al., Fig. 5B for the button “Save feedback” wherein each recorded feedback associated with generated response of a model can be interpreted as a callable endpoint as recited; also see Guttridge et al., [0052]-[0053] for storing feedback to the results of execution with each log (i.e., run entry), wherein each log can be interpreted as a callable point as recited), and
“inserting, into the version-controlled prompt tracking data structure, the one or more structured prompt experiment records” (see Gupta et al., Fig. 5B for the button “Save feedback”; also see Guttridge et al., [0052]-[0053] for storing feedback to the results of execution with each log (i.e., run entry), wherein each log can be interpreted a structured prompt experiment record as recite).
As to claim 17, this claim is rejected based on the same arguments as above to reject claim 16 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the callable endpoint is used by a respective decisioning node in an automated decisioning flow” (see Gupta et al., Fig. 5B for button “Save feedback”, wherein feedback can be used by system later; also Guttridge et al., [0031] for submitting feedback information to the GenAI model to refine the answers/responses), and
“the automated decisioning flow uses the respective decisioning node and one or more other decisioning nodes to autonomously generate a decision for a set of input variables” (see Guttridge et al., [0031] for submitting feedback information to the GenAI model to refine the answers/responses).
As to claim 18, this claim is rejected based on the same arguments as above to reject claim 16 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“tracking utilization data associated with the callable endpoint” (see Gupta et al., Fig. 5B and [0076] for tracking/counting the feedback associated with generated responses from a LLM), and
“displaying the utilization data in an observability user interface configured to visualize one or more metrics associated with the utilization data” (see Gupta et al., Fig. 5B and [0076] for tracking/counting the feedback associated with generated responses from a LLM).
As to claim 20, this claim is rejected based on the same arguments as above to reject claim 1 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“wherein a respective structured prompt experiment record of the one or more structured prompt experiment records includes” (see Gupta et al., Fig. 5A for displaying content of an experiment (i.e., prompt experiment record); also see Guttridge et al., [0053] wherein each log as disclosed can be interpreted as a prompt experiment record as recited):
“a run identifier value associated with the new prompt experiment” (see Gupta et al., Fig. 5A for a list of Run Name 514),
“a user selection indicator identifying whether the respective structured prompt experiment record corresponds to the user-preferred prompt experiment run entry” (see Gupta et al., Fig. 5B and [0076] for providing feedback to generated responses from the LLMs to identify a generated response that a user likes or dislikes), and
“the model execution data associated with a respective candidate language model of the one or more candidate language models” (see Gupta et al., Fig. 5A and Fig. 5B wherein the generated responses associated with respected LLMs is the model execution data as recited).
As to claim 22, Gupta et al. teaches:
“A computer-implemented method comprising” (see Gupta et al., Abstract and Fig. 1):
“adding, to a model repository, a plurality of language models that are invocable by a standardized programmatic interface” (see Gupta et al., Fig. 1, [0023]-[0025] wherein a storage of a set of machine learning models (e.g., LLMs) managed by the system (e.g., the model serving system) can be interpreted as a model repository as recited; also see [0045] wherein the inference module receiving a prompt from a user of a client device and providing the prompt to one or more selected LLMs deployed in the model serving system to perform inference can be interpreted as a standardized programmatic interface as recited);
“receiving, at a graphical user interface, a plurality of first user inputs for executing a new prompt experiment, wherein the plurality of first user inputs at least specify” (see Gupta et al., [0045] for receiving user inputs or selections though a user interface (UI) provided by the interface module, e.g., inputting a prompt and selecting one or more LLMs to be evaluated with a prompt, wherein a process for evaluation of one or more LLMs with a prompt can be interpreted as a prompt experiment as recited; also see Fig. 5A for an example of a user interface as a graphical user interface):
“one or more candidate language models of the plurality of language models to use in the new prompt experiment” (see Gupta et al., [0045] for selecting one or more LLMs to be evaluated), and
“a system prompt and a user prompt to provide as input to the one or more candidate language models during the new prompt experiment” (see Gupta et al., [0045] for receiving a prompt from a user, wherein a prompt used to input to the one or more selected LLMs as recited can be interpreted as equivalent to both a system prompt and/or a user prompt as broadly recited; also see [0040] wherein a prompt may include the task request of the user and additional context information provided by the user);
“executing the new prompt experiment with the one or more candidate language models based on the plurality of first user inputs” (see Gupta et al., [0045] and [0063] for performing “LLM evaluation” experiment with the one or more selected LLMs and a prompt specified by a user);
“adding, to the graphical user interface, a new experiment run container object comprising one or more prompt experiment run entries that display model execution data associated with the one or more candidate language models executed during the new prompt experiment” (see Gupta et al., Fig. 5A and [0063] for displaying, in a table UI, the contents of the experiment including input prompts to the selected LLMs, the generated responses from each of the selected LLMs corresponding to each of the input prompts, and the evaluation results of the generated responses, wherein a table UI can be interpreted as a new experiment run container object as recited, and the generated responses and/or input prompts can be interpreted as model execution data as recited, wherein each set of an input prompt and generated responses can be interpreted as a prompt experiment run entry as broadly recited);
“receiving, via the graphical user interface, a plurality of second user inputs that designate a user-preferred prompt experiment run entry within the one or more prompt experiment run entries and that save the new prompt experiment” (see Gupta et al., Fig. 5B and [0076] for receiving user feedback assigned to the generated responses from one or more LLMs and the button “Save feedback” on Fig. 5B; also see [0066] for disclosure of displaying recorded runs of the LLMs, which suggests the function of saving the run/execution/experiment of the LLMs).
Thus, Gupta et al. teaches saving contents of the experiences including feedback data (see Figs. 5A-B, [0038] and [0076] for using the interface provided by the interface module may include interface for accessing experiments and saving feedback data associated with generated responses from one or more LLMs).
However, Gupta et al. does not explicitly teach a feature for storing run/execution entries as structured records in a data structure as equivalent recited as follows:
“inserting, into a version-controlled prompt tracking data structure, one or more structured prompt experiment records corresponding to the user-preferred prompt experiment run entry and a remainder of the one or more prompt experiment run entries”.
Also, in case Gupta et al. does not explicitly teach a model repository and a standardized programmatic interface for invoking language models as equivalently recited as follows:
“adding, to a model repository, a plurality of language models that are invocable by a standardized programmatic interface”.
On the other hand, Guttridge et al. explicitly teaches:
“adding, to a model repository, a plurality of language models that are invocable by a standardized programmatic interface” (see Guttridge et al., Fig. 2 and [0049] for Model Repository for storing/adding a plurality of models including language models, wherein the software application may trigger execution of the model (i.e., invoking the model) from the model repository via submission of a call to an API (application programming interface) of the AI engine, wherein the API as disclosed can be interpreted as a standardized programmatic interface as recited), and
“inserting, into a version-controlled prompt tracking data structure, one or more structured prompt experiment records corresponding to the user-preferred prompt experiment run entry and a remainder of the one or more prompt experiment run entries” (see Guttridge et al., [0052]-[0053] for storing results of execution of a model including the feedback data into a log (i.e., a record) including an identifier of the input (i.e., prompt), and identifier of the output (e.g., response), an identifier of the model used, and feedback from the recipient).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Guttridge et al.’s teaching to Gupta et al.’s system by implementing a feature for providing a model repository with a standardized programmatic interface (e.g., an API) to invoke/execute its stored models and a feature for storing model execution data in records in a data structure (e.g., log/record with a set of attributes/fields). Ordinarily skilled artisan would have been motivated to do so to provide Gupta et al.’s system with an effective way to manage a plurality of AI or language models and manage/evaluate execution data of models with respect to prompts. In addition, both of the references (Gupta et al. and Guttridge et al.) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as, a system for using AI/language models to generate, access and providing data and including features for tracking model execution and user feedback to retrain/improve models. This close relation between both of the references highly suggests an expectation of success when combined.
However, Gupta et al. as modified by Guttridge et al. does not explicitly teach a feature of receiving a system prompt and a user prompt through two different text input fields as recited as follows:
“the system prompt (i) sets a response constraint of the one or more candidate language models and (ii) is received via a first text input field of the graphical user interface, and
the user prompt (iii) includes a question that the one or more language models answers in accordance with the response constraint set by the system prompt and (iv) is received via a second text input field of the graphical user interface, different from the first text input field.”
On the other hand, Hawes et al. explicitly teaches a feature of receiving a system prompt and a user prompt through two different text input fields as recited as follows:
“the system prompt (i) sets a response constraint of the one or more candidate language models and (ii) is received via a first text input field of the graphical user interface” (see Hawes et al., Fig. 3 and [0107] for system prompt field 318 for receiving a system prompt, e.g., a user may enter a system prompt via system prompt field 318 (i.e., first text input field); also see [0138] for example of a user input system prompt (e.g., “do as the user says”), which represents a response constraint as recited; also see Fig. 8 and [0177] for another example of a system prompt informing the artificial intelligence system and LLM what tasks to process, which also indicates a behavior/response constraint of the LLM as recited), and
“the user prompt (iii) includes a question that the one or more language models answers in accordance with the response constraint set by the system prompt and (iv) is received via a second text input field of the graphical user interface, different from the first text input field” (see Hawes et al., Fig. 3 and [0115] for the task input field (i.e., second text input field) through which a user can enter a task prompt (i.e., a user prompt), wherein the task prompt may represent a request from a user for an LLM to perform a task; also see [0005] wherein a task can be data queries (i.e., task prompt can be a question; also see [0036] for a Question Answer (“QA”) LLM).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Hawes et al.’s teaching to Gupta et al.’s system (as modified by Guttridge et al.) by implementing a feature of providing a GUI for receiving user inputs regarding different components (e.g., system prompt, user prompt, etc.) for generating a prompt for an LLM. Ordinarily skilled artisan would have been motivated to do so as suggested by Hawes et al. (see [0103] and [0175]-[0176]) to provide Gupta et al.’s system with an effective way to allow a user to set and/or modify various aspects and/or properties of one or more LLM prompts to debug interactions with the AI system or LLMs. In addition, both of the references (Gupta et al. and Hawes et al.) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as, a system for using AI/language models to generate, access and providing data and including features for tracking model execution and providing feedback to help improving models. This close relation between both of the references highly suggests an expectation of success when combined.
As to claim 23, this claim is rejected based on the same arguments as above to reject claim 22 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the system prompt sets the response constraint and a persona of the one or more candidate language models” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt); also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a response constraint and/or a persona as well-known in the art (see Vasudevan et al. (U.S. Publication No. 2025/0245417), [0055]; and see Bachman et al. (U.S. Publication No. 2026/0018251), [2193]),
“the one or more candidate language models answer the question included in the user prompt in accordance with the response constraint and the persona set by the system prompt” (see Gupta et al., [0044] wherein the LLMs generate the response to the prompt based on the contextual information (e.g., instructions or system prompt) included in the prompt; also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a response constraint and/or a persona as well-known in the art (see Vasudevan et al. (U.S. Publication No. 2025/0245417), [0055]; and see Bachman et al. (U.S. Publication No. 2026/0018251), [2193]),
“the plurality of language models are invocable by the standardized programmatic interface when each of the plurality of language models implement an instance of a score model function” (see Gupta et al., [0045] wherein the inference module receiving a prompt from a user of a client device and providing the prompt to one or more selected LLMs deployed in the model serving system to perform inference can be interpreted as a standardized programmatic interface as recited; also see Guttridge et al., Fig. 2 and [0049] for Model Repository for storing/adding a plurality of models including language models, wherein the software application may trigger execution of the model (i.e., invoking the model) from the model repository via submission of a call to an API (application programming interface) of the AI engine, wherein the API as disclosed can be interpreted as a standardized programmatic interface as recited; it should be noted that each language model inherently includes a scoring mechanism, which can be interpreted as a score model function as broadly recited, the scoring mechanism/function is a core component of how the model works (e.g., how the input is processed to generate output)),
“each instance of the score model function includes a plurality of input arguments, model invocation code, and a plurality of return values” (see Gupta et al., [0025] wherein each model (i.e., each instance of the score model function) must including input arguments/variables for managing input tokens, code for processing the input tokens to generate output tokens (i.e., model invocation code), and output arguments/variable for managing output tokens as disclosed),
“the plurality of input arguments and the plurality of return values are common across each instance of the score model function” (see Gupta et al., [0025] wherein each model (i.e., each instance of the score model function) must including input arguments/variables for managing input tokens, code for processing the input tokens to generate output tokens (i.e., model invocation code), and output arguments/variable for managing output tokens as disclosed, wherein input arguments and output argument for each model/function can be the same across models (e.g., input argument for receiving a string input (e.g., a prompt), output argument for returning a string output (e.g., a response/answer), etc. (see [0045])), and
“the model invocation code is different across each instance of the score model function” (see Gupta et al., [0025] and [0045] wherein each LLMs must have a different code/function for processing the prompt to generate a respective response, wherein code/function for processing the input data can be interpreted as model invocation code as recited).
As to claim 24, this claim is rejected based on the same arguments as above to reject claim 23 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the response constraint set by the system prompt causes the one or more candidate language models to return a response to the question in JavaScript Object Notation format” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt) to tell the LLM on how to return a response (e.g., “Please list the identified keywords along with their positions in the text”; also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a instruction to return the response/output in JSON format as well-known in the art (see Boyd et al. (U.S. Publication No. 2025/0378602), [0176]; and see Richard et al. (U.S. Publication No. 2025/0322175), [0078]),
“the persona set by the system prompt instructs the one or more candidate language modes to act as a system for translating the user prompt into searches for data catalog” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt) to the LLM; also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a instruction to define a persona of the LLM and its task as well-known in the art (see Mishra et al. (U.S. Publication No. 2025/0328525), [0127]; and see Bachman et al. (U.S. Publication No. 2026/0018251), [2193]),
“the plurality of input arguments that are common across each instance of the score model function include:
a system prompt input argument that receives the system prompt,
a user prompt input argument that receives the user prompt, and
a model options input argument that receives one or more model configuration parameters” (see Gupta et al., [0045] wherein each LLM must include one or more input arguments for receiving information from a prompt provided by the inference module as disclosed; also see [0040] wherein a prompt may include the task request of the user (i.e., user prompt) and additional context information (e.g., system prompt or other information used to train the LLMs); also see Hawes et al., [0054], [0106], Fig. 3 and Fig 8 for assigning variables to different types of data (e.g., LLM outputs, LLM input prompts, etc.), wherein a variable as disclosed can be interpreted as an argument as recited), and
“the plurality of return values that are common across each instance of the score model function include:
a response return value that returns the response generated from the system prompt and the user prompt,
a run time return value that returns a total execution time required to generate the response,
a prompt length return value that returns a total number of input tokens included in the system prompt and the user prompt, and
an output length return value that returns a total number of output tokens included in the response” (see Gupta et al., [0045] wherein each LLM generates/returns a response (i.e., a response return value); also see Hawes et al., [0054], [0106], Fig. 3 and Fig. 8 for assigning variables to different types of data (e.g., LLM outputs, LLM input prompts, etc.), wherein a variable as disclosed can be interpreted as an argument as recited).
As to claim 25, this claim is rejected based on the same arguments as above to reject claim 23 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the response constraint set by the system prompt constrains the one or more candidate language models to using bullet point when a response to the question comprises a list” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt); also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM , wherein the instructions can relate to how to format an output (see [0094])),
“the user prompt includes the question, audio, an image, and a video” (see Gupta et al., [0025] wherein the prompt or response may include text, images, audio, etc.; also see Hawes et al., [0069] wherein user inputs (e.g., user prompt) can be in a variety of different formats),
“the instance of the score model function implemented by a first language model of the plurality of language models corresponds to a first instance of the score model function, and the instance of the score model function implemented by a second language model of the plurality of language models corresponds to a second instance of the score model function” (see Gupta et al., [0045] for one or more selected LLMs deployed in the model serving system to perform inference; also see Guttridge et al., Fig. 2 and [0049] for Model Repository for storing/adding a plurality of models including language models; it should be noted that each language model inherently includes a scoring mechanism, which can be interpreted as a score model function as broadly recited, the scoring mechanism/function is a core component of how the model works (e.g., how the input is processed to generate output)),
“the model invocation code in the first instance of the score model function includes a first set of computer-executable code for invoking the first language model with the system prompt and the user prompt” (see Gupta et al., [0045] for invoking/executing each LLM to perform inference (i.e., generating a response to a prompt) through the inference module wherein any request/instruction including a prompt from the inference module to each LLM can be interpreted as a set of computer-executable code as recited, wherein invoking models deployed in the model serving system remote from the data processing service can be interpreted as invoking an external API endpoint as recited; also see Guttridge et al., [0049] for triggering execution of the model (i.e., invoking the model) via submission of a call to an API, wherein each API call can be interpreted as a set of computer-executable code; also see Hawes et al., [0038] wherein the LLMs can be locally hosted, cloud managed, accessed via one or more Application Programming Interfaces (“APIs”) and/or any combination of the foregoing), and
“the model invocation code in the second instance of the score model function includes a second set of computer-executable code for invoking the second language model with the system prompt and the user prompt” (see Gupta et al., [0045] for invoking/executing each LLM to perform inference (i.e., generating a response to a prompt) through the inference module wherein any request/instruction including a prompt from the inference module to each LLM can be interpreted as a set of computer-executable code, wherein invoking models deployed in the model serving system local to the data processing service can be interpreted as invoking a local-premise interface as recited; also see Guttridge et al., [0049] for triggering execution of the model (i.e., invoking the model) via submission of a call to an API, wherein each API call can be interpreted as a set of computer-executable code; also see Hawes et al., [0038] wherein the LLMs can be locally hosted, cloud managed, accessed via one or more Application Programming Interfaces (“APIs”) and/or any combination of the foregoing).
As to claim 27, Gupta et al. teaches:
“A computer-implemented system comprising” (see Gupta et al., Abstract and Fig. 1):
“one or more processors” (see Gupta et al., Fig. 7 and [0081] for processor system 702);
“a memory” (see Gupta et al., Fig. 7 and [0081] for memory system 704); and
“a computer-readable medium operably coupled to the one or more processors, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the one or more processors, cause a computing device to perform operations comprising” (see Gupta et al., Fig. 7 and [0083] for storage system 716):
“adding, to a model repository, a plurality of language models that are invocable by a standardized programmatic interface” (see Gupta et al., Fig. 1, [0023]-[0025] wherein a storage of a set of machine learning models (e.g., LLMs) managed by the system (e.g., the model serving system) can be interpreted as a model repository as recited; also see [0045] wherein the inference module receiving a prompt from a user of a client device and providing the prompt to one or more selected LLMs deployed in the model serving system to perform inference can be interpreted as a standardized programmatic interface as recited);
“receiving, at a graphical user interface, a plurality of first user inputs for executing a new prompt experiment, wherein the plurality of first user inputs at least specify” (see Gupta et al., [0045] for receiving user inputs or selections though a user interface (UI) provided by the interface module, e.g., inputting a prompt and selecting one or more LLMs to be evaluated with a prompt, wherein a process for evaluation of one or more LLMs with a prompt can be interpreted as a prompt experiment as recited; also see Fig. 5A for an example of a user interface as a graphical user interface):
“one or more candidate language models of the plurality of language models to use in the new prompt experiment” (see Gupta et al., [0045] for selecting one or more LLMs to be evaluated), and
“a system prompt and a user prompt to provide as input to the one or more candidate language models during the new prompt experiment” (see Gupta et al., [0045] for receiving a prompt from a user, wherein a prompt used to input to the one or more selected LLMs as recited can be interpreted as equivalent to both a system prompt and/or a user prompt as broadly recited; also see [0040] wherein a prompt may include the task request of the user and additional context information provided by the user);
“executing the new prompt experiment with the one or more candidate language models based on the plurality of first user inputs” (see Gupta et al., [0045] and [0063] for performing “LLM evaluation” experiment with the one or more selected LLMs and a prompt specified by a user);
“adding, to the graphical user interface, a new experiment run container object comprising one or more prompt experiment run entries that display model execution data associated with the one or more candidate language models executed during the new prompt experiment” (see Gupta et al., Fig. 5A and [0063] for displaying, in a table UI, the contents of the experiment including input prompts to the selected LLMs, the generated responses from each of the selected LLMs corresponding to each of the input prompts, and the evaluation results of the generated responses, wherein a table UI can be interpreted as a new experiment run container object as recited, and the generated responses and/or input prompts can be interpreted as model execution data as recited, wherein each set of an input prompt and generated responses can be interpreted as a prompt experiment run entry as broadly recited);
“receiving, via the graphical user interface, a plurality of second user inputs that designate a user-preferred prompt experiment run entry within the one or more prompt experiment run entries and that save the new prompt experiment” (see Gupta et al., Fig. 5B and [0076] for receiving user feedback assigned to the generated responses from one or more LLMs and the button “Save feedback” on Fig. 5B; also see [0066] for disclosure of displaying recorded runs of the LLMs, which suggests the function of saving the run/execution/experiment of the LLMs).
Thus, Gupta et al. teaches saving contents of the experiences including feedback data (see Figs. 5A-B, [0038] and [0076] for using the interface provided by the interface module may include interface for accessing experiments and saving feedback data associated with generated responses from one or more LLMs).
However, Gupta et al. does not explicitly teach a feature for storing run/execution entries as structured records in a data structure as equivalent recited as follows:
“inserting, into a version-controlled prompt tracking data structure, one or more structured prompt experiment records corresponding to the user-preferred prompt experiment run entry and a remainder of the one or more prompt experiment run entries”.
Also, in case Gupta et al. does not explicitly teach a model repository and a standardized programmatic interface for invoking language models as equivalently recited as follows:
“adding, to a model repository, a plurality of language models that are invocable by a standardized programmatic interface”.
On the other hand, Guttridge et al. explicitly teaches:
“adding, to a model repository, a plurality of language models that are invocable by a standardized programmatic interface” (see Guttridge et al., Fig. 2 and [0049] for Model Repository for storing/adding a plurality of models including language models, wherein the software application may trigger execution of the model (i.e., invoking the model) from the model repository via submission of a call to an API (application programming interface) of the AI engine, wherein the API as disclosed can be interpreted as a standardized programmatic interface as recited), and
“inserting, into a version-controlled prompt tracking data structure, one or more structured prompt experiment records corresponding to the user-preferred prompt experiment run entry and a remainder of the one or more prompt experiment run entries” (see Guttridge et al., [0052]-[0053] for storing results of execution of a model including the feedback data into a log (i.e., a record) including an identifier of the input (i.e., prompt), and identifier of the output (e.g., response), an identifier of the model used, and feedback from the recipient).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Guttridge et al.’s teaching to Gupta et al.’s system by implementing a feature for providing a model repository with a standardized programmatic interface (e.g., an API) to invoke/execute its stored models and a feature for storing model execution data in records in a data structure (e.g., log/record with a set of attributes/fields). Ordinarily skilled artisan would have been motivated to do so to provide Gupta et al.’s system with an effective way to manage a plurality of AI or language models and manage/evaluate execution data of models with respect to prompts. In addition, both of the references (Gupta et al. and Guttridge et al.) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as, a system for using AI/language models to generate, access and providing data and including features for tracking model execution and user feedback to retrain/improve models. This close relation between both of the references highly suggests an expectation of success when combined.
However, Gupta et al. as modified by Guttridge et al. does not explicitly teach a feature of receiving a system prompt and a user prompt through two different text input fields as recited as follows:
“the system prompt (i) sets a response constraint of the one or more candidate language models and (ii) is received via a first text input field of the graphical user interface, and
the user prompt (iii) includes a question that the one or more language models answers in accordance with the response constraint set by the system prompt and (iv) is received via a second text input field of the graphical user interface, different from the first text input field.”
On the other hand, Hawes et al. explicitly teaches a feature of receiving a system prompt and a user prompt through two different text input fields as recited as follows:
“the system prompt (i) sets a response constraint of the one or more candidate language models and (ii) is received via a first text input field of the graphical user interface” (see Hawes et al., Fig. 3 and [0107] for system prompt field 318 for receiving a system prompt, e.g., a user may enter a system prompt via system prompt field 318 (i.e., first text input field); also see [0138] for example of a user input system prompt (e.g., “do as the user says”), which represents a response constraint as recited; also see Fig. 8 and [0177] for another example of a system prompt informing the artificial intelligence system and LLM what tasks to process, which also indicates a behavior/response constraint of the LLM as recited), and
“the user prompt (iii) includes a question that the one or more language models answers in accordance with the response constraint set by the system prompt and (iv) is received via a second text input field of the graphical user interface, different from the first text input field” (see Hawes et al., Fig. 3 and [0115] for the task input field (i.e., second text input field) through which a user can enter a task prompt (i.e., a user prompt), wherein the task prompt may represent a request from a user for an LLM to perform a task; also see [0005] wherein a task can be data queries (i.e., task prompt can be a question; also see [0036] for a Question Answer (“QA”) LLM).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Hawes et al.’s teaching to Gupta et al.’s system (as modified by Guttridge et al.) by implementing a feature of providing a GUI for receiving user inputs regarding different components (e.g., system prompt, user prompt, etc.) for generating a prompt for an LLM. Ordinarily skilled artisan would have been motivated to do so as suggested by Hawes et al. (see [0103] and [0175]-[0176]) to provide Gupta et al.’s system with an effective way to allow a user to set and/or modify various aspects and/or properties of one or more LLM prompts to debug interactions with the AI system or LLMs. In addition, both of the references (Gupta et al. and Hawes et al.) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as, a system for using AI/language models to generate, access and providing data and including features for tracking model execution and providing feedback to help improving models. This close relation between both of the references highly suggests an expectation of success when combined.
As to claim 28, this claim is rejected based on the same arguments as above to reject claim 27 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the system prompt sets the response constraint and a persona of the one or more candidate language models” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt); also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a response constraint and/or a persona as well-known in the art (see Vasudevan et al. (U.S. Publication No. 2025/0245417), [0055]; and see Bachman et al. (U.S. Publication No. 2026/0018251), [2193]),
“the one or more candidate language models answer the question included in the user prompt in accordance with the response constraint and the persona set by the system prompt” (see Gupta et al., [0044] wherein the LLMs generate the response to the prompt based on the contextual information (e.g., instructions or system prompt) included in the prompt; also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a response constraint and/or a persona as well-known in the art (see Vasudevan et al. (U.S. Publication No. 2025/0245417), [0055]; and see Bachman et al. (U.S. Publication No. 2026/0018251), [2193]),
“the plurality of language models are invocable by the standardized programmatic interface when each of the plurality of language models implement an instance of a score model function” (see Gupta et al., [0045] wherein the inference module receiving a prompt from a user of a client device and providing the prompt to one or more selected LLMs deployed in the model serving system to perform inference can be interpreted as a standardized programmatic interface as recited; also see Guttridge et al., Fig. 2 and [0049] for Model Repository for storing/adding a plurality of models including language models, wherein the software application may trigger execution of the model (i.e., invoking the model) from the model repository via submission of a call to an API (application programming interface) of the AI engine, wherein the API as disclosed can be interpreted as a standardized programmatic interface as recited; it should be noted that each language model inherently includes a scoring mechanism, which can be interpreted as a score model function as broadly recited, the scoring mechanism/function is a core component of how the model works (e.g., how the input is processed to generate output)),
“each instance of the score model function includes a plurality of input arguments, model invocation code, and a plurality of return values” (see Gupta et al., [0025] wherein each model (i.e., each instance of the score model function) must including input arguments/variables for managing input tokens, code for processing the input tokens to generate output tokens (i.e., model invocation code), and output arguments/variable for managing output tokens as disclosed),
“the plurality of input arguments and the plurality of return values are common across each instance of the score model function” (see Gupta et al., [0025] wherein each model (i.e., each instance of the score model function) must including input arguments/variables for managing input tokens, code for processing the input tokens to generate output tokens (i.e., model invocation code), and output arguments/variable for managing output tokens as disclosed, wherein input arguments and output argument for each model/function can be the same across models (e.g., input argument for receiving a string input (e.g., a prompt), output argument for returning a string output (e.g., a response/answer), etc. (see [0045])), and
“the model invocation code is different across each instance of the score model function” (see Gupta et al., [0025] and [0045] wherein each LLMs must have a different code/function for processing the prompt to generate a respective response, wherein code/function for processing the input data can be interpreted as model invocation code as recited).
As to claim 29, this claim is rejected based on the same arguments as above to reject claim 28 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the response constraint set by the system prompt causes the one or more candidate language models to return a response to the question in JavaScript Object Notation format” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt) to tell the LLM on how to return a response (e.g., “Please list the identified keywords along with their positions in the text”; also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a instruction to return the response/output in JSON format as well-known in the art (see Boyd et al. (U.S. Publication No. 2025/0378602), [0176]; and see Richard et al. (U.S. Publication No. 2025/0322175), [0078]),
“the persona set by the system prompt instructs the one or more candidate language modes to act as a system for translating the user prompt into searches for data catalog” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt) to the LLM; also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a instruction to define a persona of the LLM and its task as well-known in the art (see Mishra et al. (U.S. Publication No. 2025/0328525), [0127]; and see Bachman et al. (U.S. Publication No. 2026/0018251), [2193]),
“the plurality of input arguments that are common across each instance of the score model function include:
a system prompt input argument that receives the system prompt,
a user prompt input argument that receives the user prompt, and
a model options input argument that receives one or more model configuration parameters” (see Gupta et al., [0045] wherein each LLM must include one or more input arguments for receiving information from a prompt provided by the inference module as disclosed; also see [0040] wherein a prompt may include the task request of the user (i.e., user prompt) and additional context information (e.g., system prompt or other information used to train the LLMs); also see Hawes et al., [0054], [0106], Fig. 3 and Fig 8 for assigning variables to different types of data (e.g., LLM outputs, LLM input prompts, etc.), wherein a variable as disclosed can be interpreted as an argument as recited), and
“the plurality of return values that are common across each instance of the score model function include:
a response return value that returns a response generated from the system prompt and the user prompt,
a run time return value that returns a total execution time required to generate the response,
a prompt length return value that returns a total number of input tokens included in the system prompt and the user prompt, and
an output length return value that returns a total number of output tokens included in the response” (see Gupta et al., [0045] wherein each LLM generates/returns a response (i.e., a response return value); also see Hawes et al., [0054], [0106], Fig. 3 and Fig. 8 for assigning variables to different types of data (e.g., LLM outputs, LLM input prompts, etc.), wherein a variable as disclosed can be interpreted as an argument as recited).
As to claim 30, this claim is rejected based on the same arguments as above to reject claim 28 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the response constraint set by the system prompt constrains the one or more candidate language models to using bullet point when a response to the question comprises a list” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt); also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM , wherein the instructions can relate to how to format an output (see [0094])),
“the user prompt includes the question, audio, an image, and a video” (see Gupta et al., [0025] wherein the prompt or response may include text, images, audio, etc.; also see Hawes et al., [0069] wherein user inputs (e.g., user prompt) can be in a variety of different formats),
“the instance of the score model function implemented by a first language model of the plurality of language models corresponds to a first instance of the score model function, and the instance of the score model function implemented by a second language model of the plurality of language models corresponds to a second instance of the score model function” (see Gupta et al., [0045] for one or more selected LLMs deployed in the model serving system to perform inference; also see Guttridge et al., Fig. 2 and [0049] for Model Repository for storing/adding a plurality of models including language models; it should be noted that each language model inherently includes a scoring mechanism, which can be interpreted as a score model function as broadly recited, the scoring mechanism/function is a core component of how the model works (e.g., how the input is processed to generate output)),
“the model invocation code in the first instance of the score model function includes a first set of computer-executable code for invoking an external API endpoint of the first language model with the system prompt and the user prompt” (see Gupta et al., [0045] for invoking/executing each LLM to perform inference (i.e., generating a response to a prompt) through the inference module wherein any request/instruction including a prompt from the inference module to each LLM can be interpreted as a set of computer-executable code as recited, wherein invoking models deployed in the model serving system remote from the data processing service can be interpreted as invoking an external API endpoint as recited; also see Guttridge et al., [0049] for triggering execution of the model (i.e., invoking the model) via submission of a call to an API, wherein each API call can be interpreted as a set of computer-executable code; also see Hawes et al., [0038] wherein the LLMs can be locally hosted, cloud managed, accessed via one or more Application Programming Interfaces (“APIs”) and/or any combination of the foregoing), and
“the model invocation code in the second instance of the score model function includes a second set of computer-executable code for invoking a local on-premise interface of the second language model with the system prompt and the user prompt” (see Gupta et al., [0045] for invoking/executing each LLM to perform inference (i.e., generating a response to a prompt) through the inference module wherein any request/instruction including a prompt from the inference module to each LLM can be interpreted as a set of computer-executable code, wherein invoking models deployed in the model serving system local to the data processing service can be interpreted as invoking a local-premise interface as recited; also see Guttridge et al., [0049] for triggering execution of the model (i.e., invoking the model) via submission of a call to an API, wherein each API call can be interpreted as a set of computer-executable code; also see Hawes et al., [0038] wherein the LLMs can be locally hosted, cloud managed, accessed via one or more Application Programming Interfaces (“APIs”) and/or any combination of the foregoing).
Claims 5-8 and 26 (effective filing date 07/21/2025) are rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (U.S. Publication No. 2025/0124236, Publication date 04/17/2025), in view of Guttridge et al. (U.S. Publication No. 2025/0086096, Publication date 03/13/2025), in view of Hawes et al. (U.S. Publication No. 2024/0403194, Publication date 12/05/2024), and further in view of Muir et al. (U.S. Publication No. 2025/0037014, Publication date 01/30/2025).
As to claims 5 and 26, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches all limitations as recited in claims 1 and 22 respectively.
In addition, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“the response constraint constraints the one or more candidate language models to a target output format when generating a response to the question, and the target output format is JavaScript Object Notation format” (see Gupta et al., [0050]-[0058] for an example of a LLM prompt that includes a request (i.e., user prompt) and other instructions (i.e., system prompt) to tell the LLM on how to return a response (e.g., “Please list the identified keywords along with their positions in the text”; also Hawes et al., Fig. 3, Fig. 8, [0107] and [0177] wherein a user can use the system prompt field/control to enter any instructions (i.e., system prompt) to provide to the LLM, the instructions can include a instruction to format the response/output in JSON format as well-known in the art (see Boyd et al. (U.S. Publication No. 2025/0378602), [0176]; and see Richard et al. (U.S. Publication No. 2025/0322175), [0078]),
“the plurality of language models include a plurality of on-premise language models and a plurality of API-accessible language models” (see Gupta et al., Fig. 1, [0024] and [0045] wherein the data processing service system can access large language models (LLMs) from the model serving system, which can be a part of the data processing service system (i.e., local or on-premise language models) or can be separate from the data processing service system (i.e., API-accessible language models); also see Guttridge et al., Fig. 2 and [0049] for accessing ML or language models though API calls (i.e., API-accessible language models)).
In addition, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches language models (LLMs) associated with model tags/attributes (e.g., identifier/name of a model) (see Gupta et al., [0067] for a name/identifier associated with each LLM (e.g., “hollow-broom-2” or “harsh-carbon-34”) ; also see Guttridge et al., [0049] for an identifier of the model, such as a unique ID assigned by the host platform).
However, Gupta et al. as modified by Guttridge et al. and Hawes et al. does not explicitly teach a feature for association of different sets of language modes with different sets of users based on authorization/privilege/permission levels as equivalently recited as follows:
“one or more first language models of the plurality of language models are associated with a first set of users and a first set of model tags”, and
“one or more second language models of the plurality of language models is associated with a second set of users, different from the first set of users, and a second set of model tags, different from the first set of model tags”.
On the other hand, Muir et al. explicitly teaches a feature for association of different sets of language modes with different sets of users based on authorization/privilege/permission levels (see Muir et al., [0185]-[0188] for providing access to machine learning or language models though a model registry according to groups of users or privilege/permission levels of the users; also see [0193]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Muir et al.’s teaching to Gupta et al.’s system (as modified by Guttridge et al. and Hawes et al.) by implementing a feature for association of different sets of LLMs with different sets of users based on permission/privilege levels associated with the users. Ordinarily skilled artisan would have been motivated to do so to provide Gupta et al.’s system with an effective way to manage a plurality of AI or language models with access control based on permission/authorization levels of the users. In addition, permission/authorization levels associated with users has been well-known and well-used in the art for controlling access to different types of resources/data in the system.
As to claim 6, this claim is rejected based on the same arguments as above to reject claim 5 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al., Hawes et al. and Muir et al. teaches:
“the first set of model tags indicates that the one or more first language models are authorized for use in privacy-sensitive contexts” (see Muir et al., [0185], [0188] and [0193] wherein varying levels of access (i.e., privilege/permission levels) or different visibility levels associated with machine-learning or language models can be interpreted as model tags as recited, wherein users are allowed or not allowed to access/use models based on user permission/privilege levels, wherein levels of visibility may be defined on the sensitive/privacy of models/data; also see Hawes et al., [0011]), and
“the second set of model tags indicates that the one or more second language models are not authorized for use in the privacy-sensitive contexts” (see Muir et al., [0185], [0188] and [0193] wherein varying levels of access (i.e., privilege/permission levels) or different visibility levels associated with machine-learning or language models can be interpreted as model tags as recited, wherein users are allowed or not allowed to access/use models based on user permission/privilege levels; also see Hawes et al., [0011]) .
As to claim 7, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches all limitations as recited in claim 1.
In addition, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“displaying, via the graphical user interface, one or more first language models of the plurality of language models when the graphical user interface is being accessed by a first user” (see Gupta et al., Fig. 5A and [0045] for displaying a set of LLMs through a user interface (UI) for being selected by a user).
However, Gupta et al. as modified by Guttridge et al. and Hawes et al. does not explicitly teach a feature for association of different sets of language modes with different users based on different authorization/privilege/permission levels as equivalently recited as follows:
“displaying, via the graphical user interface, one or more second language models, different from the one or more first language models, of the plurality of language models when the graphical user interface is being accessed by a second user”.
On the other hand, Muir et al. explicitly teaches a feature for association of different sets of language modes with different users based on different authorization/privilege/permission levels (see Muir et al., [0185]-[0188] for providing access to machine learning or language models though a model registry according to privilege/permission levels of the users; also see [0193]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Muir et al.’s teaching to Gupta et al.’s system (as modified by Guttridge et al. and Hawes et al.) by implementing a feature for association of different sets of LLMs with different users based on permission/privilege levels associated with the users. Ordinarily skilled artisan would have been motivated to do so to provide Gupta et al.’s system with an effective way to manage a plurality of AI or language models with access control based on permission/authorization levels of the users. In addition, permission/authorization levels associated with users has been well-known and well-used in the art for controlling access to different types of resources/data in the system.
As to claim 8, this claim is rejected based on the same arguments as above to reject claim 7 and is similarly rejected including the following:
Gupta et al. as modified by Guttridge et al., Hawes et al. and Muir et al. teaches:
“when the graphical user interface is being accessed by the first user, specifying the one or more candidate language models includes receiving, via the graphical user interface, the plurality of first user inputs selecting the one or more candidate language models from the one or more first language models” (see Gupta et al., Fig. 5A and [0045] for displaying a set of LLMs through a user interface (UI) wherein one or more LLMs are selected by a user for being evaluated; also see Muir et al., [0185] and [0193] wherein users are allowed to access to one or more models according to their privilege/permission levels), and
“when the graphical user interface is being accessed by the second user, specifying the one or more candidate language models includes receiving, via the graphical user interface, the plurality of first user inputs selecting the one or more candidate language models from the one or more second language models” (see Gupta et al., Fig. 5A and [0045] for displaying a set of LLMs through a user interface (UI) wherein one or more LLMs are selected by a user for being evaluated; also see Muir et al., [0185] and [0193] wherein users are allowed to access to one or more models according to their privilege/permission levels).
Claim 9 (effective filing date 07/21/2025) is rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (U.S. Publication No. 2025/0124236, Publication date 04/17/2025), in view of Guttridge et al. (U.S. Publication No. 2025/0086096, Publication date 03/13/2025), in view of Hawes et al. (U.S. Publication No. 2024/0403194, Publication date 12/05/2024), and further in view of Sun et al. (U.S. Publication No. 2025/0217716, effectively filed date 02/01/2024).
As to claim 9, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches all limitations as recited in claim 1 including providing a user interface (e.g., GUI) for selecting LLMs and providing a prompt including contextual information (see Gupta et al., Fig. 5A and [0045]).
However, Gupta et al. as modified by Guttridge et al. and Hawes et al. does not explicitly teach a feature for providing a GUI for inputting model configuration parameters for each model as equivalently recited as follows:
“the plurality of first user inputs further specify a plurality of model configuration parameters for the one or more candidate language models, and
specifying the plurality of model configuration parameters for a respective candidate language model of the one or more candidate language models includes:
receiving, via the graphical user interface, a selection of the respective candidate language model in the graphical user interface,
in response to receiving the selection, displaying, via the graphical user interface, a plurality of editable input fields for setting the plurality of model configuration parameters for the respective candidate language model, and
setting, via the plurality of editable input fields, values for the plurality of model configuration parameters based on receiving the plurality of first user inputs”.
On the other hand, Sun et al. explicitly teaches a feature for providing a GUI for inputting model configuration parameters for each model (see Sun et al., [0070] for obtaining model setup parameters (i.e., model configuration parameters) through a GUI that enable a user to specify values for certain model setup parameters by typing or otherwise entering the values into the GUI or selecting the values for the model setup parameters from a list of available options that are presented via the GUI; also see [0071] and [0092]-[0093]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Sun et al.’s teaching to Gupta et al.’s system (as modified by Guttridge et al. and Hawes et al.) by implementing a feature for providing a GUI for enabling a user to input model configuration parameters for a selected model. Ordinarily skilled artisan would have been motivated to do so to provide Gupta et al.’s system with an effective way to configure each of a plurality of AI or language models for execution according to user-defined criteria. In addition, both of the references (Gupta et al. and Sun et al.) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as, a system for using machine-learning models to generate, access and providing data including testing models with inputs/prompts. This close relation between both of the references highly suggests an expectation of success when combined.
Claim 11 (effective filing date 07/21/2025) is rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (U.S. Publication No. 2025/0124236, Publication date 04/17/2025 or effectively filed date 11/22/2023), in view of Guttridge et al. (U.S. Publication No. 2025/0086096, Publication date 03/13/2025 or effectively filed date 09/13/2023), in view of Hawes et al. (U.S. Publication No. 2024/0403194, Publication date 12/04/2024), and further in view of Moradian et al. (U.S. Publication No. 2025/0086342, effectively filed date 09/06/2024).
As to claim 11, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches all limitations as recited in claim 1 including providing a user interface (e.g., GUI) for selecting LLMs and providing a prompt including contextual information (see Gupta et al., Fig. 5A and [0045]; also see Hawes et al., Fig. 3, Fig. 8, [0107] and [0114]-[0115]).
In addition, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“wherein executing the new prompt experiment with a first candidate language model and a second candidate language model of the one or more candidate language models includes in parallel” (see Gupta et al., [0045] for executing the evaluation of the one or more selected LLMs by providing a prompt to the one or more selected LLMs):
“passing the system prompt to a system prompt input argument of a first instance of a score model function implemented by the first candidate language model, the user prompt to a user prompt input argument of the first instance of the score model function implemented by the first candidate language model, and first values for a plurality of model configuration parameters to a model options input arguments of the first instance of the score model function implemented by the first candidate language model” (see Gupta et al., [0045] for providing/passing the prompt to each of the one or more selected LLMs, wherein each LLMs includes a score algorithm/function for processing input into output, wherein the prompt including a task request (i.e., user prompt) and contextual information (i.e., system prompt) (see [0040]); also see Hawes et al., [0054], [0106], Fig. 3 and Fig 8 for assigning variables to different types of data (e.g., LLM outputs, LLM input prompts, etc.), wherein a variable as disclosed can be interpreted as an argument as recited),
“generating, via the first candidate language model, a first response to the user prompt by executing first model invocation code included in the first instance of the score model function” (see Gupta et al., [0040] and [0045] for generating the response to the prompt from each LLM),
“passing the system prompt to the system prompt input argument of a second instance of the score model function implemented by the second candidate language model, the user prompt to the user prompt input argument of the second instance of the score model function implemented by the second candidate language mode, and second values for the plurality of model configuration parameters to the model options input argument of the second instance of the score model function implemented by the second candidate language model” (see Gupta et al., [0045] for providing/passing the prompt to each of the one or more selected LLMs, wherein each LLMs includes a score algorithm/function for processing input into output, wherein the prompt including a task request (i.e., user prompt) and contextual information (i.e., system prompt) (see [0040]); also see Hawes et al., [0054], [0106], Fig. 3 and Fig 8 for assigning variables to different types of data (e.g., LLM outputs, LLM input prompts, etc.), wherein a variable as disclosed can be interpreted as an argument as recited), and
“generating, via the first candidate language model, a second response to the user prompt by executing second model invocation code included in the second instance of the score model function” (see Gupta et al., [0040] and [0045] for generating the response to the prompt from each LLM).
However, Gupta et al. as modified by Guttridge et al. and Hawes et al. does not explicitly teach a feature of including model configuration parameters in the prompt provided to a language model.
On the other hand, Moradian et al. explicitly teaches a feature of including model configuration parameters in the prompt provided to a language model (see Moradian et al., [0038] for receiving user input as a prompt to a large language model (LLM) including model parameters (i.e., model configuration parameters)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Moradian et al.’s teaching to Gupta et al.’s system (as modified by Guttridge et al. and Hawes et al.) by implementing a feature of including model configuration parameters in the prompt provided to a language model. Ordinarily skilled artisan would have been motivated to do so to provide Gupta et al.’s system with an effective way to configure each of a plurality of AI or language models for execution according to user-defined criteria. In addition, both of the references (Gupta et al. and Moradian et al.) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as, a system for using machine-learning models to generate, access and providing data including testing models with inputs/prompts. This close relation between both of the references highly suggests an expectation of success when combined.
Claim 19 (effective filing date 07/21/2025) is rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (U.S. Publication No. 2025/0124236, Publication date 04/17/2025), in view of Guttridge et al. (U.S. Publication No. 2025/0086096, Publication date 03/13/2025), in view of Hawes et al. (U.S. Publication No. 2024/0403194, Publication date 12/05/2024), and further in view of Sikka et al. (U.S. Publication No. 2021/0081841, Publication date 03/18/2021).
As to claim 19, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches all limitations as recited in claim 1 including storing experiment run records/entries in log (see Guttridge et al., [0053] for storing an execution/run record of a model in log).
However, Gupta et al. as modified by Guttridge et al. and Hawes et al. does not explicitly teach a feature of storing records of experiments in storage with version numbers as equivalently recited as follows:
“the version-controlled prompt tracking data structure is associated with a first version number and a plurality of previously added structured prompt experiment records before the new prompt experiment is executed,
the one or more structured prompt experiment records are inserted into the version-controlled prompt tracking data structure in response to receiving the plurality of second user inputs, and
inserting the one or more structured prompt experiment records into the version-controlled prompt tracking data structure includes:
creating a new version of the version-controlled prompt tracking data structure that includes the plurality of previously added structured prompt experiment records and the one or more structured prompt experiment records associated with the new prompt experiment, and
associating the new version of the version-controlled prompt tracking data structure with a second version number, different from the first version number.”.
On the other hand, Sikka et al. teaches a feature of storing records of experiments in storage with version numbers (see Sikka et al., [0070]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Sikka et al.’s teaching to Gupta et al.’s system (as modified by Guttridge et al. and Hawes et al.) by implementing a feature of managing storing experiment data with version numbers (version-controlled data structure). Ordinarily skilled artisan would have been motivated to do so to provide Gupta et al.’s system with an effective way to storing experiment data in a version-control environment. In addition, both of the references (Gupta et al. and Sikka et al.) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as, a system for managing and testing machine-learning models. This close relation between both of the references highly suggests an expectation of success when combined.
Claim 21 (effective filing date 07/21/2025) is rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (U.S. Publication No. 2025/0124236, Publication date 04/17/2025), in view of Guttridge et al. (U.S. Publication No. 2025/0086096, Publication date 03/13/2025), in view of Hawes et al. (U.S. Publication No. 2024/0403194, Publication date 12/05/2024), and further in view of Guo et al. (CN-120277131-A, Publication date 07/08/2025).
As to claim 21, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches all limitations as recited in claim 20.
In addition, Gupta et al. as modified by Guttridge et al. and Hawes et al. teaches:
“wherein the model execution data associated with the respective candidate language model includes:
the system prompt provided to the respective candidate language model,
the user prompt provided to the respective candidate language model,
a name of the respective candidate language model,
one or more model configuration parameters used to configure the respective candidate language model,
a response generated by the respective candidate language model,
a total amount of time required to generate the response,
a number of input tokens provided to the respective candidate language model, and
a number of output tokens included in the response” (see Gupta et al., Fig. 5A and [0067] wherein the experiment or execution data includes the input prompt (i.e., the system prompt and/or the user prompt), a generated response, and a name of LLM model (e.g., “hollow-broom-2” or “harsh-carbon-34”; see Guttridge et al., [0052]-[0053] wherein information associated with output of a model include an identifier of the input, an identifier of the output, identifier of the model used, and feedback from the user/recipient; also see Hawes et al., Fig. 4 and Fig. 9 for displaying the amount of tokens used associated with inputs/arguments and output/response, and other configured tools and/or arguments used by the LLM).
Gupta et al. as modified by Guttridge et al. and Hawes et al. does not explicitly teach a feature of recording an amount of time required to generate the output/response by the model.
On the other hand, Guo et al. teaches feature of recording an amount of time required to generate the output/response by the model (see Guo et al., [page 16, lines 16-23] for generating meta-information of the task (i.e., generating the output result from the target LLM in response to the reference request), the meta-information includes execution time).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Guo et al.’s teaching to Gupta et al.’s system (as modified by Guttridge et al. and Hawes et al.) by implementing a feature of recording an execute time associated with generating an output/response by the LLM. Ordinarily skilled artisan would have been motivated to do so to provide Gupta et al.’s system with an effective way to monitor/track experiment data including execution time. In addition, both of the references (Gupta et al. and Guo et al.) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as, a system for managing and processing/testing machine-learning models. This close relation between both of the references highly suggests an expectation of success when combined.
Allowable Subject Matter
Claim 15 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHUONG THAO CAO whose telephone number is (571)272-2735. The examiner can normally be reached Monday - Friday: 9:00 am - 6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amy Ng can be reached at 571-270-1698. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Phuong Thao Cao/Primary Examiner, Art Unit 2164