CTNF 18/898,860 CTNF 91564 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. DETAILED ACTION Claim 1-20 are pending of which Claims 1, 8, and 15 are independent. Apparent priority date is 09/27/2024. This action is Non-Final. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claims 1, 3, 4, 6, 8, 10, 11, 13, 15, 17 and 18 are r ejected under 35 U.S.C. 103 as being unpatentable over M ouleeswaran (US PG Pub 20250390711; hereinafter “Mouleeswaran”) in view of Subramanian et al. (US Patent 12,645,729; hereinafter “Subramanian”). A s per claims 1, 8 and 15 , Mouleeswaran discloses: A computer-implemented method, non-transitory computer-readable storage medium coupled to one or more processors (Mouleeswaran; p. 0069-0070 - …one or more machine readable medium(s) may be utilized… ; see also p. 0073 - The computer system includes a processor 601 (possibly including multiple processors, multiple cores, multiple nodes, and/or implementing multi-threading, etc.) ) , and system comprising: a computing device (Mouleeswaran; p. 0069-0070 - A machine readable storage medium may be, for example, but not limited to, a system, apparatus, or device, that employs any one of or combination of electronic, magnetic, optical, electromagnetic, infrared, or semiconductor technology to store program code ); and a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device (Mouleeswaran; p. 0069-0072 - The program code/ instruct ions may also be stored in a machine readable medium that can direct a machine to function in a particular manner, such that the instruct ions stored in the machine readable medium produce an article of manufacture including instruct ions which implement the function/act specified in the flowchart and/or block diagram block or blocks ), cause the computing device to perform operations for engineering of prompt templates for prompting large language models (LLMs) (Mouleeswaran; Abstract – prompt building based on prompt templates), the operations comprising: receiving user input defining configuration parameters for a set of prompt templates (Mouleeswaran; Fig. 1, item 103; p. 0019 - The figure and paragraph citations provide for receiving user textual description inputs defining configuration parameters in the form of rule induction parameters , At stage A, the rule induction prompt builder 101 obtains rule induction parameters (configuration parameters) based on a textual description 103 of a rule or rule-related query . The rule induction prompt builder 101 may receive the textual description 103 from a user interface , read it from a file, receive it from another process, etc. The textual description 103 is a natural language description of a rule or query that may have been authored by a human or generated by generative AI ; see also Fig. 2, item 201 & p. 0028 - a prompt builder obtains a textual description of a rule-related query. The textual description may be input into a user interface that passes the textual description to the prompt builder ); providing a configuration file responsive to the user input (Mouleeswaran; Fig. 2, item 205; p. 0029 - the prompt builder retrieves values for rule induction from a configuration file, the prompt builder retrieves values for rule induction parameters based on the textual description ; see p. 0030 - the prompt builder loads or reads a prompt template from a configuration file ); processing the configuration file to generate a set of prompts by populating at least one placeholder of each prompt template with at least one input parameter of input parameters groups defined in the configuration file; (Mouleeswaran; Fig. 2, item 207; p. 0032 - “arranging” the prompt by populating placeholders in the prompt template with the values of the rule induction parameters that are defined in the configuration file , the prompt builder builds a prompt based on the template, textual description, and retrieved values for rule induction parameters. To build the prompt, the prompt builder arranges elements from the template and the rule induction parameters values ; see also p. 0040-0041 - The term “arranging” or “arranged” is used to be untethered to a specific implementation broadly encompass the different implementations (e.g., copy template elements and parameter values into a blank prompt data structure, copy the template and populate placeholder s in the template with the values of the rule induction parameters , etc.)… the prompt builder arranges an initial task instruction in the prompt to generate a query in the reference programming language (“intermediate query”) based on the textual description… ); transmitting the prompts in the set of prompts to one or more LLMs (Mouleeswaran; Fig. 2, item 209; p. 0033 - the prompt builder submits (transmits) the built prompt to a generative AI model ); receiving, from the one or more LLMs, a set of outputs (Mouleeswaran; p. 0034 - Upon receipt of a response from the generative AI model (represented by the dashed line from block 209 to block 211 ), the prompt builder determines whether the generative AI model output a valid query ); transmitting a set of metric evaluation prompts to at least one LLM, each metric evaluation prompt being provided using an evaluation prompt template and an output; receiving, from the at least one LLM, a set of evaluation results, each evaluation result corresponding to a respective prompt in the set of prompts; (Mouleeswaran; p. 0036-0037 - the prompt builder indicates a syntax error(s) in the query (set of evaluation results) . Implementations can vary as to treatment of an erroneous output from the generative AI model. As examples, the output with the identified error(s) can be presented to a user for review; the output can be preserved for later analysis to gain intelligence for evaluating the prompt and/or generative AI model capabilities; and the output can be discarded and a notification returned in association with the textual description that a satisfactory query could not be acquired ); and responsive to the set of evaluation results, selectively deploying prompt templates in the set of prompt templates for production use in prompting the one or more LLMs (Mouleeswaran; Fig. 2, item 215; p. 0038 – the acquired query (or prompt) is deployed via multiple methods defined in the cited disclosure (e.g. presenting it on a display, writing it to a file, run the query and provide results, etc.). The query may be “selectively” deployed by the user by allowing the user to select to run the query, the prompt builder provides the acquired query. An implementation can present the query in a user interface, write the query to a file, run the query and provide the results in association with the query that was run . To illustrate, the textual description may have been input via a user interface. The generated query in the target programming language is presented in the user interface in relation to the textual description. The user can then select to run the query ). Mouleeswaran, however, fails to disclose generating a set of prompts , transmitting a set of metric evaluation prompts to at least one LLM, each metric evaluation prompt being provided using an evaluation prompt template and an output; and each evaluation result corresponding to a respective prompt in the set of prompts . Although Mouleeswaran discloses generating a prompt and not specifically generating a set of prompts , Subramanian does teach generating a set of prompts (Subramanian; Col. 11, lines 41-46 - An example of the gradient-free 216 function includes a meta-prompt having meta-prompt placeholders for both an initial prompt template to improve and some number of example prompts , and an instruction to generate multiple prompt variants (set of prompts) of the prompt template based on the examples provided ). Additionally, Subramanian teaches transmitting a set of metric evaluation prompts to at least one LLM, each metric evaluation prompt being provided using an evaluation prompt template and an output (Subramanian; Specific examples of evaluation metrics are provided in Col. 11, lines 4—67 and Col. 12, lines 1-24; Col. 12, lines 25-31 – An example of the LM as scorer 234 (at least one LLM) function uses an evaluation prompt template having placeholders for the prompt template under evaluation and its associated output for a given training sample. The evaluation prompt template can include the instruction to evaluate the quality and effectiveness (evaluation metrics) of the prompt given the sample and the associated output on a scale (e.g., from 1-10) ) ; and each evaluation result corresponding to a respective prompt in the set of prompts (Subramanian; Col. 12, lines 25-31 – teaches that an evaluation prompt template is used for the prompt template under evaluation , thus, the teaching of Subramanian provides for using an evaluation prompt template (and thus, also provides each evaluation result) for each corresponding prompt template under evaluation ). Therefore, it would have been obvious to one of ordinary skill in the art to modify the computer-implemented method, non-transitory computer-readable storage medium and system to include generating a set of prompts , transmitting a set of metric evaluation prompts to at least one LLM, each metric evaluation prompt being provided using an evaluation prompt template and an output; and each evaluation result corresponding to a respective prompt in the set of prompts , as taught by Subramanian, in order to enable users to easily optimize their prompt templates. Such an optimizer, also referred to herein as an “orchestrator,” can iteratively generate, evaluate, and select prompt variants to improve an initial prompt template for some threshold number of iterations or until an earlier stopping condition is reached. The optimizer leverages a language model (e.g., an LLM) for one or more of the operations associated with generating, evaluating, and selecting candidate variants (or “candidates”). Through one or more iterations, the optimizer can identify and output an optimized prompt template (Subramanian; Col. 2, lines 47-59). As per claims 3, 10 and 17 , Mouleeswaran in view of Subramanian disclose: The method, non-transitory computer-readable storage medium and system of claims 1, 8 and 15, wherein each input parameter group defines at least one input value to populate placeholders of the prompt templates (Mouleeswaran; p. 0029 - the prompt builder retrieves values for rule induction parameters based on the textual description . This is a multi-step retrieval that begins with determining a name of an API relevant to the textual description and then obtaining other parameters based on the determined API name. For example, the prompt builder generates an embedding(s) from the textual description and then accesses an embeddings database populated with embeddings of API names to determine which API name embedding is most similar and/or relevant to the embedding(s) of the textual description (retrieval of an input parameter group ) . Embodiments can generate a single embedding from the textual description for accessing the embeddings database… Other non-exhaustive examples provided for retrieving input parameter groups provided in the citation) . As per claims 4, 11 and 18 , Mouleeswaran in view of Subramanian disclose: The method, non-transitory computer-readable storage medium and system of claims 1, 8 and 15, wherein the configuration file defines a set of metrics for evaluation of outputs of the one or more LLMs (Subramanian; Exemplary configuration data 152 can include… locations of input data such as training and/or evaluation data … The job configuration data 152 can be provided as environment variables initialized within the environment of compute instance 150 , as a configuration file written to storage accessible to the compute instance 150 , via an API of the orchestrator 156 , etc .) . Therefore, it would have been obvious to one of ordinary skill in the art to modify the computer-implemented method and non-transitory computer-readable storage medium to include wherein the configuration file defines a set of metrics for evaluation of outputs of the one or more LLMs , as taught by Subramanian, in order to enable users to easily optimize their prompt templates. Such an optimizer, also referred to herein as an “orchestrator,” can iteratively generate, evaluate, and select prompt variants to improve an initial prompt template for some threshold number of iterations or until an earlier stopping condition is reached. The optimizer leverages a language model (e.g., an LLM) for one or more of the operations associated with generating, evaluating, and selecting candidate variants (or “candidates”). Through one or more iterations, the optimizer can identify and output an optimized prompt template (Subramanian; Col. 2, lines 47-59). As per claims 6 and 13 , Mouleeswaran in view of Subramanian disclose: The method and non-transitory computer-readable storage medium of claims 1 and 8, upon which claims 6 and 13 depend. And further, Subramanian teaches wherein prompts in the set of prompts comprise one or more of reference-free prompts and reference-based prompts (Subramanian; Col. 9, lines 39-45 - Training/evaluation data 174 can include samples, samples with associated labels or ground -truths (ground-truths correspond to “references” as defined in the specification of the instant application in p. 0042) , or some combination of both. For example, for a prompt template used in classification, the training data can include examples to be classified as well as their corresponding classification label (e.g., 0 or 1 for binary classification tasks; happy, sad, neutral for sentiment analysis tasks, etc.) … in this example the classification labels serve as ground-truths or “references” for a reference-based prompt. One of ordinary skill in the art would find that although reference-based prompts are exemplified by Subramanian, reference-free prompts are also covered because the examples appear to be alternatives , such that the alternative to reference-based prompts would be reference-free prompts , or prompts that do not include the attached ground-truths) . Therefore, it would have been obvious to one of ordinary skill in the art to modify the computer-implemented method and non-transitory computer-readable storage medium to include wherein prompts in the set of prompts comprise one or more of reference-free prompts and reference-based prompts , as taught by Subramanian, in order to enable users to easily optimize their prompt templates. Such an optimizer, also referred to herein as an “orchestrator,” can iteratively generate, evaluate, and select prompt variants to improve an initial prompt template for some threshold number of iterations or until an earlier stopping condition is reached. The optimizer leverages a language model (e.g., an LLM) for one or more of the operations associated with generating, evaluating, and selecting candidate variants (or “candidates”). Through one or more iterations, the optimizer can identify and output an optimized prompt template (Subramanian; Col. 2, lines 47-59) . 07-21-aia AIA Claim s 2, 9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Mouleeswaran in view of Subramanian and further in view of Tichauer (US PG Pub 20160255024) . As per claims 2, 9 and 16 , Mouleeswaran in view of Subramanian disclose: The method, non-transitory computer-readable storage medium and system of claims 1, 8 and 15, upon which claims 2, 9 and 16 depend. Mouleeswaran in view of Subramanian, however, fail to teach wherein processing the configuration file to generate a set of prompts comprises: receiving a message from a messaging queue based on a topic assigned to the message; and retrieving the configuration file from a database using an identifier provided with the message . Tichauer does teach wherein processing the configuration file to generate a set of prompts comprises: receiving a message from a messaging queue based on a topic assigned to the message (Tichauer; p. 0032 - …Message manager module 14 includes a unified communication (UC) message management agent 16 configured to examine and process messages which, in addition to message content intended for one or more recipients, may also contain one or more tags (message from a messaging queue) … a tag may be incorporated into a message in such a way that it identifies a specific topic or subject in which one or more recipients has already expressed interest via a formal or informal “subscription” process ) ; and retrieving the configuration file from a database using an identifier provided with the message (Tichauer; p. 0033 - …the presence of a tag in a message triggers the message management agent 16 to select an appropriate presentation template 26 from data repository 20 (configuration file from a database, where the configuration file contains the prompt templates)…). Therefore, it would have been obvious to one of ordinary skill in the art to modify the computer-implemented method, system and non-transitory computer-readable storage medium to include wherein processing the configuration file to generate a set of prompts comprises: receiving a message from a messaging queue based on a topic assigned to the message; and retrieving the configuration file from a database using an identifier provided with the message , as taught by Tichauer, in order to provide enhanced message content and message delivery for messages that are reformatted according to a particular topic tag associated to the message and/or provide enhanced priority of the message according to the associated topic tag (Tichauer; p. 0005-0007) . 07-21-aia AIA Claim s 5, 12 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Mouleeswaran in view of Subramanian and further in view of Whitenack (US PG Pub 20250371192) . As per claims 5, 12 and 19 , Mouleeswaran in view of Subramanian disclose: The method, non-transitory computer-readable storage medium and system of claims 1, 8 and 15, upon which claims 5, 12 and 19 depend. Mouleeswaran in view of Subramanian, however, fail to disclose wherein the configuration file identifies the one or more LLMs that are to be prompted using the set of prompts and, for each LLM, defines a set of parameters for execution of the LLM, the set of parameters comprising temperature and maximum number of tokens . Whitenack does teach wherein the configuration file identifies the one or more LLMs that are to be prompted using the set of prompts (Whitenack; p. 0096 - The request from the client application 128 may include the identification of one or more models that should specifically be used to generate a response to the text prompt ) and, for each LLM, defines a set of parameters for execution of the LLM, the set of parameters comprising temperature and maximum number of tokens (Whitenack; p. 0097 - In addition to a text prompt the client application 128 may send various optional parameters to the API Application 102 that will be passed through to the model applications 114 , and, in their absence, the API application 102 may use default values for the parameters. These parameters may include temper ature , top-k, top-p, max tokens , and max new tokens ) . Therefore, it would have been obvious to one of ordinary skill in the art to modify the computer-implemented method, system and non-transitory computer-readable storage medium to include wherein the configuration file identifies the one or more LLMs that are to be prompted using the set of prompts and, for each LLM, defines a set of parameters for execution of the LLM, the set of parameters comprising temperature and maximum number of tokens , as taught by Whitenack, in order to provide for improved platforms and technologies for controlled and validated interactions with one or more predictive language models, to detect, avoid and/or mitigate risks to computing systems and user data (Whitenack; p. 0005) . 07-21-aia AIA Claim s 7, 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Mouleeswaran in view of Subramanian and further in view of Dhaliwal (US PG Pub 20260080241; as claiming priority to Provisional application No. 63/695,102 , filed 09/16/2024). As per claims 7, 14 and 20 , Mouleeswaran in view of Subramanian disclose: The method, non-transitory computer-readable storage medium and system of claims 1, 8 and 15, upon which claims 5, 12 and 19 depend. Mouleeswaran in view of Subramanian, however, fail to explicitly teach wherein the configuration file comprises a YAML Ain't Markup Language (YAML) file. Dhaliwal does teach wherein the configuration file comprises a YAML Ain't Markup Language (YAML) file (Dhaliwal; p. 0113 - Any applicable data structures, file formats, and schemas in computer system 1300 may be derived from standards including but not limited to JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language ( YAML ) , Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representations alone or in combination ) . Therefore, it would have been obvious to one of ordinary skill in the art to modify the computer-implemented method, system and non-transitory computer-readable storage medium to include wherein the configuration file comprises a YAML Ain't Markup Language (YAML) file , as taught by Dhaliwal, in order to facilitate the automated generation of a field service technician pre-work brief because the use of a pre-work brief can reduce or eliminate a field service technician's need to query on- site individuals, such as homeowners, business premises personnel, or other field workers, to be guided to the place of the work and to understand the nature and scope of the work to be performed (Dhaliwal; p. 0017) . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The prior art made of record and not relied upon includes: Gong (US PG Pub 20260023929) discloses a non-transitory computer-readable media stores instructions executable by processors for generating a prompt configured for eliciting outputs from large language models (LLMs) based on information associated with a task, inputting the prompt to a first LLM configured to output a response based on processing the prompt, determining metrics for evaluating the first LLM based on the task, wherein each of the metrics is associated with a scoring guideline, generating metric prompts based on the respective metrics and the scoring guidelines associated with the respective metrics, inputting the response and the metric prompts to second LLMs configured to output scores corresponding to the respective metrics based on processing the response and the metric prompts, and generating an analysis report based on the metrics and their corresponding scores (Gong; Abstract) . Any inquiry concerning this communication or earlier communications from the examiner should be directed to Rodrigo A Chavez whose telephone number is (571)270-0139. The examiner can normally be reached Monday - Friday 9-6 ET. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached at 5712727602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /RODRIGO A CHAVEZ/Examiner, Art Unit 2658 /FARIBA SIRJANI/Primary Examiner, Art Unit 2659 Application/Control Number: 18/898,860 Page 2 Art Unit: 2658 Application/Control Number: 18/898,860 Page 3 Art Unit: 2658 Application/Control Number: 18/898,860 Page 4 Art Unit: 2658 Application/Control Number: 18/898,860 Page 5 Art Unit: 2658 Application/Control Number: 18/898,860 Page 6 Art Unit: 2658 Application/Control Number: 18/898,860 Page 7 Art Unit: 2658 Application/Control Number: 18/898,860 Page 8 Art Unit: 2658 Application/Control Number: 18/898,860 Page 9 Art Unit: 2658 Application/Control Number: 18/898,860 Page 10 Art Unit: 2658 Application/Control Number: 18/898,860 Page 11 Art Unit: 2658 Application/Control Number: 18/898,860 Page 12 Art Unit: 2658 Application/Control Number: 18/898,860 Page 13 Art Unit: 2658 Application/Control Number: 18/898,860 Page 14 Art Unit: 2658
Prosecution Insights