DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This Office Action is in response to the communication filed on 28 May 2024.
Claims 1-20 are being considered on the merits.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 10 June 2024 has been considered. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, initialed and dated copies of Applicant's IDS forms 1499 are attached to the instant Office action.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 10-13, and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Huang, et. al. (arXiv:2305.18498v1 [cs.PL] 29 May 2023; hereinafter, “Huang”) in view of Cai, et. al. (US20230112921; hereinafter “Cai”).
Regarding Claim 1:
A computerized method, performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising: (Cai, para. 0110: “The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations”)
receiving, via a user interface, from a user a first user input for a large language model (“LLM”); generating, based on the first user input, a first prompt; transmitting the first prompt to the LLM; receiving an output from the LLM; (Cai, para. 0010 and 0016: “The operations include receiving an initial language input. The operations include processing the initial language input with a model chain to generate a language output. The model chain comprises a plurality of model instantiations of one or more machine-learned language models. Each model instantiation is configured to receive and process a model input to generate a model output, wherein at least a portion of the model input comprises a model prompt descriptive of a task performed by the model instantiation on the model input to generate the model output.” “In some implementations, the one or more machine-learned language models comprise a single machine-learned language model; and the plurality of model instantiations comprise a plurality of model instantiations of the single machine-learned language model.”)
evaluating the output from the LLM with reference to one or more validation tests; (Cai, para. 0070: “First, an LLM may have difficulty applying common sense reasoning or complex inferences to nuanced problems, as previously mentioned. The example Classification operation can help address this by acting as a validation check or triage, before more steps are carried out (Table 1a)”)
responsive to determining that the output from the LLM is not validated, generating a second prompt for the LLM, wherein the second prompt indicates at least an aspect of the output that caused the output to not be validated; transmitting the second prompt to the LLM; and receiving an updated output from the LLM. (Huang, pg. 2, figure 1 and pg. 35: “With ANPL, users have explicit control over the high-level task decomposition using the sketch, leaving the tedious work of function-level holes implementation for LLMs.” “Upon encountering a programming error, the system issues a warning to the user. In response, the user examines the input and output of the find_smallest_unit function and discovers it doesn’t align with expectations. The user then adjusts the natural language description, leading to ANPL successfully meeting the test input and output samples following a code regeneration.” Examiner notes Huang teaches inputting a prompt into an abstracted natural programming language system and using interactive editing a debugging where LLMs are fed data from the dataflow to provide natural language output on errors i.e. “holes”.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Cai into Huang. Huang teaches ANPL on the Abstraction and Reasoning Corpus (ARC), a set of unique tasks that are challenging for state-of the-art AI systems; Cai teaches transparent and controllable human-AI interaction via chaining of machine-learned language models. One of ordinary skill would have been motivated to combine the teachings of Cai into Huang in order to enhance the interpretability, controllability, and performance of language models (Cai, para. 0009).
Regarding Claim 2:
The computerized method of Claim 1, wherein the one or more validation tests include validation tests configured to validate format of information in the output; type of information in the output; and/or business rules associated with the information in the output. (Huang, fig. 1: Examiner notes Huang illustrates validation of a business rule (one or more actions to be taken when certain conditions are met) i.e. color the tallest and shortest columns)
Regarding Claim 3:
The computerized method of Claim 1, further comprising: evaluating the updated output from the LLM with reference to the one or more validation tests; and responsive to determining that the updated output from the LLM is validated, providing the updated output via the user interface. (Huang, fig. 1 and 3: Examiner notes Huang illustrates validation of a business rule (one or more actions to be taken when certain conditions are me) i.e. color the tallest and shortest columns and upon debugging; a user interface for providing output to users is also taught in figure 3)
Regarding Claim 4:
The computerized method of Claim 1, further comprising: automatically generating, based at least in part on the first prompt, the one or more validation tests for validating the output and the updated output from the LLM. (Cai, para. 0070 and Table 1: “The example Classification operation can help address this by acting as a validation check or triage, before more steps are carried out (Table 1a)” Examiner notes Cai teaches a validation test based on the first prompt to ensure correct output, including format and type may be provided).
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Cai into Huang, as modified, as set forth above with respect to claim 1.
Regarding Claim 5:
The computerized method of Claim 1 further comprising: responsive to determining that the updated output from the LLM is not validated, generating a third prompt for the LLM, wherein the third prompt indicates at least an aspect of the updated output that caused the updated output to not be validated. (Huang, pg. 28: “ANPL is designed as an interactive programming system to significantly reduce programming complexity and assure code quality, where the users can iteratively interact with the system to safely ground their minds to code”)
Regarding Claim 6:
The computerized method of Claim 1, wherein the one or more validation tests comprise at least one of a syntax-based rule, a semantic rule, a formality-based rule, a character-based rule, an object-based rule, or a tool-based rule. (Huang, pg. 3: “Synchromesh [47] retrieves few-shot examples from training sets and then checks syntax, scope, typing rules, and contextual logic when synthesizing programs.”)
Regarding Claim 7:
The computerized method of Claim 1, wherein the one or more validation tests comprise a model, and (Cai, para. 0105: “Node view. This view in FIG. 3B allows users to inspect, implement, and test individual nodes. When a node is selected, the panel can change in accordance with the node type. For LLM nodes, the computing system can show a lightweight panel for writing and editing prompts in free-form text.”)
wherein the output and the updated output are transmitted to the model for evaluation. (Huang, pg. 6: “LLMs iterate over and fill the holes one by one in their appearing order. When filling, LLMs can automatically create a name for the hole and decompose it into sub-functions. The main problem is that LLMs cannot implement holes reliably and may generate useless func tions due to their unpredictability. The proposed solution begins with analyzing and clarifying the dependency relationships among the generated functions, and then identifying which function serves as the implementation of the target hole.” Examiner notes Huang teaches an LLM providing the output iteratively (i.e. updated output) to the model for further iteration (i.e. evaluation))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Cai into Huang, as modified, as set forth above with respect to claim 1.
Regarding Claim 8:
The computerized method of Claim 7, wherein the model is one of a language model, an AI model, a generative model, a machine learning (“ML”) model or a neural network (“NN”). (Huang, pg. 3: “We present ANPL, a programming system that allows the user to explicitly manipulate dataflows while offloading the implementation of modules to an LLM, to decompose and ground programming tasks” Examiner notes Huang teaches LLM output to test models).
Regarding Claim 10:
The computerized method of Claim 1, wherein the second prompt identifies that an object type associated with the output from the LLM is invalid, a tool associated with the output from the LLM is not available, or an item associated with the output from the LLM does not exist. (Huang, pg. 35, fig. 31: “Figure 31 serves as an illustrative instance. Initially, the user enters the ANPL code into the system, which consequently produces a function for every hole and automatically verifies the code’s validity. Upon encountering a programming error, the system issues a warning to the user. In response, the user examines the input and output of the find_smallest_unit function and discovers it doesn’t align with expectations.” Examiner notes Huang teaches an a prompt output that an item i.e. find_smallest_unit does not exist and resulted in invalid tests)
Regarding Claim 11:
The computerized method of Claim 1, wherein the second prompt is generated at least based on a template. (Cai, para. 0013: “In some implementations, the respective prompt to each model instantiation in the model chain is user-selectable from a number of pre-defined template prompts that correspond to primitive subtasks.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Cai into Huang, as modified, as set forth above with respect to claim 1.
Regarding Claim 12:
The computerized method of Claim 11, wherein the template (Cai, para. 0013, above) is selected based on a type of the one or more validation tests that caused the output to not be validated. (Cai, para. 0097: “Third, it can be beneficial for operations to handle parsing and restructuring inputs and outputs, as the data layers may take different formats in different steps. For example, in the Ideation step (b2), Alex's three presentation problems are addressed in parallel, creating three paths of model calls. But later in Compose Point (b3), the three sets of problems and suggestions are merged into one. Some examples define required data types per operation, along with corresponding prompting tweaks and parsers. To reflect Ideation which accepts a single string, and outputs a list of ideas, its prompt template can include, e.g., a “1)” right before the output text, to encourage the generation of a list, and the output is also parsed into lists based on such numbering” Examiner notes Cai teaches a template to ensure proper parsing where the template is based on invalid parsing)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Cai into Huang, as modified, as set forth above with respect to claim 1.
Regarding Claim 13:
The computerized method of Claim 11, wherein the template includes an example of valid information associated with the one or more validation tests (Cai, para. 0084: “Such prompts usually includes either the natural language task description (“Given problem, the following is a list of suggestions”), or some demonstrative “few-shot” examples (“Problem: too much text; Suggestion: (1) use more graphics, (2) use bullet points”), or a combination of both.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Cai into Huang, as modified, as set forth above with respect to claim 1.
Regarding Claim 15:
A system for managing one or more models, the system comprising: one or more processors; and a memory that stores computer-executable instructions, wherein the computer-executable instructions, when executed, cause the one or more processors to: (Cai, para. 0110: “The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations”)
receive, via a user interface, from a user a first user input for a large language model (“LLM”); generate, based on the first user input, a first prompt; transmit the first prompt to the LLM; receive an output from the LLM; (Cai, para. 0010 and 0016: “The operations include receiving an initial language input. The operations include processing the initial language input with a model chain to generate a language output. The model chain comprises a plurality of model instantiations of one or more machine-learned language models. Each model instantiation is configured to receive and process a model input to generate a model output, wherein at least a portion of the model input comprises a model prompt descriptive of a task performed by the model instantiation on the model input to generate the model output.” “In some implementations, the one or more machine-learned language models comprise a single machine-learned language model; and the plurality of model instantiations comprise a plurality of model instantiations of the single machine-learned language model.”)
evaluate the output from the LLM with reference to one or more validation tests; (Cai, para. 0070: “First, an LLM may have difficulty applying common sense reasoning or complex inferences to nuanced problems, as previously mentioned. The example Classification operation can help address this by acting as a validation check or triage, before more steps are carried out (Table 1a)”)
responsive to determining that the output from the LLM is not validated, generate a second prompt for the LLM, wherein the second prompt indicates at least an aspect of the output that caused the output to not be validated; transmit the second prompt to the LLM; and receive an updated output from the LLM. (Huang, pg. 2, figure 1 and pg. 35: “With ANPL, users have explicit control over the high-level task decomposition using the sketch, leaving the tedious work of function-level holes implementation for LLMs.” “Upon encountering a programming error, the system issues a warning to the user. In response, the user examines the input and output of the find_smallest_unit function and discovers it doesn’t align with expectations. The user then adjusts the natural language description, leading to ANPL successfully meeting the test input and output samples following a code regeneration.” Examiner notes Huang teaches inputting a prompt into an abstracted natural programming language system and using interactive editing a debugging where LLMs are fed data from the dataflow to provide natural language output on errors i.e. “holes”.)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Cai into Huang, as modified, as set forth above with respect to claim 1.
Regarding Claim 16:
The system of Claim 15, wherein the computer-executable instructions, when executed, further cause the one or more processors to: (Cai, para. 0110, above).
evaluate the updated output from the LLM with reference to the one or more validation tests; and responsive to determining that the updated output from the LLM is validated, provide the updated output via the user interface. (Huang, fig. 1 and 3: Examiner notes Huang illustrates validation of a business rule (one or more actions to be taken when certain conditions are me) i.e. color the tallest and shortest columns and upon debugging; a user interface for providing output to users is also taught in figure 3)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Cai into Huang, as modified, as set forth above with respect to claim 1.
Regarding Claim 17:
The system of Claim 15, wherein the one or more validation tests include validation tests configured to validate format of information in the output; type of information in the output; and/or business rules associated with the information in the output. (Huang, fig. 1: Examiner notes Huang illustrates validation of a business rule (one or more actions to be taken when certain conditions are met) i.e. color the tallest and shortest columns)
Regarding Claim 18:
The system of Claim 15, wherein the computer-executable instructions, when executed, further cause the one or more processors to: automatically generate, based at least in part on the first prompt, the one or more validation tests for validating the output and the updated output from the LLM. (Huang, fig. 1 and 3: Examiner notes Huang illustrates validation of a business rule (one or more actions to be taken when certain conditions are me) i.e. color the tallest and shortest columns and upon debugging; a user interface for providing output to users is also taught in figure 3)
Regarding Claim 19:
One or more non-transitory computer-readable media comprising computer-executable instructions for managing one or more models, wherein the computer-executable instructions, when executed by a computer system, cause the computer system to perform operations comprising: (Cai, para. 0110, above).
receiving, via a user interface, from a user a first user input for a large language model (“LLM”); generating, based on the first user input, a first prompt; transmitting the first prompt to the LLM; receiving an output from the LLM; (Cai, para. 0010 and 0016: “The operations include receiving an initial language input. The operations include processing the initial language input with a model chain to generate a language output. The model chain comprises a plurality of model instantiations of one or more machine-learned language models. Each model instantiation is configured to receive and process a model input to generate a model output, wherein at least a portion of the model input comprises a model prompt descriptive of a task performed by the model instantiation on the model input to generate the model output.” “In some implementations, the one or more machine-learned language models comprise a single machine-learned language model; and the plurality of model instantiations comprise a plurality of model instantiations of the single machine-learned language model.”)
evaluating the output from the LLM with reference to one or more validation tests; (Cai, para. 0070: “First, an LLM may have difficulty applying common sense reasoning or complex inferences to nuanced problems, as previously mentioned. The example Classification operation can help address this by acting as a validation check or triage, before more steps are carried out (Table 1a)”)
responsive to determining that the output from the LLM is not validated, generating a second prompt for the LLM, wherein the second prompt indicates at least an aspect of the output that caused the output to not be validated; transmitting the second prompt to the LLM; and receiving an updated output from the LLM. (Huang, pg. 2, figure 1 and pg. 35: “With ANPL, users have explicit control over the high-level task decomposition using the sketch, leaving the tedious work of function-level holes implementation for LLMs.” “Upon encountering a programming error, the system issues a warning to the user. In response, the user examines the input and output of the find_smallest_unit function and discovers it doesn’t align with expectations. The user then adjusts the natural language description, leading to ANPL successfully meeting the test input and output samples following a code regeneration.” Examiner notes Huang teaches inputting a prompt into an abstracted natural programming language system and using interactive editing a debugging where LLMs are fed data from the dataflow to provide natural language output on errors i.e. “holes”.)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Cai into Huang, as modified, as set forth above with respect to claim 1.
Regarding Claim 20:
The one or more non-transitory computer-readable media of Claim 19, wherein the computer-executable instructions, when executed by the computer system, further cause the computer system to: (Cai, para. 0110, above).
evaluate the updated output from the LLM with reference to the one or more validation tests; and responsive to determining that the updated output from the LLM is validated, provide the updated output via the user interface. (Huang, fig. 1 and 3: Examiner notes Huang illustrates validation of a business rule (one or more actions to be taken when certain conditions are me) i.e. color the tallest and shortest columns and upon debugging; a user interface for providing output to users is also taught in figure 3)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Cai into Huang, as modified, as set forth above with respect to claim 1.
Claims 9 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Huang, et. al. (arXiv:2305.18498v1 [cs.PL] 29 May 2023; hereinafter, “Huang”) in view of Cai, et. al. (US20230112921; hereinafter “Cai”) and further in view of Shethiya, A. S. (Redefining Software Architecture: Challenges and Strategies for Integrating Generative AI and LLMs. 2023. Spectrum of Research, 3(1). Retrieved from http://spectrumofresearch.com/index.php/sr/article/view/22; hereinafter, “Shethiya”)
Regarding Claim 9:
The computerized method of Claim 1, wherein the one or more validation tests are generated further based on a profile or an identity of the user. (Shethiya, pg. 3: “This includes user-specific data (such as past interactions, preferences, or behavioral signals), session-level context (like the task or conversation state), and global knowledge (domain-specific facts or external documents)”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Shethiya into Huang as modified. Shethiya teaches evolving landscape of software engineering in the age of generative intelligence. One of ordinary skill would have been motivated to combine the teachings of Cai into Huang in order to create system designs that address model integration, system reliability, data management, scalability, and ethical governance (Shethiya, pg. 1).
Regarding Claim 14:
The computerized method of Claim 1, further comprising: transmitting ontology data to the LLM, wherein the one or more validation tests or the second prompt refer to the ontology data. (Shethiya, ph. 5: “The first axis of governance is output validation. Because LLMs are generative, they can produce content that seems plausible but is factually incorrect. In domains such as healthcare, law, or finance, this is not just inconvenient—it’s potentially harmful. Systems must include post-generation validation layers. These may range from rule-based filters to semantic validators or secondary AI models trained to evaluate factual correctness. In high-stakes applications, human-in-the-loop (HITL) moderation may be necessary to ensure critical outputs are approved before delivery” Examiner notes Shethiya teaches validation layers i.e. tests for factual correctness)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Shethiya into Huang, as modified, as set forth above with respect to claim 9.
Search Notes
best results from IP.com using application number as main concept with priority date filters and additional keywords including validation and LLM
Prior Art
Lahiri, et. al. (arXiv:2208.05950v1 [cs.SE] 11 Aug 2022) teaches test-driven user-intent formalization (TDUIF), which lever ages lightweight user feedback to jointly (a) formalize the user intent as tests (a partial specification), and (b) generates code that meets the formal user intent.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sally T. Ley whose telephone number is (571)272-3406. The examiner can normally be reached Monday - Thursday, 10:00am - 6:00pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STL/Examiner, Art Unit 2147
/ERIC NILSSON/Primary Examiner, Art Unit 2151
Prosecution Insights