DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending in this office action.
Claims 1 and 10 are amended.
Response to Arguments
Applicant's arguments 07/10/2026 have been fully considered but they are not persuasive.
Double patenting is still maintained, and the rejection is recited in the double patenting heading paragraph [0003] of this office action.
Applicant’s argument:
Thus, for at least all of the above-mentioned reasons, Applicant submits that Van, either alone or in any proper combination with Vaughn, fails to disclose or render obvious every feature of claim 1 as amended.
Examiner’s response:
The issue in the argument is that arts of record fail to disclose the amended limitations. The added limitations consist of information provided with the prompt to the large language model as follows:
Vaughn discloses:
providing, by the at least one processor as the input to the first LLM, a context component, and wherein the context component includes information that relates to a goal that the first LLM is attempting to accomplish:
[0041] In addition to the information on the existing application architecture, which is provided as context for the LLM, the prompt of the user is provided to the LLM as input. This prompt is first obtained from the user-developer. The prompt comprises a textual description of a desired functionality of the additional component.
an instructions component, wherein the instructions component provides semantic grounding to tool use, available parameters and types of parameters to be used by the first LLM:
[0036] “…information on a usage of the component (e.g., a textual description of how the component is used, a reference to other component(s) using the component, parameters for using the component etc.), information on a security of the component (e.g., what security measures are used, such as authentication, cryptographic signing or encryption),”;
and a tools component, wherein the tools component includes tool definitions and instructions that expose commands that are available to the first LLM for generating code:
[0036] “ information on a programming language being used to implement the component, information on one or more programming frameworks being used to implement the component, information on one or more libraries (or frameworks) being used by the component, information on a use of one or more external API by the component (i.e., whether the component accesses an external API, and for which purpose), information on an error handling of the component, information on a scalability of the component, information on a testability of the component,”;
providing, by the at least one processor as the input to the first LLM, in-context learning from a static memory that stores hard-coded and embedded examples to be incorporated into a prompt:
[0042] “…with the information on the existing application architecture being provided as context to the query 1040 from a scope database. In FIG. 11, the respective chunks are then processed separately or together, to determine which existing API(s) is/are to be used and additional API(s) is/are to be created. Once lists of APIs have been created, the corresponding code for the APIs is generated.”:
Furthermore, see figure 8 and 9 for information provided in the prompt to LLM for code generation.
Applicant, representative is encouraged to contact the examiner using information at the bottom of this office action.
Double Patenting
The non-statutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A non-statutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on non-statutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a non-statutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based e-Terminal Disclaimer may be filled out completely online using web-screens. An e-Terminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about e-Terminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are provisionally rejected on the grounds of non-statutory double patenting as being unpatentable over claim 1-16 of co-pending Application No. 18/652,302 in view of Van et al US 20250068547A1 and of Vaughn et al US 20240111498A1.
Mapping of independents claims is as follows, where the corresponding claims have same cues while difference limitations are underlined.
Application:18/675,688
Co-pending application:18/652,302
1. A method for improving a quality of software code, the method being implemented by at least one processor, the method comprising:
receiving, by the at least one processor, a first request for performing a first task;
providing, by the at least one processor as an input to a first large language model (LLM), the first request;
receiving, by the at least one processor from the first LLM, a first set of executable code that is intended to be usable for performing the first task; automatically executing the first set of executable code in an environment
that includes at least one guardrail component that is configured to detect at least one type of error;
detecting at least one error based on a result of the executing;
providing, by the at least one processor as the input to the first LLM, a context component, , and wherein the context component includes information that relates to a goal that the first LLM is attempting to accomplish;
an instructions component wherein the instructions component provides semantic grounding to tool use, available parameters and types of parameters to be used by the first LLM
and a tools component, wherein the tools component includes tool definitions and instructions that expose commands that are available to the first LLM for generating code,
providing, by the at least one processor as the input to the first LLM, in-context learning from a static memory that stores hard-coded and embedded examples to be incorporated into a prompt,
and the in-context learning from a dynamic memory that uses a semantic model to discover previous queries or errors similar to the first request;
receiving, by the at least one processor from the first LLM and in response to the first request, the context component, the instructions component, the tools component and the in- context learning, a first set of executable code that is intended to be usable for performing the first task;
determining at least one feedback item based on the at least one error; and prompting the first LLM to generate a second set of executable code based on the first request, the first set of executable code, and the at least one feedback item.
1. A method for automatically generating software code, the method being implemented by at least one processor, the method comprising:
receiving, by the at least one processor, a first request for performing a first task and a first prompt that includes at least one from among application programming interface (API) information and at least one pre-defined helper function;
providing, by the at least one processor as an input to a first large language model (LLM), the first request and a response to the first prompt that is received via a user interface (UI);
receiving, by the at least one processor from the first LLM, a first set of executable code that implements a first function that corresponds to a skill that is usable for performing the first task;
generating, by the at least one processor, a first test that relates to the first task;
performing the first test by executing the first set of executable code and checking whether the first task has been successfully completed;
Claim 4:
a second SOP tool that corresponds to retrieving a set of executable code that corresponds to a selection from the list of available solutions;
a third SOP tool that corresponds to obtaining a description, a set of instructions, and a set of required input parameters that correspond to the selection selections from the list of available solutions;
calling at least one standard operating procedure (SOP) tool from among a first SOP tool that corresponds to obtaining information that relates to the list of available solutions that are stored in the solutions library;
Independent claim 10
Independent claim 8
Independent claim 19
Independent claim 15
The co-pending application does not explicitly disclose:
providing, by the at least one processor as the input to the first LLM, in-context learning from a static memory that stores hard-coded and embedded examples to be incorporated into a prompt,
and the in-context learning from a dynamic memory that uses a semantic model to discover previous queries or errors similar to the first request;
receiving, by the at least one processor from the first LLM and in response to the first request, the context component, the instructions component, the tools component and the in- context learning, a first set of executable code that is intended to be usable for performing the first task;
determining at least one feedback item based on the at least one error; and prompting the first LLM to generate a second set of executable code based on the first request, the first set of executable code, and the at least one feedback item.
determining at least one feedback item based on the at least one error; and prompting the first LLM to generate a second set of executable code based on the first request, the first set of executable code, and the at least one feedback item.
Van discloses:
and the in-context learning from a dynamic memory that uses a semantic model to discover previous queries or errors similar to the first request:
[0019 “ Additionally, as used herein, “repairing” refers to the process of iteratively attempting to produce improved program code. For example, if an initial synthesized program fails to solve or satisfy the task (e.g., because of compilation errors, runtime errors, or errors in the logic of the program), the system may use these errors to prompt the model to generate an updated program (e.g., to attempt to repair the first generated program).”;
determining at least one feedback item based on the at least one error; and prompting the first LLM to generate a second set of executable code based on the first request, the first set of executable code, and the at least one feedback item:
[0032]“In some aspects, if the generated program fails validation (e.g., the program caused errors to be generated and/or returned inaccurate results), the fine-tuning system 125 may prompt the machine learning model 105 to attempt to repair the code and/or to generate new code. For example, the fine-tuning system 125 may provide feedback (such as the generated error(s)) as input, allowing the machine learning model 105 to generate revised code. As discussed below in more detail, this revised code may comprise the originally generated code with revisions or repair attempts (e.g., modifications to the program) and/or may comprise entirely new code (e.g., the model may attempt to write an entirely new program)”;
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Van into teachings of co-pending application to obtain a prompt of a user for generating code for implementing an additional component for the modular application without bottlenecks inheritance. A user-developer who intends to make a change or addition to the software system may provide a description, such as example output or a mock-up, that can be used by a generative AI agent embedded within a system to iterate through available APIs to produce the desired output. The desired output is code generated that produces the results expected by the user-developer. The user-developer who intends to make a change or addition to the software system does not necessarily need to have any understanding of the underlying architectural structure of the system. [ Vaughn 0053].
But not explicitly:
providing, by the at least one processor as the input to the first LLM, in-context learning from a static memory that stores hard-coded and embedded examples to be incorporated into a prompt,
receiving, by the at least one processor from the first LLM and in response to the first request, the context component, the instructions component, the tools component and the in- context learning, a first set of executable code that is intended to be usable for performing the first task;
Vaughn discloses:
providing, by the at least one processor as the input to the first LLM, in-context learning from a static memory that stores hard-coded and embedded examples to be incorporated into a prompt:
[0042] “…with the information on the existing application architecture being provided as context to the query 1040 from a scope database. In FIG. 11, the respective chunks are then processed separately or together, to determine which existing API(s) is/are to be used and additional API(s) is/are to be created. Once lists of APIs have been created, the corresponding code for the APIs is generated.”:
receiving, by the at least one processor from the first LLM and in response to the first request, the context component, the instructions component, the tools component and the context learning, a first set of executable code that is intended to be usable for performing the first task:
[0039]” The program 210 generally comprises computer program code, generated by the machine learning model 205 based on the prompt (e.g., based on processing the task description 110 and/or unit test(s)) to (attempt to) solve the described programming task. Generally, the program 210 may comprise code in any programming language and using any suitable formatting (e.g., any formatting, depending on the particular language, that can be compiled and executed)”;
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Vaughn into teachings of Van and the application to obtain a prompt of a user for generating code for implementing an additional component for the modular application without bottlenecks inheritance. A user-developer who intends to make a change or addition to the software system may provide a description, such as example output or a mock-up, that can be used by a generative AI agent embedded within a system to iterate through available APIs to produce the desired output. The desired output is code generated that produces the results expected by the user-developer. The user-developer who intends to make a change or addition to the software system does not necessarily need to have any understanding of the underlying architectural structure of the system. [ Vaughn 0053].
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 10-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Van et al US 20250068547A1 in view of Vaughn et al US 20240111498A1.
As per claim 1, Van discloses a method for improving a quality of software code, the method being implemented by at least one processor, the method comprising:
[0062] “In some aspects, the fine-tuning system may evaluate the progress or improvement of the program repair. For example, the fine-tuning system may determine whether the most recently generated program is better than the previous version according to one or more metrics or key performance indicators (e.g., whether a revised version has fewer errors than the original version, solves more unit tests accurately, etc.). In some such aspects, if the program is improved, the fine-tuning system may determine to attempt another round of repair. If the program has not improved (or has worsened), the fine-tuning system may determine not to attempt any further repair.
providing, by the at least one processor as an input to a first large language model (LLM), the first request:
[0057] At block 305, the fine-tuning system accesses a set of task definitions (e.g., the task descriptions 110 of FIGS. 1 and 2). As discussed above, the task descriptions generally describe or indicate programming tasks, such as using natural language to describe the desired functionality of a program”;
and the in-context learning from a dynamic memory that uses a semantic model to discover previous queries or errors similar to the first request:
[0019 “ Additionally, as used herein, “repairing” refers to the process of iteratively attempting to produce improved program code. For example, if an initial synthesized program fails to solve or satisfy the task (e.g., because of compilation errors, runtime errors, or errors in the logic of the program), the system may use these errors to prompt the model to generate an updated program (e.g., to attempt to repair the first generated program).”;
receiving, by the at least one processor from the first LLM, a first set of executable code that is intended to be usable for performing the first task:
[0059] At block 315, the fine-tuning system generates a program (e.g., the program 210 of FIG. 2) using a machine learning model (e.g., the machine learning model 205 of FIG. 2). For example, as discussed above, the fine-tuning system may use the natural language task description as input to the machine learning model, prompting the model to generate a synthesized program that attempts to satisfy the task.
automatically executing the first set of executable code in an environment that includes at least one guardrail component that is configured to detect at least one type of error:
[0031]“In some aspects, the fine-tuning system 125 can process each task description 110 (e.g., natural language text describing a programming task) using the machine learning model 105 to generate a corresponding program. The fine-tuning system 125 may then validate (or attempt to validate) the generated program, such as by determining whether the program compiles successfully (as opposed to generating compilation or interpreter error(s)), whether the program runs successfully (as opposed to generating runtime error(s)), and/or whether the program processes the unit test(s) 115 correctly (e.g., whether the program returns correct results based on the tests)”;
detecting at least one error based on a result of the executing; determining at least one feedback item based on the at least one error; and prompting the first LLM to generate a second set of executable code based on the first request, the first set of executable code, and the at least one feedback item:
[0032]“In some aspects, if the generated program fails validation (e.g., the program caused errors to be generated and/or returned inaccurate results), the fine-tuning system 125 may prompt the machine learning model 105 to attempt to repair the code and/or to generate new code. For example, the fine-tuning system 125 may provide feedback (such as the generated error(s)) as input, allowing the machine learning model 105 to generate revised code. As discussed below in more detail, this revised code may comprise the originally generated code with revisions or repair attempts (e.g., modifications to the program) and/or may comprise entirely new code (e.g., the model may attempt to write an entirely new program)”;
But not explicitly:
receiving, by the at least one processor, a first request for performing a first task:
providing, by the at least one processor as the input to the first LLM, a context component, an instructions component and a tools component, wherein the tools component includes tool definitions and instructions that expose commands that are available to the first LLM for generating code, wherein the instructions component provides semantic grounding to tool use, available parameters and types of parameters to be used by the first LLM, and wherein the context component includes information that relates to a goal that the first LLM is attempting to accomplish;
providing, by the at least one processor as the input to the first LLM, in-context learning from a static memory that stores hard-coded and embedded examples to be incorporated into a prompt; receiving, by the at least one processor from the first LLM and in response to the first request, the context component, the instructions component, the tools component and the in- context learning
Vaughn discloses:
receiving, by the at least one processor, a first request for performing a first task:
[0041] “In addition to the information on the existing application architecture, which is provided as context for the LLM, the prompt of the user is provided to the LLM as input. This prompt is first obtained from the user-developer. The prompt comprises a textual description of a desired functionality of the additional component.
providing, by the at least one processor as the input to the first LLM, a context component, and wherein the context component includes information that relates to a goal that the first LLM is attempting to accomplish:
[0041] In addition to the information on the existing application architecture, which is provided as context for the LLM, the prompt of the user is provided to the LLM as input. This prompt is first obtained from the user-developer. The prompt comprises a textual description of a desired functionality of the additional component.
an instructions component, wherein the instructions component provides semantic grounding to tool use, available parameters and types of parameters to be used by the first LLM:
[0036] “…information on a usage of the component (e.g., a textual description of how the component is used, a reference to other component(s) using the component, parameters for using the component etc.), information on a security of the component (e.g., what security measures are used, such as authentication, cryptographic signing or encryption),”;
and a tools component, wherein the tools component includes tool definitions and instructions that expose commands that are available to the first LLM for generating code:
[0036] “ information on a programming language being used to implement the component, information on one or more programming frameworks being used to implement the component, information on one or more libraries (or frameworks) being used by the component, information on a use of one or more external API by the component (i.e., whether the component accesses an external API, and for which purpose), information on an error handling of the component, information on a scalability of the component, information on a testability of the component,”;
providing, by the at least one processor as the input to the first LLM, in-context learning from a static memory that stores hard-coded and embedded examples to be incorporated into a prompt:
[0042] “…with the information on the existing application architecture being provided as context to the query 1040 from a scope database. In FIG. 11, the respective chunks are then processed separately or together, to determine which existing API(s) is/are to be used and additional API(s) is/are to be created. Once lists of APIs have been created, the corresponding code for the APIs is generated.”:
receiving, by the at least one processor from the first LLM and in response to the first request, the context component, the instructions component, the tools component and the context learning, a first set of executable code that is intended to be usable for performing the first task:
[0039]” The program 210 generally comprises computer program code, generated by the machine learning model 205 based on the prompt (e.g., based on processing the task description 110 and/or unit test(s)) to (attempt to) solve the described programming task. Generally, the program 210 may comprise code in any programming language and using any suitable formatting (e.g., any formatting, depending on the particular language, that can be compiled and executed)”;
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Vaughn into teachings of Van to obtain a prompt of a user for generating code for implementing an additional component for the modular application without bottlenecks inheritance. A user-developer who intends to make a change or addition to the software system may provide a description, such as example output or a mock-up, that can be used by a generative AI agent embedded within a system to iterate through available APIs to produce the desired output. The desired output is code generated that produces the results expected by the user-developer. The user-developer who intends to make a change or addition to the software system does not necessarily need to have any understanding of the underlying architectural structure of the system. [ Vaughn 0053].
As per claim 2, the rejection of claim 1 is incorporated and furthermore Van discloses:
wherein the at least one error includes at least one from among a hallucination error, an application programming interface (API) type error, an execution error, a runtime error, a syntax error, and a return none error that relates to a failure to generate a return to a call:
[0060] “That is, the fine-tuning system may determine whether the generated program satisfies or performs the desired programming task. In some aspects, as discussed above, the fine-tuning system may determine whether the task was satisfied based on determining whether the generated program generates or results in any errors (e.g., compilation or interpreter errors, runtime errors, and the like)”;
As per claim 3, the rejection of claim 1 is incorporated and furthermore Van discloses:
wherein the hallucination error includes at least one from among a first hallucination error that relates to a tool that is usable for performing the first task, a second hallucination error that relates to a parameter that is usable for performing the first task, a third hallucination error that relates to a context of the first task, and a fourth hallucination error that relates to a semantic meaning of a variable that is usable for performing the first task.
[0019]”Additionally, as used herein, “repairing” refers to the process of iteratively attempting to produce improved program code. For example, if an initial synthesized program fails to solve or satisfy the task (e.g., because of compilation errors, runtime errors, or errors in the logic of the program)”;
As per claim 4, the rejection of claim 1 is incorporated and furthermore Van discloses:
wherein the at least one feedback item includes an identification of a specific line of code from within the first set of executable code that is indicated as causing the at least one error.
[0042]“This may be referred to as “full feedback” in some implementations. In some aspects, if available, the training component 220 may additionally provide suggestions or more detailed input. For example, if the interpreter and/or compiler are configured to provide more detailed suggestions (e.g., indicating the potential source of the error or suggesting fixes), the verification component 215 may include this detail in the feedback. Examiner interpretation: source of error within a program include line/lines/statement(s)/blocks of code(US20240256423A1.
As per claim 5, the rejection of claim 1 is incorporated and furthermore Van discloses:
providing, as an input for training the first LLM, a third set of executable code and information indicating that the third set of executable code is effective for performing a task that corresponds to the third set of executable code.
[0043]”In some aspects, the training component 220 may take a variety of actions depending on the feedback provided by the verification component 215. For example, in the illustrated workflow 200, if the training component 220 determines that the program 210 successfully solved the programming task (also referred to as satisfying the task) (e.g., because no errors were generated, and the unit tests 115 were accurately processed), the training component 220 may add the program 210 as a fine-tuning exemplar in a fine-tuning dataset 235. In some aspects, to form the fine-tuning exemplar, the training component 220 may use the task description 110 (and, in some aspects, any unit tests 115 that were included in the prompt to the machine learning model 205) as the input portion of the exemplar, and the training component 220 may use the generated program 210 as the target output. That is, rather than using a ground-truth solution (e.g., the solution 120 of FIG. 1) as the label, the training component 220 can use the output of the machine learning model 205 itself..
As per claim 6, the rejection of claim 5 is incorporated and furthermore Van discloses:
providing, as an additional input for training the first LLM, a fourth set of executable code and information indicating that the fourth set of executable code is not effective for performing a task that corresponds to the fourth set of executable code:
[0077]“Returning to block 420, if the fine-tuning system determines that the (final) revised program also failed to satisfy the task, the method 400 continues to block 430. At block 430, the fine-tuning system generates a fine-tuning record comprising the concatenated text (generated at block 415) and a ground-truth program for the task. For example, as discussed above, the concatenated text may correspond to the input portion of the record, and the ground-truth program may correspond to the target output. In some aspects, as discussed above, the ground-truth program may be a human-authored solution for the programming task”;
[0068] “At block 350, the fine-tuning system fine-tunes the machine learning model based on the fine-tuning dataset.
As per claim 7, the rejection of claim 1 is incorporated and furthermore Van
discloses:
automatically executing the second set of executable code in the environment that includes the at least one guardrail component:
[0065]” For example, in some aspects, the fine-tuning system may concatenate the original prompt (e.g., the task description, which may include one or more-unit tests), the original (or most recently generated) program (e.g., generated at block 315, and/or at block 340 during a prior repair attempt), and the feedback. This concatenated string may then be used as input to the machine learning model to generate a revised program. The method 300 then returns to block 320 to validate the revised program.”;[0060]“At block 320, the fine-tuning system determines whether the programming task was satisfied by the generated program”;
detecting at least one additional error based on a result of the executing of the second set of executable code
[0060]”In some aspects, the fine-tuning system may additionally or alternatively determine whether the task was satisfied based on determining whether the generated program accurately or successfully performs any (hidden) unit tests that are associated with the task description but that were not provided as input to the model.
[0061] If, at block 320, the fine-tuning system determines that the task was satisfied, the method 300 continues to block 330, discussed in more detail below. If the fine-tuning system determines that the task was not satisfied by the generated program, the method 300 continues to block 325”.
determining at least one additional feedback item based on the at least one additional error
[0064] At block 335, the fine-tuning system generates feedback based on the evaluation of the generated program. As discussed above, in some aspects, the feedback comprises simple feedback, such as noting that the program failed to satisfy the task (e.g., natural language text stating “That is incorrect. Please try again”) without providing additional context or explanation. In some aspects, as discussed above, the feedback may comprise more complex feedback (e.g., full feedback), such as text indicating the particular errors that occurred, text noting that the unit test(s) failed, and the like.
prompting the first LLM to generate a third set of executable code based on the first request, the first set of executable code, the second set of executable code, the at least one feedback item, and the at least one additional feedback item:
[0065] At block 340, the fine-tuning system generates a revised program based on the feedback. For example, the fine-tuning system may process the feedback using the machine learning model to generate a revised program. In some aspects, as discussed above, the fine-tuning system may concatenate the feedback with other data to generate a new prompt for the model. For example, in some aspects, the fine-tuning system may concatenate the original prompt (e.g., the task description, which may include one or more-unit tests), the original (or most recently generated) program (e.g., generated at block 315, and/or at block 340 during a prior repair attempt), and the feedback. This concatenated string may then be used as input to the machine learning model to generate a revised program. The method 300 then returns to block 320 to validate the revised program.
As per claim 8, the rejection of claim 1 is incorporated and furthermore Van discloses:
providing the second set of executable code as an input to a second LLM; and receiving, from the second LLM, information that relates to an evaluation of a suitability of the second set of executable code with respect to performing the first task:
[0069] In the illustrated example, the method 300 terminates after block 350. In some aspects, as discussed above, the fine-tuning system may perform the fine-tuning operation (e.g., perform the method 300) repeatedly. For example, the fine-tuning system may use the updated version of the model (generated at block 350) as the (new) base model and perform the method 300 again to generate another updated version. This fine-tuning can continue for any number of iterations, such as until one or more termination criteria are met (e.g., a maximum number of iterations, a desired model accuracy, and the like).
[0064]”At block 335, the fine-tuning system generates feedback based on the evaluation of the generated program. As discussed above, in some aspects, the feedback comprises simple feedback, such as noting that the program failed to satisfy the task (e.g., natural language text stating “That is incorrect. Please try again”) without providing additional context or explanation. In some aspects, as discussed above, the feedback may comprise more complex feedback (e.g., full feedback), such as text indicating the particular errors that occurred, text noting that the unit test(s) failed, and the like.
Claims 10, 11, 12, 13, 14, 15, 16, 17 are the system claims corresponding to method claims 1, 2, 3, 4, 5, 6, 7, 8 and rejected under the same rational set forth in connection with the rejection of claims 1, 2, 3, 4, 5, 6, 7, 8 above.
Claims 19, 20 are the non-transitory computer readable storage medium claims corresponding to method claims 1, 2 and rejected under the same rational set forth in connection with the rejection of claims 1, 2 above.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Van et al US 20250068547A1 in view of Vaughn et al US20240111498A1 and Brown et al US 20240281600A1.
As per claim 9, the rejection of claim 8 is incorporated and furthermore Van discloses:
an indication of an optimality of the second set of executable code:
[0108] “As another example, the verification component 624B and/or the verification circuit 627 may determine whether the generated program operates accurately or correctly, such as whether the program generates a correct output based on a given input (e.g., using unit tests)”;
[0062] “For example, the fine-tuning system may determine whether the most recently generated program is better than the previous version according to one or more metrics or key performance indicators (e.g., whether a revised version has fewer errors than the original version, solves more unit tests accurately, etc.). In some such aspects, if the program is improved, the fine-tuning system may determine to attempt another round of repair. If the program has not improved (or has worsened), the fine-tuning system may determine not to attempt any further repair.”
But not explicitly:
wherein the information received from the second LLM includes at least one from among information that relates to whether the second set of executable code calls proper application programming interfaces (APIs) for performing the first task:
Brown discloses:
wherein the information received from the second LLM includes at least one from among information that relates to whether the second set of executable code calls proper application programming interfaces (APIs) for performing the first task:
[0220] A determination is made at 3020 whether any bugs have been identified. The system may be adapted to identify a range of possible problems with the generated code, including for example, functional problems and logical problems. A logical problem is where the code will run and execute against the third-party application, but will produce an incorrect result or an unintended result due to the logic of the code not performing in a way that matches the user's goal. A functional problem may occur if the code does not function or execute properly, e.g., where a coding error exists or an API function is incorrectly referenced.
[0222]”At 3024, the procedure is run. Specifically, the generated code is run such that an integration occurs relative to the third-party system. For example, the code may include an API call that is made to the third-party system to perform a desired functionality. A saved set of credentials and/or configuration settings may be applied when calling the third-party system/application. A determination can be made at 3026 whether an error has been raised. If so, the error content 3028 may be fed back to the LLM to make another iterative attempt to fix the error. Examples of information that may be fed back include, for example, any actual error message that are received, variable values, expected results, and/or trace information that may be produced (e.g., Python trace data).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Brown into teachings of Van and Vaughn to optimize some statements, or reorder statements for computational efficiency using an improved approach to implement code generation using LLMs [Brown 0229]..
Claim 18 is the system claim corresponding to method claim 9 and rejected under the same rational set forth in connection with the rejection of claim 9 above.
Pertinent arts:
US 20250258657 A1:
a computer implemented source code generation system can be provided comprising: a computer configured to include code generation process that automatically generates modified versions of precomposed source code blocks by creating one or more prompts to a large language model that in response, generates an output comprising the modified source code blocks, and is further configured to perform automatic tests that validate the source code blocks;
US 20250284268 A1:
generation of control code, which was previously a mainly manual and onerous task, can be automated using just an automated code generator of any form, means for performing code validation, and means for executing the candidate control code in a simulation environment.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRAHIM BOURZIK whose telephone number is (571)270-7155. The examiner can normally be reached Monday-Friday (8-4:30).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wei Y Mui can be reached at 571-270-2738. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRAHIM BOURZIK/ Examiner, Art Unit 2191
/Ted T. Vo/ Primary Examiner, Art Unit 2191
Prosecution Insights