CTNF 18/499,101 CTNF 98442 DETAILED ACTION This action is responsive to the claims filed on 10/31/2023. Claims 1-20 are pending for examination. Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Information Disclosure Statement 06-52 The information disclosure statement (IDS) submitted on 10/28/2024 was filed after the is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claims 3, 10, and 17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. 07-34-05 AIA Claim s 3, 10, and 17 recites the limitation " the task output " in line 1 . There is insufficient antecedent basis for this limitation in the claim. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Statutory Categories Claims 1-7 are directed to a system. Claims 8-14 are directed to a method. Claims 15-20 are directed to a Machine-Readable Medium. Independent Claims – Claims 1, 8 and 15 Step 2A Prong 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes. Independent claims 1, 8 and 15 recites limitations that are abstract ideas in the form of mental processes: Claim 1 recites : generating… a first task output in response to a task input; ( a mental process of evaluation which can reasonably be performed in human mind or with aid of pen and paper ) and training the student model based on a training input of the first task output and the feedback and a training label of the refinement output. ( this training limitation merely recites or is directed to the use of mathematical concepts in the form of calculation or algorithms, specification paragraphs [0060-0061] recite the related mathematical disclosure (loss functions) used to train the student model ) Claim 1 also recites the following additional elements for the purposes of Step 2A Prong Two analysis: A system for training a neural network model using a teacher-student framework, the system comprising: a communication interface configured to communicate with a teacher model; ( this limitation is merely using a communication interface at a high level of generality such that it is being considered mere instructions to apply an exception, see MPEP 2106.05(f) ) a memory storing a student model and a plurality of processor-executable instructions; and a processor executing the processor-executable instructions to perform operations comprising: ( this limitation recites mere instructions to apply an exception using generic computer components, see MPEP 2106.05(f) ) by the student model, ( this limitation is merely using a student at a high level of generality such that it is being considered mere instructions to apply an exception, see MPEP 2106.05(f) ) obtaining, from an evaluation environment, a feedback relating to an accuracy of the first task output; ( receiving this set of training data and example data is being considered as mere data gathering and is considered insignificant extra-solution activity under MPEP 2106.05(g) ) obtaining a refinement output generated by the teacher model based on an input of the first task output and the feedback; ( receiving this set of training data and example data is being considered as mere data gathering and is considered insignificant extra-solution activity under MPEP 2106.05(g) ) The additional limitations fail step 2A Prong 2 of the 101 analysis because they do not transform the claim into a practical application. These limitations are too abstract or lack technical improvement that would make the concept practically useful. Without clear utility or integration into a specific field, the claim does not relate to any particular application. It does not meet the requirements of Step 2A Prong 2, as it fails to make the concept meaningfully applicable in practice. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea. This claim recites the following additional elements for the purposes of Step 2B analysis: A system for training a neural network model using a teacher-student framework, the system comprising: a communication interface configured to communicate with a teacher model; ( this limitation is merely using a communication interface at a high level of generality such that it is being considered mere instructions to apply an exception, see MPEP 2106.05(f) ) a memory storing a student model and a plurality of processor-executable instructions; and a processor executing the processor-executable instructions to perform operations comprising: ( this limitation recites mere instructions to apply an exception using generic computer components, see MPEP 2106.05(f) ) by the student model, ( this limitation is merely using a student at a high level of generality such that it is being considered mere instructions to apply an exception, see MPEP 2106.05(f) ) obtaining, from an evaluation environment, a feedback relating to an accuracy of the first task output; ( receiving this set of training data and example data is being considered as mere data gathering and is considered insignificant extra-solution activity under MPEP 2106.05(g), for the purposes of step 2B it should be noted that the courts have recognized receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information) as well-understood, routine, and conventional activity. ) obtaining a refinement output generated by the teacher model based on an input of the first task output and the feedback; ( receiving this set of training data and example data is being considered as mere data gathering and is considered insignificant extra-solution activity under MPEP 2106.05(g), for the purposes of step 2B it should be noted that the courts have recognized receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information) as well-understood, routine, and conventional activity. ) Claims 8 and 15 recites limitations substantially identical to claim 1 as such a similar analysis under 101 applies. Claim 15 also recites the following additional limitation for consideration: A non-transitory machine-readable medium comprising a plurality of machine- executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising ( Under step 2A prong II and step 2B, computer components, stated at a high level of generality, is being considered as mere instructions to apply an exception using generic computer components, see MPEP 2106.05(f) ) Dependents of Claims 1, 8 and 15 The remaining dependent claims corresponding to independent claims 1, 8 and 15 do not recite additional elements, whether considered individually or in combination, that are sufficient to integrate the judicial exception into a practical application or amount to significantly more than the judicial exception. The analysis of which is shown below: The claims below recite additional limitations which fail step 2A Prong 2 of the 101 analysis because they do not transform the claim into a practical application. These limitations are too abstract or lack technical improvement that would make the concept practically useful. Without clear utility or integration into a specific field, the claim does not relate to any particular application. It does not meet the requirements of Step 2A Prong 2, as it fails to make the concept meaningfully applicable in practice. The claims also fails Step 2B of the analysis because the additional limitations do not amount to significantly more than the abstract idea itself. The additional limitations do not enhance the claim in a way that would move it beyond its abstract ideas as they minimally elaborate on the core concept without adding any inventive or technical substance. The claims are unpatentable. Claim 2 recites the additional limitation of : The system of claim 1, wherein the teacher model is hosted at an external server accessible via an application programming interface (API). ( Under step 2A prong II and step 2B: this limitation is merely using a application programming interface (API) at a high level of generality such that it is being considered mere instructions to apply an exception, see MPEP 2106.05(f) ) Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible. Claim 3 recites the additional limitation of : The system of claim 1, wherein the task output comprises a natural language description of a target task, and the first task output comprises a programming language segment that executes the target task. ( Under step 2A prong II and step 2B: this limitation is merely using an application programming interface (API) at a high level of generality such that it is being considered mere instructions to apply an exception, see MPEP 2106.05(f) ) Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible. Claim 4 recites the additional limitation of : The system of claim 3, wherein the evaluation environment comprises an execution of the programming language segment, ( Under step 2A prong II and step 2B: this limitation is merely using an evaluation environment to execute a programming segment at a high level of generality such that it is being considered mere instructions to apply an exception, see MPEP 2106.05(f) ) and the feedback comprises an error message. ( this limitation is merely directing feedback to a field of error messages, see MPEP 2106.05(h) ) Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible. Claim 5 recites the additional limitation of : The system of claim 1, wherein the feedback is received upon a user review. ( receiving this set of training data and example data is being considered as mere data gathering and is considered insignificant extra-solution activity under MPEP 2106.05(g), for the purposes of step 2B it should be noted that the courts have recognized receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information) as well-understood, routine, and conventional activity. ) Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible. Claim 6 recites the additional limitation of : The system of claim 1, wherein the teacher model is a pretrained language model, and wherein the refinement output is generated by the teacher model based on an input prompt instructing the teacher model to generate a second task output that revises the first task output based on the feedback. ( Under step 2A prong II and step 2B: this limitation is merely using a teacher model at a high level of generality such that it is being considered mere instructions to apply an exception, see MPEP 2106.05(f) ) Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible. Claim 7 recites the additional limitation of : The system of claim 1, wherein the operation of training the student model based on a training input of the first task output and the feedback and a training label of the refinement output comprises: generating a training input by incorporating the task input, a first task output, and the feedback with a pre-defined refinement template; ( a mental process of evaluation which can reasonably be performed in human mind or with aid of pen and paper ) generating … a student training output based on the training input; ( Under step 2A prong II: generating an output is being considered as mere data outputting, which is considered insignificant extra-solution activity, see MPEP 2106.05(g), for the purposes of step 2B it should be noted that the courts have recognized receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information) as well-understood, routine, and conventional activity. ) by the student model ( this limitation is merely using a student model at a high level of generality such that it is being considered mere instructions to apply an exception, see MPEP 2106.05(f) ) and training the student model based on a loss objective comparing the refinement output and the student training output. ( Under step 2a prong II and step 2B: this training limitation merely recites or is directed to the use of mathematical concepts in the form of calculation or algorithms, specification paragraphs [0060-0061] recite the related mathematical disclosure (loss functions) used to train the student model ) Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible. Claims 8-14 recite limitations substantially similar to claims 1-7, as such a similar analysis applies. Claims 15-18 recite limitations substantially similar to claims 1-4, as such a similar analysis applies. Claims 19-20 recite limitations substantially similar to claims 6-7, as such a similar analysis applies. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-103 AIA The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or non-obviousness. 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. 07-21-aia AIA Claim s 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al.,( Chen, A., Scheurer, J., Korbak, T., Campos, J. A., Chan, J. S., Bowman, S. R., ... & Perez, E. (2023). Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303.16749 . ), hereafter referred to as Chen in view of Lin et al. ( Chen, X., Lin, M., Schärli, N., & Zhou, D. (2023). Teaching Large Language Models to Self-Debug. arXiv e-prints , arXiv-2304.) , hereafter referred to as Lin . Claim 1 : Chen teaches: A system for training a neural network model using a teacher-student framework, ( Chen, abstract, “ We build upon this observation by formalizing an algorithm for learning from natural language feedback at training time instead, which we call Imitation learning from Language Feedback (ILF)… We use ILF to im prove a CODEGEN-MONO 6.1B model’s pass@1 rate by 38% relative (and 10% absolute) on the Mostly Basic Python Problems (MBPP) bench mark, outperforming both fine-tuning on MBPP and fine-tuning on repaired programs written by humans. Overall, our results suggest that learning from human-written natural language feedback is both more effective and sample-efficient than training exclusively on demonstrations for improving an LLM’s performance on code generation tasks. ”, Chen’s Code-ILF teaches training a neural network/code-generation LLM using feedback and refinements. The original CodeGen model is the student being improved; the refinement model/LLM acts as the teacher/refiner. ) the system comprising: a communication interface configured to communicate with a teacher model; ( Chen, page 7, col. 1, paragraph 1, “ For the LLM, we use GPT-3.5 fine-tuned with Feedback Made Easy (FeedME; text-davinci-002 on the OpenAI API)3. We refer to this model as InstructGPT, which is the series of OpenAI models that FeedME belongs to (OpenAI, 2022). We use InstructGPT to generate both the feedback and refinements on the original programs. We then fine tune CODEGEN-MONO 6.1B on the model-generated refinements ”. Chen teaches using an LLM such as InstructGPT to generate feedback/refinements. To the extent the teacher model is external, communication with that model through an API/interface would have been obvious for invoking the LLM service. ) a memory storing a student model and a plurality of processor-executable instructions; and a processor executing the processor-executable instructions to perform operations comprising: ( Chen, abstract, “ We use ILF to improve a CODEGEN-MONO 6.1B model’s pass@1 rate by 38% relative (and 10% absolute) on the Mostly Basic Python Problems (MBPP) bench mark, outperforming both fine-tuning on MBPP and fine-tuning on repaired programs written by humans. ”, Chen Code-ILF trains/fine-tunes a CodeGen-Mono neural model. Implementing that disclosed ML training process necessarily uses stored model parameters, processor-executable training instructions, memory, and processing hardware. ) generating, by the student model, a first task output in response to a task input; ( Chen, page 1, figure 1, “ Given an initial LLM πθ, we sample programs from πθ that do not pass unit tests (indicated by the red X). ”, Chen, Fig. 1, “ Given an initial LLM πθ, we sample programs from πθ that do not pass unit tests ”; Chen, page 3, Algorithm 1, “ C ← {(x0, t, u)| x0 ∼ πθ(·|t), EVAL(x0, t) = 0, (t, u) ∈ D} ”; Chen, page 2, col. 2, section 2.1, “ A task (t, u) consists of a task description t… and a suite u = UNITTESTS(t). ” The mapped student model is the initial CODEGEN-MONO/LLM πθ. The claimed task input is Chen’s task description t, such as an MBPP natural language programming prompt. The claimed first task output is Chen’s generated program x0 sampled from πθ in response to the task description t. Chen further specifies that x0 is an incorrect/generated program that does not pass the unit tests, which corresponds to the first task output generated by the student model. ) obtaining a refinement output generated by the teacher model based on an input of the first task output and the feedback; ( Chen, page 1, figure 1, “ Given an initial LLM πθ, we sample programs from πθ that do not pass unit tests (indicated by the red X). Human annotators write natural language feedback for the incorrect program and a model πRefine generates a refinement- i.e. an improved version of the original program that incorporates the feedback and passes the unit tests (indicated by the green checkmark). Finally, we fine-tune πθ on the refinements. ”, Chen Code-ILF directly teaches a refinement model generating an improved program based on the original program and feedback. That improved program is the claimed refinement output. ) and training the student model based on a training input of the first task output and the feedback and a training label of the refinement output. ( Chen, page 4, col. 2, paragraph 4, “ To implement our algorithm, we independently fine-tune two separate instances of CODEGEN-MONO 6.1B to create πRefine and the final model πθ ∗ . We train πRefine using pairs of incorrect programs and human-written feedback as inputs, with human-written refinements as targets (using the format in Figure 2). In contrast, we train πθ ∗ using natural language task descriptions from MBPP as the inputs and πRefine-generated refinements as the targets. ”, Chen Code-ILF expressly teaches training a model with the original/incorrect program plus feedback as the input and a refinement as the target/label. ) Lin, in the same field of large language model feedback, teaches the following which Chen fails to teach: obtaining, from an evaluation environment, a feedback relating to an accuracy of the first task output; ( Lin, abstract, “ On TransCoder and MBPP where unit tests are available, SELF-DEBUGGING improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, SELF DEBUGGING notably improves sample efficiency, and can match or outperform baseline models that generate more than 10× candidate programs. ”; Page 3, section 3.1, “ The simplest form of automatic feedback is a sentence that just indicates the code correctness without more detailed information. For instance, in text-to-SQL generation, the few-shot prompt provides the feedback message “The SQL prediction above is correct!” for all correct SQL queries, and “The SQL prediction above is wrong. Please fix the SQL.” for wrong predictions ”, Lin teaches obtaining feedback from executing or evaluating generated code. Execution results and failed predictions relate to accuracy/correctness of the generated program, satisfying this limitation. ) It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Chen’s imitation-learning-from-language-feedback framework to use Lin’s Self-Debugging’s execution-result feedback as the evaluation-environment feedback, because both references address improving LLM code-generation outputs by using feedback on generated programs to produce corrected/refined outputs and improve model performance. Claim 2 : Chen and Lin teaches the limitations of claim 1, Chen further teaches: The system of claim 1, wherein the teacher model is hosted at an external server accessible via an application programming interface (API). ( Chen, page 10, col. 1, paragraph 2, “For the LLM, we use GPT-3.5 fine-tuned with Feedback Made Easy (FeedME; text-davinci-002 on the OpenAI API). We refer to this model as InstructGPT… We use InstructGPT to generate both the feedback and refinements on the original programs.” Chen teaches that the teacher/refinement model is GPT-3.5/InstructGPT accessed through the OpenAI API. The OpenAI API is an application programming interface for communicating with OpenAI’s model service, such that prompts/original programs are sent to the API-accessed teacher model and feedback/refinements are received from the API-accessed teacher model. Accordingly, Chen teaches or at least renders obvious a teacher model hosted externally and accessible through an API. To the extent Chen does not use the exact words “external server,” it would have been obvious to a person of ordinary skill to implement an API-accessed OpenAI model as a model hosted on an external server, because API access is the conventional mechanism by which a local system communicates with a remotely hosted model service. ) Claim 3 : Chen and Lin teaches the limitations of claim 1, Chen further teaches: The system of claim 1, wherein the task output comprises a natural language description of a target task, ( Chen, page 2, section 2.1, “ We also have a dataset of tasks D = {(t,u)}. A task (t, u) consists of a task description t ∈ T (e.g. “Write a function that computes the prime factorization of an input integer.”) and a suite u = UNITTESTS(t) ∈ U of unit tests as sociated with task t. Finally, let EVAL : V ∗ ×T → {0,1}be a unit test verification function that indicates whether a program x ∼ πθ(·|t)passes all the unit tests in UNITTESTS(t) ”, Chen’s program synthesis task uses natural-language task descriptions as prompts. This teaches the natural-language description of a target task. ) and the first task output comprises a programming language segment that executes the target task. ( Chen, page 1, figure 1, “ Given an initial LLM πθ, we sample programs from πθ that do not pass unit tests (indicated by the red X). Human annotators write natural language feedback for the incorrect program and a model πRefine generates a refinement- i.e. an improved version of the original program that incorp rates the feedback and passes the unit tests (indicated by the green checkmark). Finally, we fine-tune πθ on the refinements. ”, Chen’s model samples candidate programs for a task. Those generated programs correspond to the claimed programming-language segments. ) Claim 4 : Chen and Lin teaches the limitations of claim 1, Chen further teaches: The system of claim 3, wherein the evaluation environment comprises an execution of the programming language segment, ( Lin, page 2, figure 1, “ SELF-DEBUGGING for iterative debugging using a large language model. At each debugging step, the model first generates new code, then the code is executed and the model explains the code. The code explanation along with the execution results constitute the feedback message, which is then sent back to the model to perform more debugging steps. When unit tests are not available, the feedback can be purely based on code explanation. ”, Self-Debugging expressly performs execution of generated code as part of the feedback loop. That maps to the claimed evaluation environment executing the programming-language segment. ) and the feedback comprises an error message. ( Lin, page 3, section 3.2, “ For code generation tasks where the problem description includes unit tests, besides utilizing code execution to check code correctness, we can also present the execution results in the feedback message, which provides richer information for debugging. Figure 5 presents a sample unit test feedback message for code translation. Intuitively, inspecting runtime error messages and execution results of failed unit tests also helps human programmers debug more effectively ”, Lin teaches using runtime errors (error messages) and execution results as feedback for code correction. ) Claim 5 : Chen and Lin teaches the limitations of claim 1, Chen further teaches: The system of claim 1, wherein the feedback is received upon a user review. ( Chen, page 5, col. 1, paragraph 2, “ The final dataset consists of 195 triples of (incorrect program, human written feedback, human-written refinement). ”, Chen uses human-written natural-language feedback on generated programs. A human annotator/user reviewing the generated program and providing feedback teaches feedback received upon user review. ) Claim 6 : Chen and Lin teaches the limitations of claim 1, Chen further teaches: The system of claim 1, wherein the teacher model is a pretrained language model, ( Chen, page 7, col. 1, paragraph 2, “ For the LLM, we use GPT-3.5 fine-tuned with Feedback Made Easy (FeedME; text-davinci-002 on the OpenAI API)3. ”, Chen uses GPT/InstructGPT models through the OpenAI API to generate feedback and refinements. These are pretrained language models used as teacher/refiner models. ) PNG media_image1.png 325 571 media_image1.png Greyscale Figure 2 of Chen and wherein the refinement output is generated by the teacher model based on an input prompt instructing the teacher model to generate a second task output that revises the first task output based on the feedback. ( Chen, page 6, col. 2, paragraph 1, “ We asked human annotators to write refinements of the original code that incorporated their own previously written feedback, passed the unit tests, and made only minimal edits to the code (see Section 3). The format of the training data also matched the few-shot prompt format (Figure 2) but without the in-context examples of refinements. We denote this model as πRefine, as described in Section 2.3. ”, Chen trains/uses the refinement model with inputs including the incorrect program and feedback, in a prompt-like format, to output a refined program. That teaches generating a second task output revising the first based on feedback. Chen, page 4, col. 1, paragraph 3, “ We create a training dataset for πRefine by further annotating a subset of Cannotated with refinements x1 that repair incorrect programs x0 by incorporating feedback f, such that EVAL(x1,t) = 1 for (x0,f,t) ∈ Cannotated. Further details of our dataset and annotation procedure are in Section 3. ”, Chen’s refinement model repairs or refines the incorrect generated program by incorporating feedback ) Claim 7 : Chen and Lin teaches the limitations of claim 1, Chen further teaches: The system of claim 1, wherein the operation of training the student model based on a training input of the first task output and the feedback and a training label of the refinement output comprises: generating a training input by incorporating the task input, a first task output, and the feedback with a pre-defined refinement template; ( Chen, page 4, col. 2, paragraph 5, “ We train πRefine using pairs of incorrect programs and human-written feedback as inputs, with human-written refinements as targets (using the format in Figure 2). In contrast, we train πθ ∗ using natural language task descriptions from MBPP as the inputs and πRefine-generated refinements as the targets. Further training details are in Appendix A.1 ”, Chen, page 6, col. 2, paragraph 1, “ We asked human annotators to write refinements of the original code that incorporated their own previously written feedback, passed the unit tests, and made only minimal edits to the code (see Section 3). The format of the training data also matched the few-shot prompt format (Figure 2) but without the in-context examples of refinements. We denote this model as πRefine, as described in Section 2.3. ”, Chen’s refinement-training examples use task/program/feedback information arranged in a consistent prompt format. That teaches a predefined refinement template incorporating the task input, initial output, and feedback. ) generating by the student model a student training output based on the training input; ( Chen, page 6, col. 1, last paragraph, “ Our training examples consist of triples of incorrect program, human-written feedback, and human-written refinement. We train the model to maximize the likelihood of the refinement given the program and feedback. ”, Chen, page 4, col. 2, paragraph 4, “ To implement our algorithm, we independently fine-tune two separate instances of CODEGEN-MONO 6.1B to create πRefine and the final model πθ ∗ . ” Chen teaches training/fine-tuning a CODEGEN-MONO 6.1B model instance to generate a refinement from the training input including the program and feedback. During supervised language-model training, the model being trained generates/predicts refinement tokens conditioned on the training input. That predicted refinement/generated token output corresponds to the claimed student training output. Chen’s πRefine is a fine-tuned instance of CODEGEN-MONO 6.1B; therefore, Chen teaches or at least renders obvious that the CodeGen student model generates a student training output based on the feedback-conditioned training input. ) and training the student model based on a loss objective comparing the refinement output and the student training output. ( Chen, page 2, col. 2, section 2.1, “ We also define a fine-tuning function FINETUNE(πθ, D) that applies a gradient-based optimization algorithm to πθ using the associated loss objective calculated over dataset D. ”; Chen, page 3, col. 1, paragraph 2, “ Minimizing the objective in Equation 2 is equivalent to supervised learning, i.e. minimizing the cross-entropy loss ”; Chen, page 6, col. 1, last paragraph, “ We train the model to maximize the likelihood of the refinement given the program and feedback. ” Chen teaches a supervised language-model loss objective, including cross-entropy/negative log-likelihood, calculated during fine-tuning. In such training, the model’s predicted refinement-token distribution/student training output is compared against the target refinement/refinement output, and the loss is minimized when the predicted output matches the target refinement. Therefore, Chen teaches training the student model based on a loss objective comparing the refinement output and the student training output. ) Claims 8-14 recite limitations substantially similar to claims 1-7, as such a similar analysis applies. Claims 15-18 recite limitations substantially similar to claims 1-4, as such a similar analysis applies. Claim 15 recites the following additional limitation for consideration which Chen further teaches: A non-transitory machine-readable medium comprising a plurality of machine- executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising (Chen, page 2, footnote 1, “ We open-source our code and annotated data at… github.com/nyu-mll/ILF-for-code-generation. ”; Chen, page 2, col. 2, “ We also define a fine-tuning function FINETUNE(πθ, D) that applies a gradient-based optimization algorithm to πθ using the associated loss objective calculated over dataset D. ”; Chen, Algorithm 1, “ Imitation learning from natural language feedback for code generation ”; Chen, page 4, col. 2, “ We selected this model because it is open-source, can be fine-tuned on a single 4 × 100 A100 (80 GB) node… ”; Chen, page 4, col. 2, “ To implement our algorithm, we independently fine-tune two separate instances of CODEGEN-MONO 6.1B to create πRefine and the final model πθ ∗ . ” Chen teaches implementing the ILF training algorithm using open-source code, a defined fine-tuning function, and GPU processing hardware. The disclosed open-source code and fine-tuning implementation correspond to machine-executable instructions stored on a non-transitory machine-readable medium, and execution of those instructions by one or more processors causes the processors to perform the ILF operations. To the extent Chen does not expressly use the phrase “non-transitory machine-readable medium,” it would have been obvious to a person of ordinary skill in the art to store Chen’s disclosed training code/model instructions on a non-transitory computer-readable storage medium, such as memory or storage of the computing system used to fine-tune CODEGEN-MONO 6.1B, because storing executable code and model-training instructions on non-transitory storage for execution by processors was a conventional and necessary implementation of Chen’s disclosed software-based machine-learning training system.) Claims 19-20 recite limitations substantially similar to claims 6-7, as such a similar analysis applies . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., ... & Clark, P. (2023). Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems , 36 , 46534-46594. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems , 36 , 8634-8652. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 . Kim, Y., & Rush, A. M. (2016, November). Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing (pp. 1317-1327). US20150356461A1 US20170024641A1 Any inquiry concerning this communication or earlier communications from the examiner should be directed to HYUNGJUN B YI whose telephone number is (703)756-4799. The examiner can normally be reached M-F 9-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /H.B.Y./Examiner, Art Unit 2146 /USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146 Application/Control Number: 18/499,101 Page 2 Art Unit: 2146 Application/Control Number: 18/499,101 Page 3 Art Unit: 2146 Application/Control Number: 18/499,101 Page 4 Art Unit: 2146 Application/Control Number: 18/499,101 Page 5 Art Unit: 2146 Application/Control Number: 18/499,101 Page 6 Art Unit: 2146 Application/Control Number: 18/499,101 Page 7 Art Unit: 2146 Application/Control Number: 18/499,101 Page 8 Art Unit: 2146 Application/Control Number: 18/499,101 Page 9 Art Unit: 2146 Application/Control Number: 18/499,101 Page 10 Art Unit: 2146 Application/Control Number: 18/499,101 Page 11 Art Unit: 2146 Application/Control Number: 18/499,101 Page 12 Art Unit: 2146 Application/Control Number: 18/499,101 Page 13 Art Unit: 2146 Application/Control Number: 18/499,101 Page 14 Art Unit: 2146 Application/Control Number: 18/499,101 Page 15 Art Unit: 2146 Application/Control Number: 18/499,101 Page 16 Art Unit: 2146 Application/Control Number: 18/499,101 Page 17 Art Unit: 2146 Application/Control Number: 18/499,101 Page 18 Art Unit: 2146 Application/Control Number: 18/499,101 Page 19 Art Unit: 2146 Application/Control Number: 18/499,101 Page 20 Art Unit: 2146 Application/Control Number: 18/499,101 Page 21 Art Unit: 2146 Application/Control Number: 18/499,101 Page 22 Art Unit: 2146 Application/Control Number: 18/499,101 Page 23 Art Unit: 2146
Prosecution Insights