DETAILED ACTION
This action is responsive to RCE filed on July 08, 2026.
Claims 1, 8 and 15 have been amended.
Claims 1-20 are pending and are presented to examination.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on July 08, 2026 has been entered.
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 7 and 14 are rejected under 35 U.S.C. 112(d) as being of improper dependent form for failing to further limit the subject matter of the claim from which they depend. Claim 1, as amended, recites performing security validation by analyzing each suggested computer code block using a static application security testing tool to identify security vulnerabilities. Claim 7 depends from claim 1 and recites wherein security validation is performed by using a static application security testing (SAST) tool. The limitation of claim 7 is coextensive in scope with the security-validation limitation already recited in claim 1, differing only by the addition of the parenthetical acronym for the same tool. Claim 7 therefore does not specify a further limitation of the subject matter claimed in claim 1.
Claim 14 depends from claim 8 and recites the same limitation with respect to claim 8, which recites the identical security-validation limitation, and is rejected for the same reason.
Applicant may cancel the claims, amend the claims to place them in proper dependent form, add a new claim, or present a sufficient showing that the claims comply with the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Carrara et al. (US Pub. No. 2025/0298585, hereinafter Carrara – previously presented) in view of Fanning et al. (US Pub. No. 2012/0311535, hereinafter Fanning – previously presented) in view of Schaefer et al. (US Pub. No. 2025/0245122, hereinafter Schaefer – previously presented) in view of Bei Chen et al (“Codet: Code Generation with Generated Tests”, hereinafter Chen) in view of Dong Huang et al. (“AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation”, hereinafter Huang) and further in view of Ray et al. (US Pub. No. 2021/0406004, hereinafter Ray – previously presented).
With respect to claim 1 (Currently Amended), Carrara teaches a system comprising: at least one hardware processor (Carrara discloses that the industrial IDE system 202 includes one or more processors 218, [0060], figure 2.) and a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising: (Carrara discloses a memory 220 storing computer-executable components and instructions that are executed by the one or more processors 218 to carry out the operations of the IDE system, [0060], figure 2.) receiving a request to generate computer code for insertion into source code of a software application, the request including a description of the computer code (Carrara discloses receiving design input comprising “natural language requests received via a chat interface” that request generation of control code and that specify the functional requirements of the code to be generated, [0005], [0061], [0068]; Carrara further discloses that the generated control code is inserted at a specified location of the existing control code 702 of the project, [0103], [0105], figure 14.) generating a prompt based on the description of the computer code and contextual information regarding the computer code (Carrara discloses a specialized prompt engineering layer and associated custom models 222 that generate prompts or meta-prompts based on the user’s natural language inputs together with domain-specific contextual information contained in the custom models, [0058], [0063].) sending the prompt to a first Large Language Model (LLM) to generate a plurality of suggested computer code blocks (Carrara discloses that the generative AI component 210 generates and submits the prompts or meta-prompts “to generative AI models such as large language models (LLMs)” to generate control code, [0058], [0063], [0069].) receiving, from the first LLM, the plurality of suggested computer code blocks (Carrara discloses receiving from the generative AI model the recommended control code generated in response to the submitted prompt, [0069], [0092].)
causing display of a [[highest ranking valid]] suggested computer code block (Carrara discloses that the copilot window 802 causes the recommended control code to be displayed to the user in a code window 1104, [0092], figures 11 and 14. The bracketed portion is addressed below with respect to the discarding and ranking limitations, such that, in the combination, the block that is displayed is the highest ranking valid block.) Carrara does not expressly disclose the three specific validation operations recited in the limitation validating each of the plurality of suggested computer code blocks for syntax, security, and functionality, wherein the validating includes: (This limitation is rendered obvious by the combination set forth below, in which each recited validation operation is taught by a reference in an analogous art.) Carrara is silent to disclose; however, in an analogous art, Fanning teaches performing syntax validation by constructing an abstract syntax tree (AST) representing a hierarchical structure of each suggested computer code block and traversing the AST to perform semantic analysis (Fanning discloses that one or more parsers 108 create abstract syntax trees (ASTs) from source code, [0022]; that the AST representation of the source code is subsequently traversed to construct a hierarchical symbol table expressing the scoping and hierarchical structure of the code, [0024], [0028]; and that a multi-pass traversal of the AST provides semantic analysis by static means, including semantic checks that identify potential correctness issues in the code, [0016], [0021].) It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to validate the syntax of each suggested computer code block generated in Carrara’s system by constructing an abstract syntax tree representing the hierarchical structure of the block and traversing the AST to perform semantic analysis, as taught by Fanning, because Fanning teaches that constructing an AST from source code and traversing it yields significant useful semantic analysis that identifies correctness issues in the code by static means, before the code is run (Fanning, [0016]). Applying Fanning’s AST-based syntactic and semantic analysis to the control code generated by Carrara’s generative AI model is the application of a known source-code-analysis technique, in the same field of source code analysis, to a known code-generation system ready for improvement, to yield the predictable result of identifying syntactically and semantically defective generated code before it is presented to the developer, with a reasonable expectation of success. Carrara in view of Fanning is silent to disclose, however, in an analogous art, Schaefer teaches performing security validation by analyzing each suggested computer code block using a static application security testing tool to identify security vulnerabilities (Schaefer discloses that validation of the LLM-generated code is performed in combination with a code analysis tool comprising “a Static Application Security Testing (SAST) tool” that analyzes the source code to determine whether the code resolves the identified issues, [0042]; Schaefer further discloses that “SAST tools analyze source code at rest, i.e., without executing it, to identify potential security vulnerabilities or other errors”, [0002].) It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to validate the security of each suggested computer code block in the system of Carrara in view of Fanning by analyzing each block with a static application security testing tool to identify security vulnerabilities, as taught by Schaefer, because Schaefer teaches that SAST tools analyze source code at rest, without executing it, to identify potential security vulnerabilities or other errors (Schaefer, [0002], [0042]), and because Schaefer is directed to the same problem of ensuring that code produced with the assistance of a large language model is fit to be presented to and adopted by a developer (Schaefer, [0005]). Applying Schaefer’s static application security testing to the generated code of the combined system would predictably identify insecure generated code before it reaches the developer, thereby improving the reliability of the presented suggestion, with a reasonable expectation of success. Carrara in view of Fanning and Schaefer is silent to disclose the functionality, the automatic discarding of the blocks that fail validation and the ranking of the validated blocks; however, in an analogous art, Chen teaches:
performing functionality validation by generating, by a second LLM [[separate from the first LLM]], a plurality of unit test, and executing the plurality of unit tests by executing each of the plurality of suggested computer code blocks under test conditions defined by the plurality of unit tests generated by the second LLM (Chen discloses generating a plurality of candidate code solutions from a pre-trained language model based on a context, and additionally using a pre-trained language model, prompted with an instruction, to generate a set of test cases for the same programming problem, wherein “A test case is a pair of input and expected output for the function defined in the context” and the instruction specifies an “assert” statement format, such that “The language model will try to complete the instruction by generating plausible input-output pairs for the function”, Chen, Section 2.1 and Figure 2; Chen further discloses that “We execute each generated code solution on each generated test case”, Chen, Section 1 and Figure 1, which comprises “running the function defined by x on the input part of y and comparing the output with the output part of y”, and that the code solution passes the test case where it executes without errors and the output matches the expected output, Chen, Section 2.2. The bracketed portion is addressed below with respect to Huang.)
automatically discarding any suggested computer code block that fails any of the syntax, security, or functionality validations (Chen discloses that the generated code solutions are executed against the test cases and that “we filter out the generated solutions that cannot pass the given example input-output cases”, which Chen designates the “Baseline Filter” method, Chen, Appendix G; the filtering is carried out by the system on the basis of the execution results, and the selection of a solution is thereafter performed among the solutions that remain.) ranking any valid suggested computer code blocks that were not discarded, [[based on one or more quality metrics]] (Chen discloses grouping the code solutions that pass the same generated test cases into consensus sets and scoring each consensus set, and further discloses that “we rank each consensus set by both the number of test cases and solutions in it, and choose the best solution from the highest-ranked consensus set”, Chen, Section 1 and Section 2.2; Chen further discloses that where k code solutions are desired, “we can select the top k consensus sets with the highest scores”, Chen, Section 2.2. The bracketed portion is addressed below with respect to Ray.)
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to validate the functionality of each of the plurality of suggested computer code blocks of the combined Carrara/Fanning/Schaefer system by generating a plurality of unit tests with a large language model and executing each suggested computer code block under the test conditions defined by those generated unit tests, to automatically discard any suggested computer code block that fails a validation and to rank the valid suggested computer code blocks that were not discarded, as taught by Chen, because Chen teaches that a natural way to evaluate the quality and correctness of a code solution is to run it against a set of test cases but that the manual creation of such test cases is costly and time-consuming, and that in real-world applications such as a code generation tool that assists developers in writing code, “it is unrealistic to expect users to provide test cases for every problem they want to solve”, for which reason Chen proposes “to automatically generate test cases for arbitrary programming problems and use them to quickly verify any solution” (Chen, Abstract and Section 1). Carrara itself supplies an express reason to adopt this technique: Carrara teaches generating test scripts by generative AI, which define simulated test inputs and the expected responses of the targeted portion of code, and executing those test scripts to validate the code, specifically “To mitigate the need for a system developer to create custom test scripts 2202 to validate operation of control code 1008, 2102 prior to deployment” (Carrara, [0152], [0156], [0158]-[0159]). A person of ordinary skill therefore had an express reason, supplied by the primary reference itself, to generate the functional validation tests with the generative AI model rather than requiring the developer to author them, and Chen supplies the known technique for doing so, including the filtering out of the generated solutions that fail the tests before selection is performed among those that remain (Chen, Appendix G), yielding the predictable result of automatically verifying the functionality of each generated candidate, eliminating the candidates that fail and ranking the candidates that are verified, with a reasonable expectation of success. Carrara in view of Fanning, Schaefer, and Chen is silent to disclose that the large language model generates the plurality of unit tests is separate from the first LLM; however, in an analogous art, Huang teaches:
separate from the first LLM (Huang discloses a multi-agent code generation framework in which a programmer agent generates the code and a test designer agent generates the test cases against which that code is executed, and expressly states that “AgentCoder requires separate agents for generating code and tests (i.e., the programmer agent and the test designer agent)” and that “Both agents are powered by LLMs”, Huang, Section 4.7; Huang further discloses that “The test designer agent is also powered by LLMs” and that its prompts are designed to generate basic test cases, edge test cases, and large-size inputs, Huang, Section 3.3; and that, “Distinct from the programmer agent and test designer agent that are powered by LLMs”, the test executor agent is implemented as a script that validates the code snippets against the generated test cases in a local environment, Huang, Section 3.4. Huang therefore discloses that the large language model which generates the tests is a second model, separate from the large language model which generates the code.)
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to generate the plurality of unit tests of the combined Carrara/Fanning/Schaefer/Chen system using a second large language model separate from the first large language model that generated the plurality of suggested computer code blocks, as taught by Huang, because Huang teaches that where the code and its test cases are generated by one and the same model, “functional errors may still go undetected as both the code and its test cases are generated by the same model”, and that a tester “being the same LLM as the coder, might erroneously assess erroneous code as correct” (Huang, Section 2.2), and that generating the code and the tests within a single agent leaves “the code generation and test generation processes are not independent” (Huang, Section 1). Huang expressly poses the question whether it is necessary to separate the programmer and test designer roles into two agents, contrasting the claimed arrangement with the alternative of letting “a single agent first generate code and then generate tests, within the same conversation”, and answers it in the affirmative, reporting that the separate-agent arrangement yields materially higher pass@1 and test-case accuracy than the single-agent alternative (Huang, Section 4.7, Tables 6 and 7). Employing a second large language model, separate from the first, to generate the unit tests of the combined system would therefore predictably provide a basis for functionality validation that is independent of the model which produced the code being validated, and would predictably improve the reliability of that validation, with a reasonable expectation of success.
Carrara in view of Fanning, Schaefer, Chen, and Huang is silent to disclose; however, in an analogous art, Ray teaches based on one or more quality metrics (Ray discloses a code audit tool having an analysis engine that executes a series of processes computing objective quality metrics for a body of code, including Algorithmic Complexity Measures 120, [0031]; Coupling/Cohesion Metrics, [0037]; and Testing Metrics 132 including Code Coverage, [0039]; and that generates a standardized report card scoring the code on the basis of those metrics, [0025].)
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to rank the valid suggested computer code blocks that were not discarded in the combined Carrara/Fanning/Schaefer/Chen/Huang system based on one or more quality metrics, as taught by Ray, because Ray teaches computing objective code quality metrics that quantify the quality of a body of code across a set of software dimensions and scoring the code on the basis of those metrics (Ray, [0025], [0031], [0037], [0039]). Ranking the surviving, validated candidate code blocks on the basis of such objective quality metrics would predictably permit the highest-quality validated candidate to be identified and surfaced for display, with a reasonable expectation of success. With respect to claim 2 (Original), Carrara teaches wherein the request is received from an Integrated Development Environment (IDE) and the causing the display includes causing the IDE to display the highest ranking valid suggested computer code block (Carrara discloses that the user interface component 204 of the industrial IDE system 202 receives the user input, [0061], and that the IDE system’s development services include a control code generation copilot having a generative AI component 210 that responds to natural language prompts submitted by the user as part of design input 312, [0068]; Carrara further discloses that the recommended control code is rendered to the user in the code window 1104 of the copilot window 802 of that IDE, [0092], figures 11 and 14.) With respect to claim 3 (Original), Carrara teaches wherein the contextual information includes a location within existing source code at which the requested computer code is to be inserted (Carrara discloses that the user selects a location of the existing control code 702 at which the generated code is to be added, and that the generated control code is inserted at that specified location, [0103], [0105], figure 14.)
With respect to claim 4 (Original), Carrara in view of Fanning, Schaefer, Huang, and Ray is silent to disclose; however, in an analogous art, Chen teaches wherein the contextual information includes examples of input parameters of a function within the requested computer code (Chen discloses that the code solutions are generated by the language model from a context c that “contains natural language problem description in the form of code comment, and a code snippet that includes statements such as imports and the function header”, Chen, Section 2, and that “The original contexts include example input-output cases” for the function defined in the context, Chen, Section 3; Chen further discloses building a one-shot version of a benchmark “by appending a single input-output example to the problem description” supplied to the model, Chen, Appendix G. The input portion of such example input-output cases constitutes examples of the input parameters of the function for which the code is requested.) It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to include, within the contextual information on which the prompt of the combined Carrara/Fanning/Schaefer/Huang/Ray system is based, examples of input parameters of a function within the requested computer code, as taught by Chen, because Chen teaches that “the example input-output cases may provide useful information for code generation” and demonstrates experimentally that the code generation performance obtained using contexts that include the example input-output cases exceeds that obtained using contexts from which those cases have been removed (Chen, Appendix B and Table 6), and further teaches that appending an input-output example to the problem description serves as a formatting hint to the model (Chen, Appendix G). Supplying such input-parameter examples as part of the contextual information of the combined system would predictably guide the large language model to generate code conforming to the expected inputs of the requested function, thereby improving the quality of the generated suggestions, with a reasonable expectation of success.
With respect to claim 5 (Original), Carrara in view of Fanning, Schaefer, Huang, and Ray is silent to disclose; however, in an analogous art, Chen teaches wherein the contextual information includes examples of output of a function within the requested computer code (Chen discloses that “The original contexts include example input-output cases” for the function defined in the context from which the code solutions are generated, Chen, Section 3, and that a test case for that function is “a pair of input and expected output for the function defined in the context”, Chen, Section 2.1; Chen further discloses appending “a single input-output example to the problem description” supplied to the model, Chen, Appendix G. The output portion of such example input-output cases constitutes examples of the output of the function for which the code is requested.)
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to include, within the contextual information on which the prompt of the combined Carrara/Fanning/Schaefer/Huang/Ray system is based, examples of input parameters of a function within the requested computer code, as taught by Chen, because Chen teaches that “the example input-output cases may provide useful information for code generation” and demonstrates experimentally that the code generation performance obtained using contexts that include the example input-output cases exceeds that obtained using contexts from which those cases have been removed (Chen, Appendix B and Table 6), and further teaches that appending an input-output example to the problem description serves as a formatting hint to the model (Chen, Appendix G). Supplying such input-parameter examples as part of the contextual information of the combined system would predictably guide the large language model to generate code conforming to the expected inputs of the requested function, thereby improving the quality of the generated suggestions, with a reasonable expectation of success.
With respect to claim 6 (Original), Carrara in view of Fanning, Schaefer, Chen, and Huang, is silent to disclose; however, in an analogous art, Ray teaches wherein the one or more quality metrics include a cohesion metric (Ray discloses that the computed quality metrics include coupling/cohesion metrics, [0037].) It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to rank the valid suggested computer code blocks that were not discarded in the combined Carrara/Fanning/Schaefer/Chen/Huang system based on one or more quality metrics, as taught by Ray, because Ray teaches computing objective code quality metrics that quantify the quality of a body of code across a set of software dimensions and scoring the code on the basis of those metrics (Ray, [0025], [0031], [0037], [0039]). Ranking the surviving, validated candidate code blocks on the basis of such objective quality metrics would predictably permit the highest-quality validated candidate to be identified and surfaced for display, with a reasonable expectation of success.
With respect to claim 7 (Original), Carrara in view of Fanning, Chen, Huang and Ray, is silent to disclose; however, in an analogous art, Schaefer teaches wherein security validation is performed by using a static application security testing (SAST) tool (Schaefer discloses performing the security validation using a Static Application Security Testing (SAST) tool, [0042]). It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to validate the security of each suggested computer code block in the system of Carrara in view of Fanning by analyzing each block with a static application security testing tool to identify security vulnerabilities, as taught by Schaefer, because Schaefer teaches that SAST tools analyze source code at rest, without executing it, to identify potential security vulnerabilities or other errors (Schaefer, [0002], [0042]), and because Schaefer is directed to the same problem of ensuring that code produced with the assistance of a large language model is fit to be presented to and adopted by a developer (Schaefer, [0005]). Applying Schaefer’s static application security testing to the generated code of the combined system would predictably identify insecure generated code before it reaches the developer, thereby improving the reliability of the presented suggestion, with a reasonable expectation of success.
With respect to claim 8, the claim recites a method comprising limitations similar to the operations recited in claim 1 and is rejected for the same reasons set forth for claim 1; Carrara further teaches a corresponding method carried out by the industrial IDE system 202 ([0005], [0058], [0060], [0092]). With respect to claims 9-14, the claims are directed to a method that corresponds to the system recited in claims 2-7, respectively (see the rejection of claims 2-7 above).
With respect to claim 15, the claim recites a non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations similar to those recited in claim 1, and is rejected for the same reasons set forth for claim 1; Carrara further teaches a memory 220 storing computer-executable instructions that are executed by one or more processors 218 ([0060], figure 2). With respect to claims 16-20, the claims are directed to a medium that corresponds to the method recited in claims 2-6, respectively (see the rejection of claims 2-6 above).
Response to Arguments
Applicant’s arguments filed July 8, 2026 have been fully considered. The arguments directed to the previously applied combination are moot in view of the new grounds of rejection set forth above, which were necessitated by Applicant’s amendment. The following is provided to address Applicant’s arguments as they bear upon the present rejections.
A. The rejection of claims 7 and 14 under 35 U.S.C. 112(d)
Applicant states that claims 7 and 14 have been canceled and requests withdrawal of the rejection as moot. As set forth above, the claim listing filed with the amendment presents claims 7 and 14 in full text with the status identifier “(Original)” and does not present either claim as “(Canceled).” Under 37 C.F.R. 1.121(c), the claim listing governs. Claims 7 and 14 are therefore pending, and the rejection is maintained. The rejection will be withdrawn upon entry of a corrected claim listing canceling these claims, or upon authorization of an examiner’s amendment to that effect.
B. Applicant’s characterization of Li is not supported by the reference
At pages 10–11 of the Remarks, under the heading “Li Does Not Teach Generating Unit Tests by a Second LLM Separate From the First LLM,” Applicant argues that Li does not teach the recited second, separate LLM. Those arguments are moot. Li is not applied in the present rejection; the limitation is addressed by Huang.
Huang discloses a multi-agent code generation framework in which the code and the tests are generated by two different LLM-powered agents. Huang states that “AgentCoder requires separate agents for generating code and tests (i.e., the programmer agent and the test designer agent)” and that “Both agents are powered by LLMs” (Huang, Section 4.7), and that “The test designer agent is also powered by LLMs” (Huang, Section 3.3). Huang expressly contrasts that arrangement with the single-model alternative of letting “a single agent first generate code and then generate tests, within the same conversation,” and reports that the two-agent arrangement yields materially higher pass@1 and test-case accuracy than the single-agent alternative (Huang, Section 4.7, Tables 6 and 7).
Huang further supplies the very consideration Applicant identifies as the point of the amendment. Applicant urges at pages 9–10 of the Remarks that “where a single model generates both the code and the tests that purport to validate that code, the validation is not independent of the artifact being validated,” and that “None of the cited references recognizes this consideration, much less teaches the claimed solution to it.” Huang recognizes precisely that consideration, and does so more than a year before Applicant’s effective filing date: “functional errors may still go undetected as both the code and its test cases are generated by the same model,” and a tester, “being the same LLM as the coder, might erroneously assess erroneous code as correct” (Huang, Section 2.2); and, of a single-agent approach, “the code generation and test generation processes are not independent” (Huang, Section 1). Huang recognizes the problem Applicant asserts is unrecognized, and adopts the claimed solution to it.
Chen corroborates that test-generation models separate from the code-generating model were known in the art, characterizing the AlphaCode approach as one that “relies on a separate test input generation model, which needs extra training and annotation” (Chen, Appendix I).C. The cited combination teaches generating a plurality of unit tests with a large language model
At pages 11–13 of the Remarks, Applicant contends that no cited reference teaches functionality validation in which a plurality of unit tests is generated by an LLM and each candidate code block is executed under the test conditions so defined. Chen is now applied for this limitation. Chen teaches using a pre-trained language model to generate a set of test cases for the function defined in the context, where “A test case is a pair of input and expected output for the function defined in the context” and the model “will try to complete the instruction by generating plausible input-output pairs for the function” (Chen, Section 2.1), and further teaches that “We execute each generated code solution on each generated test case” (Chen, Section 2.2). Chen thus teaches generating a plurality of unit tests with a language model and executing each of a plurality of candidate code solutions under the test conditions those unit tests define.
Applicant’s own characterization of Carrara at page 12 of the Remarks further undercuts the argument. Applicant acknowledges that “Carrara also describes a project testing component that can generate and execute test scripts to validate control code, and even contemplates using generative AI to infer test scenarios and generate test scripts (Carrara at paragraphs [0150]-[0159]).” That acknowledgment is correct. Carrara discloses that each test script 2202 defines “simulated test inputs 2212 to be provided to the portion of the control code” and can also define “expected responses of the targeted portion of code to the simulated inputs 2212” ([0152]); that the generative AI component 210 automatically generates such test scripts “To mitigate the need for a system developer to create custom test scripts 2202 to validate operation of control code” ([0156]); that the generative AI component generates test scripts “for executing functional validation tests” ([0158]); and that the project testing component 212 executes those generated test scripts “in order to comprehensively validate proper operation of the control code” ([0159]). Applicant’s position therefore reduces to the single contention that the model generating the tests must be separate from the model generating the code, which Huang teaches as set forth in Section B above.
D. The asserted absence of a rational underpinning and the number of references
At pages 13–15 of the Remarks, Applicant contends that the rejection assembles references from “markedly different technical contexts,” that the stated reasons for combining do not withstand scrutiny, and that “The claimed invention solves the problem of validating candidate code in a setting where no external test suite exists by generating the validation tests using a second, separate LLM. None of the five references confronts that problem, and none suggests that solution.”
Chen confronts precisely that problem and proposes precisely that solution. Chen states that evaluating a code solution by running it against test cases “relies heavily on the quality and quantity of test cases, which are often costly and time-consuming to create and maintain,” and that “in real-world applications like Copilot, a code generation tool that assists developers in writing code, it is unrealistic to expect users to provide test cases for every problem they want to solve. Therefore, we propose to automatically generate test cases for arbitrary programming problems and use them to quickly verify any solution” (Chen, Section 1). The setting Chen addresses — a code generation tool assisting a developer as code is written, where no external test suite accompanies the request — is the same setting Applicant identifies at page 11 of the Remarks as distinguishing the claimed invention. The problem Applicant asserts is unaddressed in the art is the express motivation of the Chen reference.
The reason to combine is further supplied by the primary reference itself. Carrara teaches generating validation test scripts with generative AI expressly “To mitigate the need for a system developer to create custom test scripts 2202 to validate operation of control code 1008, 2102 prior to deployment” ([0156]). A person of ordinary skill reading Carrara therefore had an express reason, drawn from Carrara and not from Applicant’s disclosure, to have the generative model produce the functional validation tests, and to look to the art of automatically generating tests for model-generated code for a known technique for doing so. The rationale supporting the present rejection is drawn from the teachings of the references themselves, and accordingly does not rest on impermissible hindsight. See MPEP 2145(X)(A); In re McLaughlin, 443 F.2d 1392 (CCPA 1971).
The references are further not drawn from unrelated fields. Carrara, Schaefer, Chen, and Huang are each directed to the generation of computer code by a large language model and to evaluating or improving the code so generated; Chen is specifically directed to selecting the best code solution from among a plurality of candidates generated by a large language model by executing those candidates against generated tests, and Huang to validating such generated code against tests produced by a separate large language model. Fanning is directed to static analysis of source code, and Ray to the measurement of code quality metrics. All are within the field of Applicant’s endeavor or reasonably pertinent to the problem with which Applicant was concerned. See MPEP 2141.01(a).
With respect to the number of references, reliance on a plurality of references does not, of itself, establish nonobviousness. See In re Gorman, 933 F.2d 982, 986 (Fed. Cir. 1991) (reliance on thirteen references in a rejection did not, standing alone, weigh against the obviousness conclusion); MPEP 2145(IV). The dispositive inquiry is whether the prior art, taken as a whole, would have suggested the claimed combination to a person of ordinary skill, and for the reasons set forth above it would have.
E. The specification does not establish criticality of the separate second LLM
Applicant argues at pages 9–10 of the Remarks that generating the unit tests by a second LLM separate from the first “provides an independent basis for functionality validation” and addresses “a specific reliability and integrity consideration.” The specification, however, presents the use of one LLM and the use of separate LLMs as alternative embodiments, without ascribing criticality to either. Paragraph [0014] states that the LLM used “to generate the candidate code suggestions and to generate the functional tests could either be the same one used for both those purposes, or separate LLMs.” Paragraph [0038] states that the unit tests “may be generated by an LLM (either the same LLM used to provide the code block/function suggestions or a separate LLM).” Paragraph [0048] states that “it is not necessary that the same LLM be used to generate both the code and the unit tests, and in some cases it may be preferable to have separate LLMs devoted to these different tasks.”
The specification thus discloses the recited separateness as one of two disclosed alternatives and as a matter of preference in some cases, and identifies no result attributable to the separateness that would be unexpected to a person of ordinary skill. Applicant’s assertion that generating the tests with a separate LLM “provides an independent basis for functionality validation” is an assertion of an advantage, unaccompanied by any showing of unexpected results, and the argument of counsel cannot take the place of evidence in the record. See In re De Blauwe, 736 F.2d 699, 705 (Fed. Cir. 1984); MPEP 2145(I) and 716.01(c). In any event, the present rejection does not rest upon design choice: as set forth in Section B above, Huang expressly teaches generating the tests with an LLM-powered agent separate from the LLM-powered agent that generates the code, and expressly identifies the same reliability consideration Applicant invokes. The recitation of an advantage attending a feature that is expressly taught by the prior art does not rebut that teaching.F. The dependent claims
Applicant’s arguments at pages 16–17 of the Remarks directed to Liguori are moot. The rejections over Liguori and over Blum have been withdrawn, and those references are no longer applied. Applicant’s contention that Liguori’s prompt-template placeholders for data sources and response definitions are not “examples of input parameters of a function within the requested computer code” or “examples of output of a function within the requested computer code” has been considered and is well taken with respect to Liguori; claims 4–5, 11–12, and 18–19 are now rejected over Chen, which teaches that the context from which the language model generates the code solution for a function includes example input-output cases for that function (Chen, Sections 2 and 3; Appendices B and G), as set forth above.
Applicant’s remaining arguments that the dependent claims are allowable by virtue of their dependency from the independent claims are not persuasive for the reasons set forth above with respect to the independent claims. The additional features recited in claims 2–3, 9–10, and 16–17 are addressed on the merits in the rejections set forth above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chen et al. (US Pub. No. 2024/0020096) discloses generating, from a docstring representing natural language text, one or more computer code samples using a trained machine learning model, causing each of the code samples to be executed, verifying each generated code sample based on at least one unit test and computing a functional correctness score therefrom, disregarding or discarding the samples that do not produce the desired result, and outputting the identified sample via a user interface; the reference further discloses that unit tests “may be written by developers, or may be generated by a machine learning model, or a combination of both.” Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANIBAL RIVERACRUZ whose telephone number is (571)270-1200. The examiner can normally be reached Monday-Friday 9:30 AM-6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hyung S Sough can be reached at 5712726799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANIBAL RIVERACRUZ/Primary Examiner, Art Unit 2192
Prosecution Insights