DETAILED ACTION
This action is responsive to the Application filed 9/16/2024.
According to a preliminary amendment, claims 16-29 remain and are submitted for prosecution on merits.
Claim Objections
Claim 17 is objected to because of the following informalities: the term “techical system” (line 2) exhibits a clear typo error. Appropriate correction is recommended.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 16, 18, 20-29 is/are rejected under § 35 U.S.C. 103 as being unpatentable over Shaeffer et al, USPubN: 2024/0311582 (herein Schaeffer) in view of Lemieux et al, “CodaMosa: Escaping Coverage Plateaus in Test Generating with Pre-Trained Large Language Models”, ICSE’23, 26 July 2023, pp. 919-931 (herein Lemieux)
As per claim 16, Schaeffer discloses a computer-implemented method for the automated generation of test code for testing software, comprising the following steps:
generating, via a machine learning model (model trained on source code and natural language to perform a task given a prompt – para 0075), at least one test case and/or a test code, at least based on a code (model to generate the unit test – para 0075; pre-trained on natural language and source code – para 0026; generates unit tests for a change from a pull request of a code – para 0006; change in a pull request – para 0014; Fig. 5A; a pull request needing a unit test – para 0018 ;unit tests to test changes in a pull request – para 0028; to test changes in a pull request – para 0035; changes to a file of the code repository – para 0044) of the software and a prompt (receives a prompt – para 0015; engine 106 generates a prompt to the large language model – para 0059; prompt 310 – Fig. 3; prompts – para 0028, Fig. 2; para 0035; Fig. 4 and related text)
Schaeffer does not explicitly disclose
evaluating the at least one test case and/or the test code wherein an evaluation result is obtained
The testing in Schaeffer utilizes conditioning or natural language instruction from prompt based on which result of a test is based on an expected response that is to match some criterion or requirement (checking for syntax errors in the unit test and upon detecting a … error, correcting the syntax error – para 0078; apply each prompt … ordered sequence … and obtaining a response to each prompt – para 0079; applying the edits … checking for syntax correctness of the existing file having the edits – para 0080; check code of the existing file … and correct the syntax error - para 0072) thus test unit evaluation in terms of expected correctness or satisfying a requirement to be attained is recognized, directly in relation to instructions and query statements set or described from prompts so that a prompt-associated unit test when verified entails a check or else raise an error to then is to be corrected.
Similar to using OpenAI Codex (para 0026) by Schaeffer, Lemieux discloses a prompt based CodaMosa methodology for generation of test code via a LLM per a OpenAI methodology, where code text generated from the prompt-based input into the LLM contains code statements that perform test written in Python under Codex (see get test hints from LLM - Fig. 1, pg. 920; B. Large Language Models of Code – pg. 921-922) like that of search-based software testing (SBST) where evaluation of each CodaMosa module/suite under test along with hints (indicative of prompt conditions) included with test can be performed, such as via deserialization of a coverage test to investigate the core of each code or values obtained from an invocation statement – callables and functions - (R col middle pg. 920 to L col. pg. 921; L col. pg. 922, 923) in order to evaluate if percentage achieved from the module meets a Codex baseline, with result of the evaluation correlated back to the questions pre-established setup by the prompts, such as determining acceptance from coverage values by the Codamosa benchmarking via collection of module results, and comparing the results with a coverage baseline (see Evaluation, A. Experimental Setup, B. Comparison to Baselines pg. 924; Fig. 2, pg. 925)
Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to implement the OpenAI Codex test unit generation in Schaeffer so that test units are implement Codex type of code statements, which, according to this prompt-based test and benchmark methodology -as per Lemieux – can be evaluated or verified at statement levels and correlated to known baseline of the benchmark; because
Evaluating test constructs from the test units generated under the OpenAI codex generator tool by Schaeffer or Lemieux, would enable the evaluation to distinguish functions, conditional constraints in association with “callables” in form of function calls and expected behavior of parameters associated with the callables, so that values, metrics collected by evaluation of these callables in light of embedded hints/conditions of the Codex code (as per Lemieux) can be assessed in relevance to what is expected of them, for mismatch or deviation from a reference, a criterion to be detected, raised so to trigger a corresponding test case in this Codex environment to be returned for corrective adjust to be imparted to its input thereby to minimize difference between a answer and its response as construed from prompt language setting included with the LLM training configuration, the iterative improvement to effort of matching inquiries and responses via a generative model that operates on natural language prompt imparting a enhanced level to the human-adapted cognitive capability to a conventional format input-output matching of a machine learning
As per claim 18, Schaeffer discloses method according to claim 16, wherein the method is performed in an electronic programming environment (conversational test generation ... may be a … service that interacts with code … through application programming interfaces – para 0018; OpenAI Codex … large language model … hosted on a external server … accessed over a network through application programming interfaces – para 0026).
As per claim 20, Schaeffer discloses method according to claim 16, further comprising:
generating, via a further machine learning model (OpenAI Codex model … developers to create their own models … third parties have created a large language model … to developers – para 0026), a test code at least based on the at least one test case and a further prompt (refer to claim 1), wherein the test code is configured to test the software (refer to evaluating the at least one test case per rationale A of claim 1) with regard to the at least one test case.
As per claim 21, Schaeffer does not explicitly disclose method according to claim 16, wherein the evaluating of the test code includes:
executing the test code, wherein the code of the software is executed in an execution environment, at least to the extent required by the at least one test case and/or the test code, wherein an execution result is obtained;
checking whether the execution result matches a reference result in the test code; and
optionally, correcting the at least one test case and/or the reference result, when the execution result does not match the reference result in the test code.
Executing evaluation of generated test code within a environment to the extend required by at least one test case for which execution result is obtained and compared to a reference baseline so that adjust or corrective action is become an option when the obtained metric does not match the baseline has been shown in CodaMosa by Lemieux with test written in Python under Codex (see get test hints from LLM - Fig. 1, pg. 920; B. Large Language Models of Code – pg. 921-922) like that of search-based software testing (SBST) where evaluation of each module/suite under test to the extend of the hints (i.e. prompt conditions) included with test can be performed, in form of investigating core of each code along with values obtained from an invocation statement – “callables” and functions - (R col middle pg. 920 to L col. pg. 921; L col. pg. 922, 923) in order to evaluate if percentage (quantized outcome) achieved from the module meets a Codex baseline, according to which, result of the evaluation is correlated back to the questions pre-established setup by the prompts, to determine acceptance of coverage values based on comparing the results with a coverage baseline; i.e. exceed existing coverage percent (see Evaluation, A. Experimental Setup, B. Comparison to Baselines pg. 924; Fig. 2, pg. 925) such that the benchmark can effect re-run of the algorithm with differently emphasized coverage objectives (para 924) if the number of percent coverage by the SBST fail to meet the baseline in order to only retaining the fittest test cases defined by some fitness metrics (middle R col. pg. 920). Hence, checking whether the execution result matches a reference result in the test code; and optionally, correcting the at least one test case when the execution result does not match the reference result in the test code is recognized.
Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to implement test environment in Schaeffer so that test code generated by the OpenAI codex LLM is carried out with test evaluation that provides execution of the test code, in an execution environment, according to which, at least to the extent required by the at least one test case and/or the test code as set forth in the SBST code experiment environment by CodaMosa (by Lemieux) wherein an execution result is obtained to the effect of checking whether the execution result matches a reference result in the test code – as set forth in Lemieux, so that the evaluation can carry out, optionally, correcting the at least one test case when the execution result does not match the reference result in the test code – so to improve coverage metrics as shown in Lemieux SBST coverage experiment; because
purpose of a test is to evaluate if as constructed, the software under test can fulfill a design target, a expected metric or reference baseline, and by collecting data achieved from the prompt-based test code per a OpenAI codex and verifying if collected test run values match a baseline as set forth above, any sequence of post-run data check or benchmarking instance based on such the evaluation technique would progressively improve baseline matching purposes, more promptly achieve convergence between questions and answers paradigm configured with this prompt-based training, the convergence thereby assisting the OpenAI iterative framework in identifying the best set of test code attained via a large data training model whose smart inference and/or cognitive capability is solely predicated on capturing semantic of the natural text or human language provided in prompt.
As per claim 22, Schaeffer does not explicitly disclose method according to claim 16, wherein the evaluating of the test code includes:
executing the test code, wherein the code of the software is executed in the execution environment, at least to the extent required by the at least one test case and/or the test code;
measuring a code coverage when executing the code of the software;
checking whether the code coverage is sufficiently large.
However, use of test code evaluation to determine if convergence value attained from a test run meet a coverage reference under the SBST experiment is shown in Lemieux based capturing metrics of the code coverage by the Codex test constructs to see if the percent of coverage size exceeds existing coverage of past experiments - refer to rationale A of claim 1 and teachings by CodaMosa per claim 21.
Therefore, evaluating of the test code including steps of executing the test code, wherein the code of the software is executed in the execution environment, at least to the extent required by the at least one test case and/or the test code; measuring a code coverage when executing the code of the software; checking whether the code coverage is sufficiently large would have been obvious for the same reasons set forth with claim 21 as set forth above.
As per claims 23-25, Schaeffer does not explicitly disclose method according to claim 16, wherein the evaluating of the test code comprises
(i) a dynamic assert for redundancy of the test code.
(ii) measuring suitability of the test code based on mutation testing; and
(iii) executing the test code, optionally depending on the evaluation result, wherein the software is tested.
Applying a self-augmenting test with repeated instances of mutating the test code as shown in Lemieux to facilitate tracking of coverage with redundantly repeated insertion of mutant version into the original test code with bump-up versions at bump-up locations (Fig. 1 pg. 920) in order to promote higher coverage by the code module under test and increase amount of fitness of the code portions being evaluated is depicted in the evolutionary mutation test (R col. pg. 920) by the CodaMosa test experiment by Lemieux, where benchmarking thereby is based on test result evaluation or correlating to baseline values (L middle pg. 921) with repeated mutation of code portions as needed to meet higher coverage points (R col. pg. .921) so to enable the most fitted test cases to be retained (middle R col. pg 920). Thus, the evolutionary algorithm creating mutants per a self-bumping of the original code portions per an incremental need basis to promote certain versions of the portions under test toward meeting higher fitness criteria for further mutations (middle L col. pg. 919) entails a evolutionary approach that randomly augments the code using mutants replacement as versions dynamically inserted during the evaluation to promote a better code portions that fits a higher coverage metric.
Thus, executing the test code, optionally depending on the evaluation result, including determination of the most fitted code portions to retain on basis of measuring suitability of the test code based on mutation testing and dynamic and dynamic assert for redundancy as part of the mutation test is recognized.
Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to implement the prompt-based Codex test code generation in Schaeffer so that evaluating of the test code obtained for the Codex LLM arrangement in Schaeffer would include
executing the test code optionally depending on the evaluation result, the dependency thereof including determination of the most fitted code portions to retain on basis of measuring suitability of the test code based on mutation testing – as set forth in the mutation bumping by Lemieux - and dynamic asserting of redundantly inserted mutants as part of the mutation test to attain a higher suitable test cases that satisfy the SBST experiment baseline – as per the self evolutionary code evaluation in Lemieux CodaMosa approach; because
augmenting (bump-up) a test code generated by a Codex OpenAI LLM engine to investigate quality a search algorithm on basis of repeatedly mutating test cases and random location and frequency so to keep the mutated test cases eligible for further mutations helps redirect search space of the target search algorithm to amore useful portions of the space notably when the space is pertinent to a experiment for which developers wish to find bugs and improve quality of the search program; and by combining iteration of stages that (i) includes test result being correlated against a desired developer criterion, and (ii) augments space of significance in the test code via injection of mutants replacements made repeatedly by (iii) random redundancy into the test code for increasing likelihood that such mutated test code can be further mutated - as indicated in the evolutionary incremental mutation test in Lemieux CodaMosa approach - would have for effect in increasing test cases that are deemed best to support the highest code search coverage translatable into a highest quality and efficient search coverage program, notably when this coverage is investigated by the prompt-based LLM model test generation per the OpenAI codex – as in Schaeffer, so that the most fitted test cases for the target search algorithm can be identified as fulfilling a design objective.
As per claim 26, Schaeffer discloses a computer-implemented method for further training a machine learning model and/or further machine learning model (refer to claim 20), wherein the machine learning model is configured to generate at least one test case (refer to claim 1) and/or a test code for testing software at least based on a code of the software (refer to claim 1) and a prompt (refer to claim 1), and the further machine learning model (see claim 20) is configured to generate a test code for testing the software at least based on (see above) at least one test case and a further prompt,
the method comprising the following steps:
adapting the machine learning model (refer to claim 1) and/or the further machine learning model (refer to claim 20) at least based on at least one test case and/or the test code (see above), and at least one evaluation result, wherein the at least one evaluation result is obtained (refer to rationale A of claim 1) by evaluating the at least one test case and/or the test code;
wherein the at least one test case and/or the test code has been generated and evaluated (refer to rationale A of claim 1) according to a method for automated generation of test code for testing the software (checking for syntax errors in the unit test and upon detecting a … error, correcting the syntax error – para 0078; apply each prompt … ordered sequence … and obtaining a response to each prompt – para 0079; applying the edits … checking for syntax correctness of the existing file having the edits – para 0080; check code of the existing file … and correct the syntax error - para 0072), the method for automated generating including:
generating, via the machine learning model, the at least one test case and/or the test code, at least based on a code of the software and a prompt,
(all of which having been addressed in claim 1) and
evaluating the at least one test case and/or the test code, wherein the evaluation result is obtained.
(all of which having been addressed in claim 1)
As per claim 27, Schaeffer does not explicitly disclose method according to claim 26, wherein the adapting of the machine learning model and/or the further machine learning model at least based on the at least one test case and/or the test code and on the at least one evaluation result includes:
calculating at least one reward at least based on the at least one evaluation result; and
adapting the machine learning model and/or the further machine learning model at least based on the at least one test case and/or the test code, and based on the at least one reward.
However, the incremental mutant replacement and evolutionary mutation test under the Codamosa approach based on generated test cases governed by directives of prompt language as in Lemieux (refer to claims 23-24) is geared to seek the highest quality of search for a SBST experiment with increased code coverage that overcome snag caused by coverage stall or length thereof (Fig. 1, pg. 920) to that code coverage achieved by the Codamosa represents an over achievement respective to past SBST experiments, thanks to mutation test and evolutionary replacement method (refer to Lemieux per claims 23-25) with proper identification of test cases that fit this best coverage scenario (L col. A. Search-based software testing - pg. 921); therefore a higher coverage percent of test associated with search achieved via benchmarking over past experimentation (B. Comparison to Baselines, pg. 924 R col. to Fig. 2, pg. 925) in Lemieux represent a form of reward, therefore adapting the machine learning model and/or the further machine learning model based a first prompt and a further prompt to generate test cases that following the coverage achievement percent reward would have been obvious.
Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to implement Codex and LLM generation of test code based on prompt in Schaeffer system so that for a code coverage maximization experiment, adapting of the machine learning model and/or the further machine learning model at least based on the at least one test case and/or the test code and on the at least one evaluation result would necessarily include
calculating at least one reward at least based on the at least one evaluation result – as shown in the percent increase in CodaMosa approach by Lemieux; and by virtue of obviousness
adapting the machine learning model and/or the further machine learning model at least based on the at least one test case and/or the test code, and based on the at least one reward because
this adaptation (LLM generating of test to support efficiency and quality of a target algorithm) based on known reward would not only make good use of experimentation result and findings from benchmarks, but would also optimize further resources for seeking the most optimum set of test code for implementing a algorithm for which optimum reward has been attained from a past and substantially verified amount of experimentation.
As per claim 28, Schaeffer discloses a computer system configured to automated generation of test code for testing software, the computer system configured to:
generate, via a machine learning model, at least one test case and/or a test code, at least based on a code of the software and a prompt (refer to claim 26); and
evaluate the at least one test case and/or the test code, wherein an evaluation result is obtained (see claim 26 and rationale A of claim 1).
As per claim 29, Schaeffer discloses a non-transitory computer-readable medium on which is stored a computer program for automated generation of test code for testing software, the computer program, when executed by a computer, causing the computer to perform the following steps:
generating, via a machine learning model, at least one test case and/or a test code, at least based on a code of the software and a prompt; and
evaluating the at least one test case and/or the test code, wherein an evaluation result is obtained.
((all of which having been addressed in claim 1)
Claims 17 is/are rejected under § 35 U.S.C. 103 as being unpatentable over Shaeffer et al, USPubN: 2024/0311582 (herein Schaeffer) in view of Wu et al, USPubN: 2024/0420418 (herein Wu) and Sicconi et al, USPubN: 2024/0112562 (herein Sicconi) and Gupta et al, USPubN: 2024/0411808 (herein Gupta)
As per claim 17, Schaeffer discloses method according to claim 16, wherein the software is configured to control and/or regulate and/or monitor a technical system (para 0068-0069),
Schaeffer does not explicitly disclose the technical system including a cyber-physical system including at least one computing unit of a vehicle.
Gupta discloses generative AI system in cyber environment (Bluetooth, NFC, RF – para 0069; wireless - para 0081) or communication NW equipped advanced capabilities of training models in form of LLM (para 0205; Fig. 7) for generating digital message(s) based on instructions from text-based input prompt (claim 19, pg. 25; para 0195-0196) by converting said message response into speech-based audio output to user of a vehicle as part of the vehicle voice assistance system configured to perform tasks such as answering questions (Fig. 2; para 0091)
Wu discloses systems implemented in a cyber environment (para 0152) with Large Language Models for performing in part conversational AI operations applicable to automotive systems such as autonomous or semi-autonomous vehicle (para 0034), the generative AI (LLM) trained to provide language representation to test autonomous machine application (para 0345, 0357), to fill gaps or correct errors in the sensors underlying the environment (para 0036) e.g. to comply with real-world rules useful for testing autonomous or semi-autonomous vehicles or to convert output of LLM to suitable map formats, to solve problems related to mapping, such as maps for tasks in autonomous vehicles or for advanced driver assistance systems (para 0037)
Sicconi discloses large language model (para 0037-0038) situated in vehicle environment operating with bluetooth, wireless connection (para 0041) in form of a digital assistant using AI model capable of understanding human language and engage in conversations, answer question, including assessing risk levels perceived from prompts for a safety response (Fig. 1), where the AI model recognize voice commands from the driver, detect or perceive critical levels from the dialog flow and return a emergency override recommendation that displays a visual alert or suggests relevant voice commands to engage emergency protocols (para 0036) via issuing prompts to the driver (para 0046)
Thus, LLM based software configured to control and/or regulate and/or monitor a technical system such as a cyber-physical system (bluetooth, wireless connection) having computing unit of a vehicle is recognized.
Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to apply the prompt-based LLM generation of test code in Schaeffer so that the form of testing can be applied to answer-response methodology such as smart assistance system in autonomous vehicle, as a form of advanced technical system associated with compute unit in a vehicle – as per Wu, Gupta and Sicconi – set in a cyber-physical environment that combines wireless, Bluetooth technology to software control for a physical automobile; because
this Codex function using the prompt-based smart AI response as an advanced cognitive and code-generation layer in automotive or autonomous vehicle enable shifting from a rule-based rigid machine to a agentic, adaptive and human aligned mechanism as this Codex function – as set forth above in Lemieux – can operate as a semantic/reasoning bridge between high-intent (prompt/natural language) and low level execution of control and simulation, that signifies a edge over fixed or predictable control established in automotive system, as this prompt-based cognitive approach can translate high-level of abstract into scripted expression of dynamic modification of parametric tuning like deceleration or steering, into self-driving directive passed to mechanical controls or actuators of the car system, while be able to interpret nuanced intent (user uttered prompt) and contextualized environmental data, telemetric values or unsafe conditions and convert them into safer automation script, including optimizing the vehicle route and speed constraints within safe or more secure legal bounds.
Claims 19 is/are rejected under § 35 U.S.C. 103 as being unpatentable over Shaeffer et al, USPubN: 2024/0311582 (herein Schaeffer) in view of Lemieux et al, “CodaMosa: Escaping Coverage Plateaus in Test Generating with Pre-Trained Large Language Models”, ICSE’23, 26 July 2023, pp. 919-931 (herein Lemieux) further in view of Padgett et al, USPubN: 2024/0160902 (herein Padgett)
As per claim 19, Schaeffer does not explicitly disclose method according to claim 16, further comprising:
retaining the at least one test case and/or the test code according to a predetermined criterion based on the evaluation result; and
optionally, discarding the at least one test case and/or the test code otherwise.
Lemieux provides iterative mutation test expanded as needed with expansion of callables into the Codamosa code to promote the callables used with the evolutionary mutation test (R col. pg. 920) where benchmarking based on test result evaluation enable the most fitted test cases to be retained (middle R col. pg 920), the evolutionary algorithm creating mutants version of the portion under test to keep test cases with higher fitness for further mutations, hence to promote higher coverage percent of the program under test (middle L col. pg. 919)
Padgett discloses analysis of outcome generated from a ML model training according to a OpenAI methodology using a generative AI (e.g. LLM) trained with taking input prompt (para 0019) in text form and providing output that reflects to some degree the input prompt (para 0029-0030), with evaluating the output for matches (input prompt and output – para 0034), shortcomings such as “problematic” matches or falsehoods such as “too similar” matches to a pre-existing content (para 0031; Fig. 2) with effect of iteratively adjusting the AI model input, as part of a two-stage process to carry out filtering of generated outputs (Fig. 4 and related text; para 0107) in reply to input prompts, the filtering to determine a exact match to item in repository which may result in blocking or discard of the result (para 0096) or trigger a re-run using a second AI model with altered input to mitigate the error (para 0122)
Hence, filtering with effect to retain a (LLM) result or discard a (LLM) result from correlating generative AI training output of a prompt-driven training to condition, criterion or content of a input prompt is recognized.
Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to implement the prompt-based generative AI training under the OpenAI codex by Schaeffer so that the automated test code generation would include capability to filter output from the evaluated test code including therewith effect of retaining the at least one test case/code – as in Lemieux - according to a predetermined criterion based on the evaluation result; and optionally, discarding the at least one test case and/or the test code otherwise as set forth above by Padgett output checking approach; because
removing or discarding non-matching code result in reference to baseline can reduce the payload to restructure the LLM for generate test at a subsequent stage to a filtering instance, and retaining code results from evaluation of Codex constructs as set forth above (see Lemieux) would reduce the amount of test coverage resulting from each instance of test code evaluation; e.g. one that exposes which generated output to consider good and which to consider detrimental toward reconfiguring input to the next cycle of training, optimizing therewith resources of a LLM code development environment.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Tuan A Vu whose telephone number is (571) 272-3735. The examiner can normally be reached on 8AM-4:30PM/Mon-Fri.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Chat Do can be reached on (571)272-3721.
The fax phone number for the organization where this application or proceeding is assigned is (571) 273-3735 ( for non-official correspondence - please consult Examiner before using) or 571-273-8300 ( for official correspondence) or redirected to customer service at 571-272-3609.
Any inquiry of a general nature or relating to the status of this application should be directed to the TC 2100 Group receptionist: 571-272-2100.
/Tuan A Vu/
Primary Examiner, Art Unit 2193
June 26, 2026
Prosecution Insights