DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending and examined in this office action.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 3-5, 9, 11-13, 17, 19-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated Bucaioni et al. (Programming with ChatGPT: How far can we go? January 2024, provided in applicant's IDS) hereinafter Bucaioni.
Per claim 1, Bucaioni further discloses a method of generating a code output in response to a natural language problem description, comprising: receiving, via a communication interface, the natural language problem description; generating, by a neural network based language model, a first candidate code snippet based on a first input prompt combining the natural language problem description and a first instruction to generate the code output; executing, at a code execution environment, the first candidate code snippet based on a unit test thereby producing a first feedback reflecting a correctness of the first candidate code snippet (page 3, right column, 2nd paragraph; “Once the prompt was generated using the GUI tool, it was stored in a GitHub repository. The prompt was then passed as input to ChatGPT to produce an output response, which was then documented in the same GitHub repository for further reference and analysis. Subsequently, the output response was fed as input to LeetCode’s built-in editor to submit it as a solution to the corresponding problem. If the solution generated by ChatGPT passes the test cases on LeetCode, the corresponding feedback, including metrics and statistics, was documented and collected using Google Sheets. In contrast, if an error occurs during the submission process, it is documented, and passed back to the GUI tool”; i.e. ChatGPT receives a prompt in natural language, then generates an output code, the output code is tested in LeetCode (code execution environment), the feedback (error info) is provided to ChatGPT); generating, by the neural network based language model, a second candidate code snippet based on a second input prompt combining the natural language problem description, the first candidate code snippet, and the first feedback; executing, at the code execution environment, the second candidate code snippet based on a runtime test thereby producing a second feedback reflecting a runtime efficiency of the second candidate code snippet; generating, by the neural network based language model, a third candidate code snippet based on a third input prompt combining the natural language problem description, the second candidate code snippet, and the second feedback; and executing the third candidate code snippet at an application associated with the natural language problem description (page 3, right column, 2nd paragraph; “The error message was then formatted into our template for error occurrences, and the modified prompt was once again passed to ChatGPT for another iteration of generating a response. This process was repeated for up to a maximum of three iterations”; page 4, right column; “When a solution was not accepted as correct, the next iteration was based on the error message from previous solution. The format for prompts generated from error message was also straightforward and followed this structure: With input “x” output was “y” but expected result was “z”. If the error was a runtime error the complete error message was used as prompt to be fed to ChatGPT”; i.e. based on the natural language input and the test feedback, ChatGPT produces a second code, then tests the second code in LeetCode to produce second feedback, then repeats the same process again to produce and test a third code).
Per claim 3, Bucaioni further discloses wherein the first feedback takes a form of one or more of a pass of the unit test, an execution failure, a syntax error, a program error, or a timeout error of the unit test, and in response to a failure feedback, the method further comprises: generating, by the neural network based language model, a corrected first candidate code snippet based on an updated input prompt combining the first feedback, the first candidate code snippet, and the natural language problem description; and executing, at the code execution environment, the corrected first candidate code snippet based on the unit test thereby producing an updated first feedback reflecting a correctness of the corrected first candidate code snippet (page 3, right column, 2nd paragraph; “The error message was then formatted into our template for error occurrences, and the modified prompt was once again passed to ChatGPT for another iteration of generating a response. This process was repeated for up to a maximum of three iterations”; i.e. based on the test feedback (error message), generate a new modified prompt, then ChatGPT produces a second code, then tests the second code in LeetCode to produce a second feedback).
Per claim 4, Bucaioni further discloses wherein the runtime test includes measuring an execution time consumed by the second candidate code snippet based on the unit test (page 3, right column, 2nd paragraph; “If the solution generated by ChatGPT passes the test cases on LeetCode, the corresponding feedback, including metrics and statistics, was documented and collected using Google Sheets”; page 2, right column, last paragraph; “In this study, runtime performance refers to the total duration required for a solution to execute”).
Per claim 5, Bucaioni further discloses revalidating a correctness of a neural network generated code output, wherein the revalidation comprises: executing, at the code execution environment, the second candidate code snippet based on the unit test thereby producing a third feedback reflecting a correctness of the second candidate code snippet (page 3, right column, 2nd paragraph; “The error message was then formatted into our template for error occurrences, and the modified prompt was once again passed to ChatGPT for another iteration of generating a response. This process was repeated for up to a maximum of three iterations”; page 3, right column, 2nd paragraph; “LeetCode includes a built-in editor and compiler, which allows users to test their code against a set of predefined test cases for evaluating its correctness and efficiency”; i.e. different users can test/validate a code (second candidate code) using a set of test cases in LeetCode to produce different (second and third) feedbacks).
Claims 9, 11-13 recite similar limitations as claims 1, 3-5. Therefore, claims 9, 11-13 are rejected under similar rationales as claims 1, 3-5.
Claims 17, 19-20 recite similar limitations as claims 1, 3-4. Therefore, claims 17, 19-20 are rejected under similar rationales as claims 1, 3-4.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2, 10 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Bucaioni, in view of Kadel et al. (US patent 11226810) hereinafter Kadel.
Per claim 2, Bucaioni does not explicitly teach wherein the generating of the first feedback comprises: executing the first candidate code snippet based on a testing input to generate a testing output; comparing the testing output with an expected value corresponding to the unit test; determining the first candidate code snippet pass the unit test in response to a difference between the testing output and the expected value being in a predetermined range; and determining the first candidate code snippet fails the unit test in response to the difference between the testing output and the expected value being outside the predetermined range. However, Kadel suggests the above (column 13, line 25-35; testing a code, determine if the test result is within an error range of the expected result, if it does, the code passes the test and can be deployed). Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Bucaioni and Kadel to determine if the test result of a code is within an error range of the expected result, to ensure the fault tolerance and quality of the code.
Claims 10 and 18 recite similar limitations as claim 2. Therefore, claims 10 and 18 are rejected under similar rationales as claim 2.
Claims 6-8, 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Bucaioni, in view of Vembu et al. (US PGPUB 2024/0281225) hereinafter Vembu.
Per claim 6, Bucaioni suggests in response to repeatedly receiving a negative feedback on correctness after a pre- defined quantity of regeneration: measuring execution times of generated candidate code snippets based on the unit test (page 3, right column, 2nd paragraph; “If the solution generated by ChatGPT passes the test cases on LeetCode, the corresponding feedback, including metrics and statistics, was documented and collected using Google Sheets. The error message was then formatted into our template for error occurrences, and the modified prompt was once again passed to ChatGPT for another iteration of generating a response. This process was repeated for up to a maximum of three iterations”; page 2, right column, last paragraph; “In this study, runtime performance refers to the total duration required for a solution to execute”; i.e. iterative testing and receiving error messages and execution time information). Bucaioni does not explicitly teach selecting a candidate code snippet having a shortest execution time. However, Vembu suggests the above (Vembu, paragraph [0130]; testing a plurality of candidate codes, profiling their runtime performance, selecting the candidate code with the best execution time). Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Vembu and Bucaioni to select the candidate code with the best execution time and providing it as feedback to a language model, as this feedback info would help the language model to generate better performing code (for optimization purpose).
Per claim 7, Bucaioni does not explicitly teach wherein multiple candidate code snippets are executed based on the runtime test, each producing a respective runtime efficiency metric, and the method further comprises: selecting one of the multiple candidate code snippets with a highest runtime efficiency metric and a corresponding feedback as part of the third input prompt. However, Vembu in view Bucaioni suggests the above (Vembu, paragraph [0130]; testing a plurality of candidate codes, profiling their runtime performance, selecting the candidate code with the best execution time). Bucaioni further suggests using the runtime profiling info as a part of the third input prompt (page 3, right column, 2nd paragraph). Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Vembu and Bucaioni to select the candidate code with the best execution time and providing it as feedback to a language model, as this feedback info would help the language model to generate better performing code (for optimization purpose).
Per claim 8, Bucaioni does not explicitly teach wherein the code execution environment comprises a hardware environment based on one or more of a central processing unit (CPU), a graphics processing unit (GPU), or an application specific integrated circuit (ASIC). However, Vembu in suggests the above (paragraph [0046]; profiling code performance in an execution environment including CPU utilization rate). Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Vembu and Bucaioni that the code execution environment comprises a CPU, which is a commonly used hardware component to execute a code.
Claims 14-16 recite similar limitations as claims 6-8. Therefore, claims 14-16 are rejected under similar rationales as claims 6-8.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HANG PAN whose telephone number is (571)270-7667. The examiner can normally be reached 9 AM to 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chat Do can be reached at 571-272-3721. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HANG PAN/Primary Examiner, Art Unit 2193
Prosecution Insights