Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
1. This initial office action is based on the application filed on 07/03/2024, which claims 1-9 have been presented for examination.
Status of Claim
2. Claims 1-9 are pending in the application and have been examined below, of which, claims 1, 4 and 7 are presented in independent form.
Priority
3. Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. INDIA 202321051754, filed on 08/01/2023.
Information Disclosure Statement
4. The information disclosure statement (IDS) submitted on 07/03/2024. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Examiner Notes
5. Examiner cites particular columns and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Claim Objections
6. Claims 1-9 are objected to because of the following informalities:
Claims 1, line 39; claim 7, line 38 and claim 7, line 35 recite “if” should be removed or replaced.
Claim 1, line 43; claim 4, line 42 and claim 7, line 39 recite “the program” respectively. There is insufficient antecedent basis for this limitation in the claim.
Claims 2-3, 5-6 and 8-9 depend on claims 1, 4 and 7 are also objected.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
7. Claims 1-9 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1, line 28; claim 4, line 8 and claim 7, line 25 recite “modifying language” respectively render the claims undefine since it is not clear whether the language is natural language or programming language.
Claims 2-3, 5-6 and 8-9 depend on claims 1, 4 and 7 are also rejected.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claim(s) 1-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sungmin Kang (Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction, 2023 – IDS filed on 07/03/2024 – herein after Kang) in view of Rudenko et al. (US Pub. No. 2025/0004915 A1 (Provisional application No. 63/523,916 filed on 06/28/2023) – herein after Rudenko) and in view of Ngiam et al. (US Pub. No. 2025/0045027 A1 – herein after Ngiam).
Regarding claim 1.
Kang discloses
A processor implemented method, comprising:
receiving, via one or more hardware processors, a plurality of textual documents (receiving bug reports, generating tests from bug reports – See Abstract, page 1 and Fig. 1) and extracting context related to each field comprised in the plurality of textual documents (extracting relevant code elements in bug report and synthesizing code aligned with given a natural language description – See page 10, right column. Code synthesis has been approached via SMT solvers in the context of Syntax-Guided Synthesis – See page 11);
rephrasing, via the one or more hardware processors (extracting relevant code elements in bug report and synthesizing code aligned with given a natural language description – See page 10, right column), the extracted context by:
(i) implementing a plurality of rules to obtain a rephrased context having a meaning identical to the extracted context (the test class contains similar method invocations and constants with those used by the LLM-generated test – See page 4, right column); and
(ii) passing each extracted context along with a first set of prompts to a Large Language Model (LLM) to obtain a set of rephrased contexts having a meaning identical to the extracted context (we propose prompting Large Language Models (LLMs) to generate tests. Our use of LLMs is based on their impressive performance on a wide range of natural language processing tasks and programming tasks – See page 1, right column. Prompt engineering, each example report – See Fig. 1);
generating, via the one or more hardware processors, a program (program – See page 3, left column), a validator (test generation – See page 11, left column) and a first set of constraints (rule-based – See page 11, left column) for each extracted context (prompting Large Language Models (LLMs) to generate tests – See page 1, right column. Generates edge-case tests by analyzing structured specifications using rule-based heuristics – See left column, page 11), the rephrased context and the set of rephrased contexts by passing a second set of prompts to the Large Language Model (LLM) (reproduction performance for different prompts – See page 7, left column. Examiner respectfully notes that the different prompts could be interpreted as first, second and third prompts and Table III);
generating, via the one or more hardware processors, one or more test data by running the generated program (LIBRO processes the tests to make them executable in the target program (Figure 1:(C)). LIBRO subsequently identifies and curates tests that are likely to be bug reproducing – See page 3, left column);
assigning, via the one or more hardware processors, a rank to the one or more test data (ranks them to minimize developer inspection effort (Figure 1:(D)). The rest of this section explains each stage in more detail using the running example provided in Table II – See page 3, left column), wherein the ranking is assigned based on a number of validators which are successfully validated (produced an actual bug reproducing test as its first suggestion for 149 bugs. For further validation, we evaluate LIBRO on a recent bug report dataset that we built, finding that we could reproduce 32.2% of bugs in this distinct dataset as well, and verifying that our test suggesting heuristics work in this different dataset as well – See page 2, left column) and selecting the one or more test data with highest ranking (Test Selection and Ranking and Selection and Ranking – See page 5. We focus on finding heuristics indicative of high precision, and minimize the hassle that a developer would have to deal with when using LIBRO – See page 3, left column);
statically refining, via the one or more hardware processors, the generated program using a static refinement engine by (This can be achieved with fine-tuning the LLMs, as studied in other domains (note that Codex is GPT-3 fine tuned with source code data – See page 10, right column):
(i) calling a mathematical library function on the highest ranked one or more test data to generate structural information pertaining to the highest ranked one or more test data for the Large Language Model (LLM) (LLM result from bug report. Call Math test function – See pages 3-4); and
(ii) modifying language of the second set of prompts passed to the Large Language Model (LLM) based on the structural information generated (this can be achieved with fine-tuning the LLMs, as studied in other domains (note that Codex is GPT-3 fine tuned with source code data). As a due diligence, we checked how many tests generated from the Defects4J benchmark verbatim matched developer-committed bug reproducing tests – See page 10, right column);
Kang does not discloses
executing, via the one or more hardware processors, the highest ranked one or more test data on a web application and receiving feedback from the web application; and
dynamically refining, via the one or more hardware processors, each generated program using a dynamic refinement engine by:
(i) passing the feedback to the Large Language Model (LLM) with a third set of prompts, wherein the Large Language Model (LLM) takes content from the feedback and provides:
a) a response if there is an error message;
b) a field corresponding to the error message; and
c) type of a second set of constraints being violated in the error message; and
(ii) refining the program for the field corresponding to the error message dynamically based on the error message received from the feedback by comparing the first set of constraints with the second set of constraints using the dynamic refinement engine.
Rudenko discloses
executing, via the one or more hardware processors, the highest ranked one or more test data on [[a web]] application (the refiner includes a trained ranking model used to rank the candidate patches. The refiner then selects candidate patches based on the ranking and applies each to the corresponding flawed code fragment and presents the different patched versions of the flawed fragment for selection – See paragraph [0020]) and receiving feedback from the [[web]] application (The prompt generator 105 generates a batch of prompts 111 which is input to the LLM 101. The LLM 101 has been fine tuned to generate responses that are code modifications – See paragraph [0021]); and
dynamically refining, via the one or more hardware processors, each generated program using a dynamic refinement engine by: (operations for refining generated responses/code modifications and obtaining a candidate set of patches – See paragraph [0010]. The refiner can include filters and algorithms for refining the candidate patches – See paragraphs [0019-0021]).
(i) passing the feedback to the Large Language Model (LLM) with a third set of prompts, wherein the Large Language Model (LLM) takes content from the feedback and provides (the code security scanner 103 outputs to the prompt generator 105 an indication 104 of a detected flaw(s) including a corresponding weakness type(s) – See paragraphs [0023]):
a) a response if there is an error message (Refinement filters and/or augments the generated responses based on faulty responses – See paragraph [0120]);
b) a field corresponding to the error message (to weakness type, the indication 104 indicates the location of the flawed code fragment 118 (e.g., line number). The weakness type is indicated as CWE-79 Improper Neutralization of Input During Web page Generation (Cross-site Scripting) – See paragraph [0023]); and
c) type of a second set of constraints being violated in the error message (Refinement reduces the generated responses to the amount to be presented and can increase quality. Refinement filters and/or augments the generated responses based on faulty responses and analysis of the responses as a group. Embodiments can also use the ranking model previously discussed to rank the responses by quality measure – See paragraph [0120]); and
(ii) refining the program for the field corresponding to the error message dynamically based on the error message received from the feedback by comparing the first set of constraints with the second set of constraints using the dynamic refinement engine (the refiner 115 ranks the refined pool of code modifications with a ranking model that has been trained to rank code modifications based on similarity metrics relating to structure and change/modification – See paragraph [0036]. With a trained ranking model, the multiple generated responses can be ranked according to predicted quality measures based ranking to allow the highest ranked p generated responses/patches to be selected and provide alternatives for patching a detected flaw – See paragraph [0067-0068]).
It would have been obvious to one ordinary skill in the art before the effective filing date of claimed invention to use Rundenko’s teaching into Kang’s invention because incorporating Rundenko’s teaching would enhance Kang to enable to provide refiner with ranking model has been trained that generated responses can be ranked according to predicted quality measures based ranking to allow the highest ranked p generated responses/patches to be selected as suggested by Rundenko (paragraphs [0067-0068]).
Kang and Rundenko do not disclose
web application.
Ngiam discloses
executing, via the one or more hardware processors, the highest ranked one or more test data on a web application (Select the top-ranked app idea as the final choice. 6. Create a document outlining the chosen app idea, its potential impact, and the reasons for selecting it. 7. Create a detailed implementation plan for the chosen app idea. 8. Send a chat message to the CEO with a summary of the chosen app idea, its potential impact, and a link to the implementation plan document. 9. Begin executing the implementation plan as outlined in the document. – See paragraphs [0239-0248]) and receiving feedback from the web application (providing feedback on the UI about progress while executing the code, the feedback comprising information based on comments associated with function calls in the code – See paragraphs [0342-0345]).
It would have been obvious to one ordinary skill in the art before the effective filing date of claimed invention to use Ngiam’s teaching into Kang’s and Rundenko’s inventions because incorporating Ngiam’s teaching would enhance Kang and Rundenko to enable to provide ChatGPT that provides responses based on the data used for training as suggested by Ngiam (paragraphs [0032-0039]).
Regarding claim 2, the processor implemented method of claim 1,
Kang discloses
wherein the first set of constraints (rule-base – See page 11, left column), the program (program – See page 3, left column), the validator (test generation – See page 11, left column) and the one or more test data with the corresponding ranks are provided as inputs to the static refinement engine (test selection and ranking and selection and ranking – See page 5).
Regarding claim 3, the processor implemented method of claim 1,
Rudenko discloses
wherein the step of refining each generated program statically further comprises updating the second set of prompts (generate multiple responses per prompt and can input a batch of prompts to the code fix model with the model configured to generate one or more responses per prompt. Different techniques can be employed to ascertain quality of the multiple patches to aid in selection – See paragraphs [0067-0071]) and passing the updated second set of prompts to the Large Language Model (LLM) to fix an error in the generated program using an output of the mathematical library function (similarity of text, change, and/or structure between each generated response and the fixed reference code in the corresponding prompt can be used to measure quality. Embodiments can also employ a ranking model that generates a quality prediction based ranking of generated responses based on features corresponding to textual, change, and/or structural similarities of code fragments. With a trained ranking model, the multiple generated responses can be ranked according to predicted quality measures based ranking to allow the highest ranked p generated responses/patches to be selected and provide alternatives for patching a detected flaw – See paragraph [0067]).
It would have been obvious to one ordinary skill in the art before the effective filing date of claimed invention to use Rundenko’s teaching into Kang’s invention because incorporating Rundenko’s teaching would enhance Kang to enable to provide refinement that includes eliminating incorrect response as suggested by Rundenko (paragraphs [0111-0112]).
Regarding claim 4.
Kang and Rundenko disclose
A system, comprising:
a memory storing instructions;
one or more communication interfaces; and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
Regarding claim 4, recites the same limitations as rejected claim 1 above.
Regarding claim 5, recites the same limitations as rejected claim 2 above.
Regarding claim 6, recites the same limitations as rejected claim 3 above.
Regarding claim 7.
Kang and Rundenko disclose
One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
Regarding claim 7, recites the same limitations as rejected claim 1 above.
Regarding claim 8, recites the same limitations as rejected claim 2 above.
Regarding claim 9, recites the same limitations as rejected claim 3 above.
Conclusion
9. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Fields et al. (US Pub. No. 2024/0311922 A1) discloses send a target code and a prompt for code checking to a machine learning (ML) chatbot to cause the ML chatbot to check the target code for errors. The apparatuses, systems and methods may determine whether there is an error in the target code based at least partially on a response from the ML chatbot. The apparatuses, systems and methods may, responsive to determining that there is an error in the target code, determine, via an interaction with the ML chatbot, whether there is a solution to fix the error. The apparatuses, systems and methods may, responsive to determining that there is a solution to fix the error, cause the ML chatbot to (i) fix the error, and/or (ii) present the error and/or the solution to a user – See Abstract and specification for more details.
Lee et al. (US Pub. No. 2023/0315856 A1) discloses the ML model 212 can be configured to respond to the first signal by evaluating if the NL phrase is incorrect and/or incomplete and based on the evaluation perform auto-correction and/or auto-completion. The ML model 212 can be configured to receive a second signal or prompt when a user finishes providing the NL phrase (e.g., a return key press or any other suitable input indicating completion of typing) – See paragraph [0062].
Garg et al. (US Pub. No. 2024/0248686 A1) discloses a pre-trained neural code generation model generates repair code for a method containing a performance bug given a prompt including a code transformation instruction. The code transformation instruction guides the model on how to predict the repair code when the model has not been fine-tuned for the repair code task. The code transformation instruction is retrieved from abstract bug patterns derived from historical performance bug fixes found in commits to a source code repository. The augmentation of the code transformation instruction in the prompt to the pre-trained neural code generation model provides the model with a hint on how the repair code may be generated based on similar performance bug fixes – See Abstract and specification for more details.
Clement et al. (US Pub. No. 2024/0419917 A1) discloses a customized prompt generation service automates prompts to a large language model to perform a specified software engineering task. The service stores the custom data of a client that includes code diff hunks, source code segments, code reviews, repaired code, and unit tests from a code base or repository of the client. Prompt templates are associated with each software engineering task that include the requisite information needed for the large language model to perform the target task. A prompt to a large language model includes examples of the software engineering task from the custom data of the client of the service – See Abstract and specification for more details.
Bowden (US Pub. No. 2025/0013435 A1) discloses provide the list to a large language model (LLM) configured to summarize actions for each role based on the desired features; generate a set of instructions for each action for each role; and generate content for each set of instructions – See Abstract and specification for more details.
Almaer et al. (US Pub. No. 2024/0362209 A1) discloses receiving a request for retrieval of data satisfying one or more criteria, the request including at least one data request parameter; searching a database storing example queries based on the request to identify at least one matching query; providing, to a large language model (LLM), an input prompt to generate a query purporting to retrieve data satisfying the one or more criteria, the input prompt including the at least one data request parameter and the at least one matching query as an example; and receiving, from the LLM, a result including the generated query – See Abstract and specification more details.
Williams et al. (US Pub. No. 2024/0333746 A1) discloses receive the network security vulnerability testing code from the ML chatbot (or voice bot), scan a network to identify network computing devices, scan one or more of the network computing devices to identify security vulnerabilities and vulnerable network computing devices, and/or communicate the security vulnerabilities and/or vulnerable network computing devices to a user – See Abstract and specification for more details.
Bathwal et al. (US Pub. No. 2024/0281446 A1) discloses fine-tunes the model to generate a task-specific generative model. The system employs the task-specific generative model to generate a search result to the search query and analyzes the search result based on a performance metric associated with the task-specific generative model. The system refines the task-specific generative model based on the analyzing of the search result – See Abstract and specification for more details.
Anirudh Khatry (FromWords to Code: Harnessing Data for Program Synthesis from Natural Language, 2023) discloses creating programs to correctly manipulate data is a difficult task, as the underlying programming languages and APIs can be challenging to learn for many users who are not skilled programmers. Large language models (LLMs) demonstrate remarkable potential for generating code from natural language, but in the data manipulation domain, apart from the natural language (NL) description of the intended task, we also have the dataset on which the task is to be performed, or the data context. Existing approaches have utilized data context in a limited way by simply adding relevant information from the input data into the prompts sent to the LLM – See Abstract.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MONGBAO NGUYEN whose telephone number is (571)270-7180. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hyung S. Sough can be reached at 571-272-6799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MONGBAO NGUYEN/ Examiner, Art Unit 2192
Prosecution Insights