DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to pending claims 1-20 filed 9/29/2023.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 10-13, 16 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Nashid ("Retrieval-based prompt selection for code-related few-shot learning", published 7/14/2023).
For claim 10, Nashid discloses: a computer-implemented method for dynamically generating prompts for a generative artificial intelligence (AI) model (fig.1 gives overview of prompt generation for a generative AI model such as Codex), the method comprising:
receiving input content for evaluation by a generative Al model (fig.1, §III.B ¶3: “Incomplete Code” is received as a query for tasks, such as assertion generation or code repair);
receiving an input-content embedding for the input content (p.3: “Retrieval-based Demonstration Selection” contemplates various input embeddings);
receiving trait data and trait-data embeddings for the trait data (ibid: embeddings for additional data from the demonstration pool are received for comparison, the demonstration pool comprising various trait data (see fig.2-3: various tags, names, data types, assertion types, error types, etc.) and corresponding embeddings according to p.3);
identifying similar trait data by comparing the input-content embedding with the trait- data embeddings, wherein the similar trait data is a subset of the trait data that is similar to the input content (p.3: Retrieval-based Demonstration Selection: similar trait data is identified by comparing features of the received demonstration data with the input, the similar trait data being subsets identified by the two projection techniques as being similar);
generating a prompt including the input content and the identified similar trait data (fig.1: demonstrations are combined with the query via a template to generate the final prompt, see p.3: “Template Selection”);
providing the prompt to the generative Al model (fig.1: the prompt is provided to Codex for inference); and
receiving, from the generative AI model in response to the prompt, an output payload including an evaluation of the input content (ibid: an output is received from Codex for evaluation).
For claim 11, Nashid discloses the system of claim 10, as described above. Nashid further discloses: wherein trait data other than the similar trait data is omitted from the prompt (fig.1, p.3: “Retrieval-based Demonstration Selection”: closest data is selected and the remainder of the dataset not included in the prompt).
For claim 12, Nashid discloses the system of claim 10, as described above. Nashid further discloses: wherein identifying the similar trait data by comparing the trait-data embeddings with the input-content embedding comprises:
generating a ranked list of trait data based on a similarity of the trait-data embeddings to the input-content embedding (§II.B¶3 (p.2), p.3: “Embedding (SRoBERTa)”: top-K, top-N, hence, generating a ranked list based on similarity); and
selecting a top N number of trait data, from the ranked list, as the similar trait data (ibid).
For claim 13, Nashid discloses the system of claim 10, as described above. Nashid further discloses: wherein the prompt further includes examples of the identified similar trait data (figs.2-3 shows examples or demonstrations of trait label or name data).
For claim 16, Nashid discloses the system of claim 10, as described above. Nashid further discloses: wherein the trait data includes data that is pre-tagged with classifications (figs.2-3 show various tags for the output, e.g., assertion type, error fix type, etc.), and wherein the evaluation of the input content is a classification of the input content as one of the pre-tagged classifications (figs.2-3: output classification as assertion type, bug fix statement type, etc.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-9, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Nashid ("Retrieval-based prompt selection for code-related few-shot learning", published 7/14/2023) in view of Berglund (US 20240403341 A1).
For claim 1, Nashid discloses: a system for dynamically generating prompts for a language model (fig.1, §II.B gives overview of the system, with various data sources being used to create a prompt for the Codex engine), the system comprising:
receive input content for evaluation by the language model (fig.1, §III.B ¶3: “Incomplete Code” is received as a query for tasks, such as assertion generation or code repair);
receive trait data that includes pre-tagged data (fig.1: Assertion Generation, Program Repair; see also samples in fig.2-3: receiving demonstrations based on the input query code, the demonstration pool being demonstrations having various traits such as assertion type, program type, program error type, etc., the demonstration data being tagged or associated with various Assertions data, various correct fixes data, etc.);
identify similar trait data by comparing the received trait data to the input content, wherein the similar trait data is a subset of the trait data that is similar to the input content (p.3: Retrieval-based Demonstration Selection: similar trait data is identified by comparing features of the received demonstration data with the input, the similar trait data being subsets identified by the two projection techniques as being similar);
generate a prompt including the input content and data based on the identified similar trait data (fig.1: demonstrations are combined with the query via a template to generate the final prompt, see p.3: “Template Selection”);
provide the prompt to the language model (fig.1: the prompt is provided to Codex for inference); and
receive, from the language model in response to the prompt, an output payload including an evaluation of the input content (ibid: an output is received from Codex for evaluation).
Nashid does not disclose: at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the system to perform operations.
Berglund discloses: at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the system to perform operations (fig. 5: 502, 506, 0085-90).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the system of Nashid by incorporating the computing device of Berglund. Both concern the art of querying LLM’s via prompting, and the incorporation would have, according to Berglund, allow implementation on a computing system (0085).
For claim 2, Nashid modified by Berglund discloses the system of claim 1, as described above. Nashid further discloses: wherein the operation of identifying the similar trait data further comprises:
receive an input-content embedding for the input content (p.3: “Retrieval-based Demonstration Selection” ¶1, §II.B ¶3: selecting Top-K based on two different vectorization techniques, embedding (sRoBERTa) or frequency (BM-25));
receive trait-data embeddings for the trait data (ibid: trait data embeddings are received for the various demonstration pool elements for identification during the query); and
identify the similar trait data by comparing the trait-data embeddings with the input- content embedding (ibid: trait data is identified via similarity for demonstration, Top-K selection).
For claim 3, Nashid modified by Berglund discloses the system of claim 2, as described above. Nashid further discloses: wherein comparing the trait-data embeddings with the input- content embedding comprises performing a cosine similarity analysis (p.3: Embedding (sRoBERTa): cosine similarity is used for sRoBERTa embedding).
For claim 4, Nashid modified by Berglund discloses the system of claim 2, as described above. Nashid further discloses: wherein identifying the similar trait data by comparing the trait-data embeddings with the input-content embedding comprises:
generate a ranked list of trait data based on a similarity of the trait-data embeddings to the input-content embedding (§II.B¶3 (p.2), p.3: “Embedding (SRoBERTa)”: top-K, top-N, hence, generating a ranked list based on similarity); and
select a top N number of trait data, from the ranked list, as the similar trait data (ibid).
For claim 5, Nashid modified by Berglund discloses the system of claim 1, as described above. Nashid further discloses: wherein the data based on the identified similar trait data is the similar trait data and the remainder of the trait data is omitted from the prompt (fig.1, p.3: “Retrieval-based Demonstration Selection”: closest data is selected and the remainder of the dataset not included in the prompt).
For claim 6, Nashid modified by Berglund discloses the system of claim 5, as described above. Nashid further discloses: wherein the data based on the identified similar trait data further comprises examples of the identified similar trait data (figs.2-3 shows examples or demonstrations).
For claim 7, Nashid modified by Berglund discloses the system of claim 1, as described above. Nashid further discloses: the trait data includes traits and statements (figs.2-3 shows exemplary trait data including tagged traits, e.g., labels, identifiers, etc. and code statements);
the similar trait data includes at least one statement from a particular trait (ibid: similar statements are identified from the traits, e.g., Method Under Test, Buggy, Fixed, function name traits, rule-id, evidence, etc.); and
the data based on the identified similar trait data includes all the statements from the particular trait (figs.2-3 shows prompt generated from statements for the various traits) and statements from other traits in the trait data are omitted from the prompt (ibid: other statements from other demonstrations are excluded).
For claim 8, Nashid modified by Berglund discloses the system of claim 1, as described above. Berglund further discloses: wherein the output payload comprises relevant scores for a plurality of proposed evaluations for the input content, and the operations further comprise (0051-53: calculating confidence scores for a plurality of responses from a LLM):
postprocess the output payload to identify proposed evaluations that have relevant scores exceeding a threshold (ibid: evaluate confidence scores); and
at least one of transmit or cause display of the proposed evaluations having the relevant scores exceeding the threshold (0053: provision for output such as via GUI).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the system of Nashid by incorporating the ranking technique of Berglund. Both concern the art of querying LLM’s via prompting, and the incorporation would have, according to Berglund, improve effectiveness of LLM-based querying techniques (0011-13).
For claim 9, Nashid modified by Berglund discloses the system of claim 1, as described above. Nashid further discloses: wherein the evaluation of the input content is a classification of the input content (fig.1: classification as type of assertion statement, as type of buggy statement, etc., see figs.2-3: Output).
For claim 15, Nashid discloses the system of claim 10, as described above. Nashid does not disclose: wherein the output payload comprises relevant scores for a plurality of proposed evaluations for the input content, and the method further comprises postprocessing the output payload to identify proposed evaluations that have relevant scores exceeding a threshold.
Berglund further discloses: wherein the output payload comprises relevant scores for a plurality of proposed evaluations for the input content, and the method further comprises (0051-53: calculating confidence scores for a plurality of responses from a LLM):
postprocessing the output payload to identify proposed evaluations that have relevant scores exceeding a threshold (ibid: evaluate confidence scores).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the system of Nashid by incorporating the ranking technique of Berglund. Both concern the art of querying LLM’s via prompting, and the incorporation would have, according to Berglund, improve effectiveness of LLM-based querying techniques (0011-13).
Claim(s) 14 are rejected under 35 U.S.C. 103 as being unpatentable over Nashid ("Retrieval-based prompt selection for code-related few-shot learning", published 7/14/2023) in view of snk ("Java Class with single method OK", published 9/8/2010).
For claim 14, Nashid discloses the system of claim 10, as described above. Nashid further discloses: the trait data includes categories and subcategories (figs.2-3 shows various hierarchies of tags, e.g., AssertNull as a subcategory under Assert, getFullYear as a subcategory to the date class, etc.);
the similar trait data includes at least one subcategory from a particular category (p.3: “Retrieval-based Demonstration Selection”: both embeddings would include encodings and embeddings of the various subcategory tokens).
Nashid does not disclose: the prompt further comprises all the subcategories from the particular category.
However, including all subcategories of a class would arise naturally during operation of the system. In particular, snk discloses: all the subcategories from the particular category are included (p.1 discloses the advantages of wrapping a single method in a class, such as to allow for later expansion), hence, combination with Nashid yielding: the prompt further comprises all the subcategories from the particular category (inclusion of any single-method class, such as taught by snk, in demonstration code would yield the prompt including all subcategories of a class, as claimed).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the system of Nashid by incorporating the java class hierarchy of snk. Both concern the art of java code development, and the incorporation would have, according to snk, allow for future expansion of a class (p.1 footnote 1).
Claim(s) 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Nashid ("Retrieval-based prompt selection for code-related few-shot learning", published 7/14/2023) in view of Kats (US 10409561 B1).
For claim 17, Nashid discloses: a computer-implemented method for dynamically generating prompts for a language model, the method comprising:
receiving input content for classification by a language model (fig.1, §III.B ¶3: “Incomplete Code” is received as a query for tasks, such as assertion generation or code repair);
receiving an input-content embedding for the input content (p.3: “Retrieval-based Demonstration Selection” contemplates various input embeddings);
receiving trait data comprising statements pre-tagged with classifications (figs.2-3 show pre-tagged with traits outputs for demonstrations);
receiving trait-data embeddings that include embeddings of the statements (p.3: “Retrieval-based Demonstration Selection”: embeddings for additional data from the demonstration pool are received for comparison, the demonstration pool comprising various trait data (see fig.2-3: various tags, names, data types, assertion types, error types, etc.) and corresponding embeddings according to p.3);
identifying similar statements by comparing the input-content embedding with the trait-data embeddings, wherein the similar statements are the statements that are similar to the input content (p.3: Retrieval-based Demonstration Selection: similar trait data is identified by comparing features of the received demonstration data with the input, the similar trait data being subsets identified by the two projection techniques as being similar);
generating a prompt including the input content and the identified similar statements (fig.1: demonstrations are combined with the query via a template to generate the final prompt, see p.3: “Template Selection”);
providing the prompt to the language model (fig.1: the prompt is provided to Codex for inference); and
receiving, from the language model in response to the prompt, an output payload including a classification of the input content (ibid: an output is received from Codex for evaluation).
Nashid does not disclose: requesting an embedding for the input content; wherein the receiving is in response to the request.
Kats discloses: requesting an embedding for the input content; wherein the receiving is in response to the request (fig.3:310-340, c.7¶2: requesting code completion information based on code query).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the system of Nashid by incorporating the code querying service of Kats. Both concern the art of code completion, code querying, and software development, and the incorporation would have, according to Kats, improved efficiency of code querying (c.1 ¶1-2)
For claim 18, Nashid modified by Kats discloses the system of claim 17, as described above. Nashid further discloses: wherein the classification of the input content is one of the pre-tagged classifications of the similar statements (fig.1, figs.2-3: output of prompt would be classifications similar to the input, e.g., assert statement type, bug repair statement type, etc.).
For claim 19, Nashid modified by Kats discloses the system of claim 17, as described above. Nashid further discloses: wherein the prompt does not include statements, in the trait data, that are not identified as the similar statements (fig.1, p.3: “Retrieval-based Demonstration Selection”: closest data is selected and the remainder of the dataset not included in the prompt).
For claim 20, Nashid modified by Kats discloses the system of claim 19, as described above. Nashid further discloses: wherein identifying the similar statements by comparing the trait-data embeddings with the input-content embedding comprises:
generating a ranked list of statements based on a similarity of the trait-data embeddings to the input-content embedding (§II.B¶3 (p.2), p.3: “Embedding (SRoBERTa)”: top-K, top-N, hence, generating a ranked list based on similarity); and
selecting a top N number of statements, from the ranked list, as the similar statements (ibid).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Eisenschlos (US 20240394545 A1) discloses a technique of few-shot in context learning.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LIANG LI whose telephone number is (303)297-4263. The examiner can normally be reached Mon-Fri 9-12p, 3-11p MT (11-2p, 5-1a ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. The examiner is available for interviews Mon-Fri 6-11a, 2-7p MT (8-1p, 4-9p ET).
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Jennifer Welch can be reached on (571)272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center or Private PAIR to authorized users only. Should you have questions about access to Patent Center or the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/LIANG LI/
Primary examiner AU 2143
Prosecution Insights