DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 26-27 are objected to because of the following informalities: Claims 26-27 both depend on claim 4, but should be depending on claim 25. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 22-38 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 22-38 all depend on canceled claims which has been canceled and it is indefinite which claim they are dependent to. The applicant is requested to amend the dependency of the claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 21-40 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim(s) 21 and 39 recite(s):
A computer-implemented method, comprising: receiving a docstring representing natural language text indicating a programming result;
generating, using a machine learning model and based on the docstring, computer code samples;
identifying computer code samples that produce candidate results associated with the programming result;
computing functional scores for each of the identified computer code samples;
verifying at least one of the identified computer code samples based on the functional scores;
outputting the at least one verified identified computer code sample / and the generated natural language text;
and fine-tuning the trained machine learning model based on the at least one verified identified computer code sample.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 21 is a method
Yes. Claim 39 is a system
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
The limitation of "generating", as drafted in #2 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a machine learning model", nothing in the claim element precludes the step from being performed by a person on paper.
The limitation of "identifying", as drafted in #3 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
The limitation of "computing", as drafted in #4 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
The limitation of "verifying", as drafted in #5 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The "fine-tuning" limitation in #7 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "fine-tuning" in the context of this claim encompasses merely training a machine learning model with data. See in the MPEP §§2106.05(f).
The "receiving" limitations in #1 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "receiving" in the context of this claim encompasses data transmission. See in the MPEP §§ 2106.05(g).
The "outputting" limitations in #6 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "outputting" in the context of this claim encompasses data transmission. See in the MPEP §§ 2106.05(g).
Additionally, the claims recite the following additional element:
at least one memory,
at least one processor
The element that is recited in the claims are stated at a high level of generality (i.e. as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using generic computer component. See the MPEP §§ 2106.05(f). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limitation on practicing the abstract idea(s).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Additionally, with regards to #1 and 6 above, per MPEP 2106.05(d)(ll), the courts have recognized the following computer function(s) as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity:
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information);
Claim(s) 22 recite(s):
wherein the verifying is performed in a testing environment associated with the machine learning model
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 22 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The "verifying is performed" limitation in #8 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "performed in a testing environment" in the context of this claim encompasses merely executing code within an environment. See in the MPEP §§2106.05(f).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 23 recite(s):
wherein each of the code samples are further verified based on at least one unit test,
the at least one unit test being generated by the machine learning model
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 23 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
The limitation of "further verified", as drafted in #9 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
The limitation of "unit test being generated", as drafted in #10 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 24 recite(s):
further comprising outputting natural language text with the at least one verified identified computer code sample.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 24 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The "outputting" limitations in #11 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "outputting" in the context of this claim encompasses mere data transmission. See in the MPEP §§ 2106.05(g).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Additionally, with regards to #11 above, per MPEP 2106.05(d)(ll), the courts have recognized the following computer function(s) as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity:
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information);
Claim(s) 25 recite(s):
wherein verifying at least one of the identified computer code samples further includes evaluating each of the identified computer code samples based on a time-related threshold.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 5 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
The limitation of "evaluating", as drafted in #12 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 26 recite(s):
wherein the machine learning model is further fine- tuned based on the evaluated computer code samples.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 6 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The "fine-tuned" limitation in #13 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "further fine-tuned" in the context of this claim encompasses merely training a machine learning model. See in the MPEP §§2106.05(f).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 27 recite(s):
wherein the time-related threshold is used to classify each of the code samples into different categories.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 7 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
The limitation of "classify", as drafted in #14 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 28 recite(s):
wherein identifying computer code samples comprises identifying at least one of the computer code samples that passes a unit test
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 8 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
The limitation of "identifying", as drafted in #15 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 29 recite(s):
wherein each of the generated computer code samples is associated with at least one text token or at least one whitespace token
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 29 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The limitations in #16 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "associated with" in the context of this claim encompasses merely associating data together. See in the MPEP §§ 2106.05(g).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Additionally, with regards to #15 above, per MPEP 2106.05(d)(ll), the courts have recognized the following computer function(s) as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity:
Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;
Claim(s) 30 recite(s):
outputting the candidate results associated with each verified identified computer code sample.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 30 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The "outputting" limitations in #17 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "outputting" in the context of this claim encompasses mere data transmission. See in the MPEP §§ 2106.05(g).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Additionally, with regards to #17 above, per MPEP 2106.05(d)(ll), the courts have recognized the following computer function(s) as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity:
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information);
Claim(s) 31 recite(s):
wherein the machine learning model is further fine- tuned based on at least one of a public web source or a software repository.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 31 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The limitation in #18 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "fine-tuned" in the context of this claim encompasses merely training a machine learning model. See in the MPEP §§2106.05(f).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 32 recite(s):
wherein the machine learning model is fine-tuned based on a set of training problems constructed from examples within the at least one public web source or software repository.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 12 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The limitation in #19 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "fine-tuned" in the context of this claim encompasses merely training a machine learning model. See in the MPEP §§2106.05(f).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 33 recite(s):
wherein identifying computer code samples is based on a mean-log probability.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 33 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
The limitation of "identifying", as drafted in #20 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 34 recite(s):
compiling the verified identified computer code samples;
transmitting the verified identified computer code samples to a recipient device;
storing the verified identified computer code samples;
and re-executing the verified identified computer code samples.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 34 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The "compiling" limitation in #21 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "compiling" in the context of this claim encompasses merely compiling code. See in the MPEP §§2106.05(f).
The "re-executing" limitation in #24 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "re-executing" in the context of this claim encompasses merely executing code snippets. See in the MPEP §§2106.05(f).
The "transmitting" limitations in #22 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "transmitting" in the context of this claim encompasses mere data transmission. See in the MPEP §§ 2106.05(g).
The "storing" limitations in #23 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "storing" in the context of this claim encompasses mere storing information into memory. See in the MPEP §§ 2106.05(g).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Additionally, with regards to #22-23 above, per MPEP 2106.05(d)(ll), the courts have recognized the following computer function(s) as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity:
Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;
Claim(s) 35 recite(s):
generating natural language text associated with the verified identified computer code samples, wherein the generated natural language text includes a definition of a function, method, class, or module associated with the verified identified computer code samples.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 35 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
The limitation of "generating", as drafted in #25 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 36 recite(s):
wherein the machine learning model is developed by applying training data comprising annotated computer code to a precursor model, the precursor model comprising a machine learning model trained on natural language prompts.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 36 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The limitation in #26 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "developed" in the context of this claim encompasses merely training a machine learning model. See in the MPEP §§2106.05(f).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 37 recite(s):
wherein the machine learning model generates training data based on a result of the computing of the functional scores,
wherein the machine learning model is further trained using the generated training data.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 37 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
The limitation of "generates", as drafted in #27 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The "trained" limitation in #28 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "trained" in the context of this claim encompasses merely training a machine learning model. See in the MPEP §§2106.05(f).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 38 recite(s):
wherein the machine learning model comprises a plurality of layers, at least one of the layers having a transformer decoder architecture.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 38 is a method
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The limitation in #29 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "comprises a plurality of layers" in the context of this claim encompasses merely code infrastructure. See in the MPEP §§2106.05(f).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Claim(s) 40 recite(s):
A networked device comprising one or more processors to perform operations comprising: receiving a docstring representing natural language text specifying a programming result;
generating, using a machine learning model and based on the docstring, computer code samples;
causing each of the computer code samples to be executed in a testing environment associated with the machine learning model,
wherein each of the computer code samples are evaluated based on a unit test, the unit test being generated by the machine learning model;
identifying, based on a result of the executing in the testing environment, computer code samples that produce candidate results associated with the programming result;
computing functional scores for each of the identified computer code samples;
verifying at least one of the identified computer code samples based on the functional scores;
outputting the at least one verified identified computer code sample;
and fine-tuning the machine learning model based on the at least one verified identified computer code sample.
Step 1: are the claims to a process, machine, manufacture, or a composition of matter?
Yes. Claim 40 is a machine
Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes: (an) abstract idea(s).
The limitation of "generating", as drafted in #31 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a machine learning model", nothing in the claim element precludes the step from being performed by a person on paper.
The limitation, as drafted in #33 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a machine learning model", nothing in the claim element precludes the step from being performed by a person on paper.
The limitation of "identifying", as drafted in #34 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
The limitation of "computing", as drafted in #35 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
The limitation of "verifying", as drafted in #36 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a computer-implemented method", nothing in the claim element precludes the step from being performed by a person on paper.
Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
The "causing" limitation in #32 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "executing" in the context of this claim encompasses merely running code. See in the MPEP §§2106.05(f).
The "fine-tuning" limitation in #38 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "fine-tuning" in the context of this claim encompasses merely training a machine learning model with data. See in the MPEP §§2106.05(f).
The "receiving" limitations in #30 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "receiving" in the context of this claim encompasses data transmission. See in the MPEP §§ 2106.05(g).
The "outputting" limitations in #37 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "outputting" in the context of this claim encompasses data transmission. See in the MPEP §§ 2106.05(g).
Additionally, the claims recite the following additional element:
a network device,
one or more processors
The element that is recited in the claims are stated at a high level of generality (i.e. as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using generic computer component. See the MPEP §§ 2106.05(f). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limitation on practicing the abstract idea(s).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step.
Additionally, with regards to #30 and 37 above, per MPEP 2106.05(d)(ll), the courts have recognized the following computer function(s) as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity:
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information);
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 21-22, 24-27, 29-35, and 39 is/are rejected under 35 U.S.C. 102(a)(1) as being unpatentable in view of US 20200097261 A1 (Hereinafter referred to as Smith) and US 20220244937 A1 (Hereinafter referred to as Prasad).
Regarding claim 21, Smith teaches:
A computer-implemented method, comprising:
receiving a docstring representing natural language text indicating a programming result (Para. [99-102], Smith shows " In some embodiments, a match of a keyword increases a score of a result and the matching of features may also lead to a higher score. Code snippets may be ranked by score with higher scores indicating a higher rank and greater prominence in the search results... The keywords associated with code snippets in code storage may be automatically extracted from codebases and code libraries.. the keywords may be generated automatically based on the code snippet and other code in the code base, without referring to external text. Moreover, keywords may also be generated based on documentation, such as docstrings, written in the source code itself." Para. [97], Smith shows “the keywords may comprise the text of a function name, such as a function definition “def getHttpResponse” being converted to keywords “get http response.” In one embodiment, the keywords received are natural language keywords such as “make HTTP request.” Examiner notes the citation above shows receiving code snippets and generating the keywords from the code snippets received which include docstrings used to generate keywords. The keywords themselves indicating toward a programming result like “get http response”);
generating, using a machine learning model and based on the docstring, computer code samples (Para. [96-97], Smith shows "the code completion system identifies relevant code snippets based on both the features of the code that were identified and also the keywords that were received from the user in step 501b. Also, rather than identifying relevant code snippets from code storage, the code completion system 342 may instead generate the code snippet on the fly by using machine learning. When the code snippet is identified or generated, both the features of the code and the keywords may be inputs to the machine learning model 200 that performs the inference and returns the result.. In one embodiment, the keywords may comprise words in code comments that are near the cursor. In one embodiment, the keywords may comprise the text of a function name, such as a function definition “def getHttpResponse” being converted to keywords “get http response.” In one embodiment, the keywords received are natural language keywords such as 'make HTTP request.' " Examiner notes the above citations show using features and keywords received in order to generate code snippets.);
identifying computer code samples that produce candidate results associated with the programming result (Para. [96-97], Smith shows "The keywords may be used by the machine learning model 200 for identifying code snippets to return. At step 503b, the code completion system 342 identifies relevant code snippets from code storage based on features, in the same manner as in step 503. However, in step 503b, the code completion system identifies relevant code snippets based on both the features of the code that were identified and also the keywords that were received from the user in step 501b.. In one embodiment, the keywords may comprise words in code comments that are near the cursor. In one embodiment, the keywords may comprise the text of a function name, such as a function definition “def getHttpResponse” being converted to keywords “get http response.” In one embodiment, the keywords received are natural language keywords such as 'make HTTP request.'" Examiner notes Para. [155], Smith shows "using the features to search for code snippets in a team or local data set or to generate code snippets based on one or more models trained using the team or local data set; identifying or generating one or more candidate code snippets" Examiner notes the above citations show identifying candidate code snippets that are generated according to keywords indicating a programming result.);
outputting the at least one verified identified computer code sample (Para. [155], Smith shows "identifying or generating one or more candidate code snippets; displaying the one or more candidate code snippets to the programmer; receiving a selection of one of the candidate code snippets; inputting the selected code snippet into the source code." Examiner notes in the prior citation above smith teaches ranking the code snippets by score. The above citation shows using these ranked code snippets to output into the source code);
Smith does not implicitly disclose: computing functional scores for each of the identified computer code samples
verifying at least one of the identified computer code samples based on the functional scores
and fine-tuning the trained machine learning model based on the at least one verified identified computer code sample
However, in the analogous art of automated software code modification, Prasad teaches:
computing functional scores for each of the identified computer code samples (Para. [27], Prasad shows "In some implementations, the developer system causes the modified software code to be implemented in production when a confidence score associated with identifying the portion of the software code and/or associated with modifying the portion of the software code satisfies a confidence score threshold" Examiner notes the above citation shows every piece of modified code receives a confidence score to score the functionality of the modified code before implementation after it reaches a confidence threshold);
verifying at least one of the identified computer code samples based on the functional scores (Para. [27], Prasad shows "In some implementations, the developer system causes the modified software code to be implemented in production when a confidence score associated with identifying the portion of the software code and/or associated with modifying the portion of the software code satisfies a confidence score threshold" Examiner notes the above citation shows using a confidence score to score the functionality of the modified code. Verifying that the code will work because of the confidence score behind it);
and fine-tuning the trained machine learning model based on the at least one verified identified computer code sample (Para. [29], Prasad shows "the one or more actions include receiving feedback associated with implementing the modified software code and updating the modified software code based on the feedback. For example, the developer system may receive feedback associated with implementing the modified software code based on providing the modified software code for display to the user. In some implementations, the feedback includes additional requirement data associated with the modified software code. The developer system may further modify the modified software code based on the additional requirement data. In some implementations, the developer system further modifies the modified software code in a manner similar to that described above." Para. [32], Prasad shows "the one or more actions include the developer system retraining the machine learning model, the code locator model, and/or the code developer model based on the modified software code. The developer system may utilize the modified software code as additional training data for retraining the machine learning model, the code locator model, and/or the code developer model, thereby increasing the quantity of training data available for training the machine learning model, the code locator model, and/or the code developer model." Examiner notes the above citation shows receiving feedback, which can be considered verifying code functionality, and using the generated modified code to retrain the model").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Prasad into the teachings of Smith to implement “computing functional scores for each of the identified computer code samples, verifying at least one of the identified computer code samples based on the functional scores, and fine-tuning the trained machine learning model based on the at least one verified identified computer code sample”. The modification would have been obvious as one of ordinary skill in the art would be motivated to conserve computing resources and train machine learning models to identify and properly correct the lines of software code (Prasad, Para. [13]).
Regarding claim 22, Smith as modified teaches:
is performed in a testing environment associated with the machine learning model (Fig. 3; Para. [60-61], Smith shows "FIG. 3 illustrates an exemplary system for software development. Source code 310 may be provided and edited in a programming environment 300. The programming environment may allow interactive editing of the source code 310 by a user, such as a programmer. A programming environment may include an editor 302 and an interface 304. The editor 302 may provide for the developing, such as writing and editing, of source code 310. The interface 304 may present a human viewable or usable interface for using the editor 302... A compiler or interpreter 320 may compile the code 310 into executable instructions or an intermediate representation, or interpret the source code 310 for execution. The compiler/interpreter 320 may comprise a namespace 322 that can be used to store symbols, such as identifiers and types, and to allow for name resolution 330. In some embodiments, the compiler/interpreter 320 may comprise a scanner 324, parser 326, semantic checker 328, name resolver 330, and code generator 332... Code generator 332 may translate the parse tree, or other intermediate representation of the source code, into a target language. The target language may be executable instructions, such as a binary executable, or an intermediate language that may be interpreted for execution. In an execution environment 370, code may be executed, such as for testing or production." Para. [65], "Code completion may provide the programmer with one or more options to automatically insert, complete, or modify code in the programming environment. For example, as the programmer types code, the code completion system 342 may predict and present options for code snippets to insert into the source code 310 displayed in the editor 302. In response to user selection, the code completion system 342 may insert the code snippet into the source code 310 in the editor 302. In other embodiments, the code completion system 342 may predict and insert the code snippets into the source 310 automatically without user input or selection." Examiner notes in Fig. 3 it shows the code completion system executes within the execution environment which in the citation shown above can be the testing or production environment. The machine learning model is included in the co-pilot system which also executes in the execution environment.)
Smith does not explicitly disclose:
wherein the verifying
However, in the analogous art of automated software code modification, Prasad teaches:
wherein the verifying (Para. [29], Prasad shows "the one or more actions include receiving feedback associated with implementing the modified software code and updating the modified software code based on the feedback. For example, the developer system may receive feedback associated with implementing the modified software code based on providing the modified software code for display to the user. In some implementations, the feedback includes additional requirement data associated with the modified software code. The developer system may further modify the modified software code based on the additional requirement data. In some implementations, the developer system further modifies the modified software code in a manner similar to that described above." Examiner notes the above citation shows receiving feedback about modified code, this is a type of verification.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Prasad into the teachings of Smith to implement “computing functional scores for each of the identified computer code samples, verifying at least one of the identified computer code samples based on the functional scores, and fine-tuning the trained machine learning model based on the at least one verified identified computer code sample”. The modification would have been obvious as one of ordinary skill in the art would be motivated to conserve computing resources and save developers time by automatically implementing code when confident (Prasad, Para. [27]).
Regarding claim 24, Smith as modified teaches:
further comprising outputting natural language text with the at least one verified identified computer code sample (Para. [155], Smith shows " identifying or generating one or more candidate code snippets; displaying the one or more candidate code snippets to the programmer; receiving a selection of one of the candidate code snippets; inputting the selected code snippet into the source code." Examiner notes in the prior citation above smith teaches ranking the code snippets by score. The above citation shows using these ranked code snippets to input into source code Para. [136], Smith shows "FIG. 8C illustrates an exemplary user interface 820 for presenting code completion options using natural language. A code completion interface component 826 is illustrated containing multiple options for code completions 828, which are presented in natural language. After receiving a selection from the user, a code snippet corresponding to the natural language description may be selected and inserted into the programmer's code. In other embodiments, after selection, another list is shown of actual code snippets corresponding to the natural language text. For example, there may be several code snippets available that accomplish the same described text. A selection may be received from the user in the new list, and the selected code snippet may be inserted." Examiner notes the above citation shows displaying natural text along with a code completion option for a user to see)
Regarding claim 25, Smith as modified teaches
wherein verifying at least one of the identified computer code samples further includes evaluating each of the identified computer code samples based on a time-related threshold. (Para. [79], Smith shows "The ground-truth value for the input object may be set to 1, or positive, if the code completion was presented by the code completion system 342 and received a positive input from the user, whereas the ground-truth value for the input object may be set to 0, or negative, if the code completion was presented by the code completion system and received a negative input from the user. Examples of positive inputs include detecting selection of the code snippet by the user, hovering or focusing on the code snippet by the user, long dwell time on the code snippet by the user, skipping higher ranked code snippets to select this code snippet, and so on. Examples of negative inputs include detecting that the user did not select the code snippet and skipping this code snippet to select a lower ranked code snippet. In this manner, the code completion system 342 may adjust the code snippets that it suggests based on past usage of code completion system 342 by the programmer, by programmers on the same team, or by other programmers in general." Examiner notes the above citation shows the machine learning algorithm bases scores of identified code samples on criteria that includes time spent looking at the code snippet offered. Verifying the code snippet by having more positive data correlated to the code snippets)
Regarding claim 26, Smith as modified teaches claim 25 as cited above, but does not disclose
wherein the machine learning model is further fine- tuned based on the evaluated computer code samples
However, in the analogous art of automated software code modification, Prasad teaches:
wherein the machine learning model is further fine- tuned based on the evaluated computer code samples (Para. [31-32], Prasad shows "The developer system may provide the modified software code to a software development and operations environment for testing and may obtain a result of the testing. The developer system may determine one or more modifications associated with the software code based on the result of the testing.. The developer system may utilize the modified software code as additional training data for retraining the machine learning model, the code locator model, and/or the code developer model, thereby increasing the quantity of training data available for training the machine learning model, the code locator model, and/or the code developer model.")
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Prasad into the teachings of Smith to implement “wherein the machine learning model is further fine- tuned based on the evaluated computer code samples”. The modification would have been obvious as one of ordinary skill in the art would be motivated to conserve computing resources by increasing the efficiency and quality of the machine learning model (Prasad, Para. [32]).
Regarding claim 27, Smith as modified teaches:
wherein the time-related threshold is used to classify each of the code samples into different categories. (Para. [79], Smith shows "The ground-truth value for the input object may be set to 1, or positive, if the code completion was presented by the code completion system 342 and received a positive input from the user, whereas the ground-truth value for the input object may be set to 0, or negative, if the code completion was presented by the code completion system and received a negative input from the user. Examples of positive inputs include detecting selection of the code snippet by the user, hovering or focusing on the code snippet by the user, long dwell time on the code snippet by the user, skipping higher ranked code snippets to select this code snippet, and so on. Examples of negative inputs include detecting that the user did not select the code snippet and skipping this code snippet to select a lower ranked code snippet." Examiner notes the above citation shows that long period of time looking at a code snippet is classified as a positive input while skipping or low period of time looking at a code snippet is classified as a negative input.)
Regarding claim 29, Smith as modified teaches:
wherein each of the generated computer code samples is associated with at least one text token or at least one whitespace token (Para. [76], Smith shows "Code snippets may comprise a single token, such as a single keyword, identifier, type, or other token. In other embodiments, code snippets for code completion may comprise multiple tokens. In some embodiments, code snippets may comprise a plurality of lines of code. For example, code snippets may include a function definition, a programming pattern, a programming idiom, a function call, an API function call, a chained attribute expression, a binary expression, a Boolean expression, a list, set, or dictionary comprehension, a variable assignment, or other multi-token code snippets." Para. [80], Smith shows "he code snippets may be generated by deterministic, hard-coded rules. In other embodiments, the relevant code snippets are generated by machine learning model 200 by using a learning algorithm, such as supervised or unsupervised learning." Para. [86], Smith shows "In one embodiment, the selected candidate code snippet includes whitespace, and the whitespace is customized for the programmer. Different programming teams may use different norms for how they use whitespace in source code, such as “if(foo)” or “if (foo)”, where the first example has no whitespace between the if, parentheses, and foo tokens and the second example has whitespace between each of these tokens. The code completion system 342 may customize the whitespace of suggested code completions, predictive edits, or keyword snippet generation according to the preferences of the team or programmer." Examiner notes in the citations above they show generated code snippets that contain both white space and a plurality of other tokens that may include text.)
Regarding claim 30, Smith as modified teaches:
further comprising outputting the candidate results associated with each verified identified computer code sample. (Para. [155], Smith shows "identifying or generating one or more candidate code snippets; displaying the one or more candidate code snippets to the programmer; receiving a selection of one of the candidate code snippets; inputting the selected code snippet into the source code." Examiner notes the above citation shows displaying candidate results for a user to pick from.)
Regarding claim 31, Smith as modified teaches:
based on at least one of a public web source or a software repository (Para. [71], Smith shows " the code storage accessed is an external code storage 110, 111, which is globally and publicly accessible. For example, external code storage 110, 111 may comprise code snippets collected from online communities, apps, or websites for sharing code or expert knowledge, examples being Stack Overflow or Github. A web scraper may be used to access and download snippets from an online community, app, or website to store the snippets in a code storage 153." Para. [78], Smith shows "relevant code snippets are identified from the code storage based on features, such as any of the features described above, for example but not limited to, in step 501. In one embodiment, the relevant code snippets are determined by machine learning model 200 using deterministic, hard-coded rules. For example, in one embodiment, typing a particular token for a class automatically selects as potential code completions the sub-methods or variables of the class. In another example, typing the name of a function automatically selects as a potential code completion a code snippet representing a pattern for making a call to the class." Para. [79], Smith shows "In other embodiments, the relevant code snippets are determined by machine learning model 200 by using a learning algorithm, such as supervised or unsupervised learning. Any of the machine learning methods described herein may be used. The machine learning algorithm may be trained based on prior code examples of this programmer, other programmers on the team, or other programmers not on the team, where a code completion was suggested by code completion system 342." Examiner notes the above citations show extracting code snippets from public websites and storing them in code storage, using the stored code snippets for further training of the machine learning model).
Smith does not explicitly disclose:
wherein the machine learning model is further fine- tuned
However, in the analogous art of automated software code modification, Prasad teaches:
wherein the machine learning model is further fine- tuned (Para. [32], Prasad shows “The developer system may utilize the modified software code as additional training data for retraining the machine learning model, the code locator model, and/or the code developer model, thereby increasing the quantity of training data available for training the machine learning model, the code locator model, and/or the code developer model.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Prasad into the teachings of Smith to implement “wherein the machine learning model is further fine- tuned”. The modification would have been obvious as one of ordinary skill in the art would be motivated to conserve computing resources by increasing the efficiency and quality of the machine learning model (Prasad, Para. [32]).
Regarding claim 32, Smith as modified teaches:
based on a set of training problems constructed from examples within the at least one public web source or software repository. (Para. [74], Smith shows "Code storage may store may than just code and may store text and multimedia related to the code. In some embodiments, code storage may include documentation, questions and answers, tutorials, and other information about code. For example, code storage may store code downloaded from a crowd-sourced website like StackOverflow and associated questions and answers. Code storage may include stored links between code and associated content." Para. [78], Smith shows "relevant code snippets are identified from the code storage based on features, such as any of the features described above, for example but not limited to, in step 501. In one embodiment, the relevant code snippets are determined by machine learning model 200 using deterministic, hard-coded rules. For example, in one embodiment, typing a particular token for a class automatically selects as potential code completions the sub-methods or variables of the class. In another example, typing the name of a function automatically selects as a potential code completion a code snippet representing a pattern for making a call to the class." Para. [79], Smith shows "In other embodiments, the relevant code snippets are determined by machine learning model 200 by using a learning algorithm, such as supervised or unsupervised learning. Any of the machine learning methods described herein may be used. The machine learning algorithm may be trained based on prior code examples of this programmer, other programmers on the team, or other programmers not on the team, where a code completion was suggested by code completion system 342." Examiner notes the above citation shows training the machine ;earning model based on examples stored within code storage and as shown above this includes questions / answers, tutorials, and other types of documentation.)
Smith does not explicitly disclose:
wherein the machine learning model is fine-tuned
However, in the analogous art of automated software code modification, Prasad teaches:
wherein the machine learning model is fine- tuned (Para. [32], Prasad shows “The developer system may utilize the modified software code as additional training data for retraining the machine learning model, the code locator model, and/or the code developer model, thereby increasing the quantity of training data available for training the machine learning model, the code locator model, and/or the code developer model.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Prasad into the teachings of Smith to implement “wherein the machine learning model is fine- tuned”. The modification would have been obvious as one of ordinary skill in the art would be motivated to conserve computing resources by increasing the efficiency and quality of the machine learning model (Prasad, Para. [32]).
Regarding claim 33, Smith as modified teaches:
wherein identifying computer code samples is based on a mean-log probability (Para. [56], Smith shows "The lexical language model may be used to model the probability of a portion of text in a language. In an embodiment, the lexical language model may model the probability of text in a programming language. The probabilities may be computed by statistical analysis of pre-existing examples. In some embodiments, the lexical language model is entirely statistical and does not encode grammar rules of the language." Examiner notes the above citation shows the machine learning model may contain a lexicon language model which is capable of mean log probability as it uses statistical analysis to predict text probability.)
Regarding claim 34, Smith as modified teaches:
further comprising: compiling the verified identified computer code samples (Para. [61], Smith shows "A compiler or interpreter 320 may compile the code 310 into executable instructions or an intermediate representation, or interpret the source code 310 for execution. The compiler/interpreter 320 may comprise a namespace 322 that can be used to store symbols, such as identifiers and types, and to allow for name resolution 330. In some embodiments, the compiler/interpreter 320 may comprise a scanner 324, parser 326, semantic checker 328, name resolver 330, and code generator 332. Scanner 324 may accept as input the source code 310 and split expressions and language statements into tokens that can be processed by the parser 326 to determine the grammatical structure of a program. A token may be a single element of a programming language such as a constant, identifier, operator, separator, reserved word, or other element. In some embodiments, a token is atomic and is the smallest semantic unit of a programming language, such that the token cannot be broken down further into units with semantic meaning in the language. The parser 326 may parse the tokens and organize them according to a grammar of a programming language. In some embodiments, parser 326 builds a parse tree. Semantic checker 328 may perform semantic checking of a computer program and may identify and throw errors that are semantic in nature. The name resolver 330 may resolve names in the parse tree to elements of the namespace 322. Code generator 332 may translate the parse tree, or other intermediate representation of the source code, into a target language. The target language may be executable instructions, such as a binary executable, or an intermediate language that may be interpreted for execution. In an execution environment 370, code may be executed, such as for testing or production.");
transmitting the verified identified computer code samples to a recipient device (Para. [155], Smith shows "identifying or generating one or more candidate code snippets; displaying the one or more candidate code snippets to the programmer; receiving a selection of one of the candidate code snippets; inputting the selected code snippet into the source code." Examiner notes the above citation shows displaying the candidate results which would need to be transmitted to the recipient device to view or input into source code);
storing the verified identified computer code samples (Para. [113], Smith shows "A ranking algorithm may be used that incorporates one or several ranking factors. In one embodiment, one of the factors used in ranking is which code storage location the code snippet appears in. In one embodiment, personal code storage is scored the most highly, team code storage is scored the second mostly highly, and external code storage is scored the least highly. In an embodiment, code snippets are scored based on the number of occurrences or the context of occurrence in different codebases. For example, in an embodiment, the appearance of a code snippet in several different codebases, related to different projects, increases its score. Moreover, a code snippet appearing in a more commonly downloaded, used, or reputationally-scored codebase may increase the score of the code snippet. In further embodiments, a code snippet may be scored based on the code surrounding the programmer's current cursor position. For example, code snippets may be scored based on appearing in a similar context in their codebase or usage context or may be scored based on having similar code to the programmer's current cursor position. In further embodiments, a code snippet may be scored based on click data, such as how often the code snippet has been selected by users. The click data may be specific to the current user or may be across a group or all users." Examiner notes that the code samples are stored based on score showing they are verified and being stored within the code storage shows they were identified and saved.);
and re-executing the verified identified computer code samples (Fig. 8J; Para. [146], Smith shows "FIG. 8J illustrates a shell, terminal, or operating command line 890 where code completion system 342 may operate. All of the functionality described herein applies to shells, terminals, and command lines. Code 894 is received on the command line. A code completion interface 896 is displayed with a plurality of potential code completions 898. Upon selection, the selected code completion may be added to the shell and, optionally, executed." Examiner notes the citation above shows executing optionally upon selection after receiving the verified identified code sample.)
Regarding claim 35, Smith as modified teaches:
further comprising generating natural language text associated with the verified identified computer code samples, wherein the generated natural language text includes a definition of a function, method, class, or module associated with the verified identified computer code samples. (Para. [76], Smith shows " For example, code snippets may include a function definition, a programming pattern, a programming idiom, a function call, an API function call, a chained attribute expression, a binary expression, a Boolean expression, a list, set, or dictionary comprehension, a variable assignment, or other multi-token code snippets." Para. [136], Smith shows "FIG. 8C illustrates an exemplary user interface 820 for presenting code completion options using natural language. A code completion interface component 826 is illustrated containing multiple options for code completions 828, which are presented in natural language. After receiving a selection from the user, a code snippet corresponding to the natural language description may be selected and inserted into the programmer's code. In other embodiments, after selection, another list is shown of actual code snippets corresponding to the natural language text. For example, there may be several code snippets available that accomplish the same described text. A selection may be received from the user in the new list, and the selected code snippet may be inserted." Examiner notes the above citation shows displaying natural text along with a code completion option for a user to see. Para. [107], Smith shows " In one embodiment, the keywords may comprise the text of a function name, such as a function definition “def getHttpResponse” being converted to keywords “get http response.” In one embodiment, the keywords received are natural language keywords such as “make HTTP request.” In one embodiment, the keywords received are natural language keywords having programming code syntax such as “makeHTTPRequest” or “make_http_request.” It may be desirable to allow the system to detect when the user does not know an exact entity name, such as for a variable or function, and is approximating it with natural language that looks like code. In an embodiment, the system may parse the keywords by splitting them into tokens based on interpreting the keywords using programming syntax. ”Examiner notes the citation above shows generating natural language text to provide keywords of names or definitions of functions)
Regarding claim 39, Smith teaches:
A system comprising: at least one memory storing instructions; at least one processor configured to execute the instructions to perform operations comprising:
receiving a docstring representing natural language text specifying a programming result (Para. [99-102], Smith shows " In some embodiments, a match of a keyword increases a score of a result and the matching of features may also lead to a higher score. Code snippets may be ranked by score with higher scores indicating a higher rank and greater prominence in the search results... The keywords associated with code snippets in code storage may be automatically extracted from codebases and code libraries.. the keywords may be generated automatically based on the code snippet and other code in the code base, without referring to external text. Moreover, keywords may also be generated based on documentation, such as docstrings, written in the source code itself." Para. [97], Smith shows “the keywords may comprise the text of a function name, such as a function definition “def getHttpResponse” being converted to keywords “get http response.” In one embodiment, the keywords received are natural language keywords such as “make HTTP request.” Examiner notes the citation above shows receiving code snippets and generating the keywords from the code snippets received which include docstrings used to generate keywords. The keywords themselves indicating toward a programming result like “get http response”);
generating, using a machine learning model and based on the docstring, computer code samples (Para. [96-97], Smith shows "the code completion system identifies relevant code snippets based on both the features of the code that were identified and also the keywords that were received from the user in step 501b. Also, rather than identifying relevant code snippets from code storage, the code completion system 342 may instead generate the code snippet on the fly by using machine learning. When the code snippet is identified or generated, both the features of the code and the keywords may be inputs to the machine learning model 200 that performs the inference and returns the result.. In one embodiment, the keywords may comprise words in code comments that are near the cursor. In one embodiment, the keywords may comprise the text of a function name, such as a function definition “def getHttpResponse” being converted to keywords “get http response.” In one embodiment, the keywords received are natural language keywords such as 'make HTTP request.' " Examiner notes the above citations show using features and keywords received in order to generate code snippets.);
identifying computer code samples that produce candidate results associated with the programming result (Para. [96-97], Smith shows "The keywords may be used by the machine learning model 200 for identifying code snippets to return. At step 503b, the code completion system 342 identifies relevant code snippets from code storage based on features, in the same manner as in step 503. However, in step 503b, the code completion system identifies relevant code snippets based on both the features of the code that were identified and also the keywords that were received from the user in step 501b.. In one embodiment, the keywords may comprise words in code comments that are near the cursor. In one embodiment, the keywords may comprise the text of a function name, such as a function definition “def getHttpResponse” being converted to keywords “get http response.” In one embodiment, the keywords received are natural language keywords such as 'make HTTP request.' " Para. [155], Smith shows "using the features to search for code snippets in a team or local data set or to generate code snippets based on one or more models trained using the team or local data set; identifying or generating one or more candidate code snippets" Examiner notes the above citations show identifying candidate code snippets that are generated according to keywords indicating a programming result.);
generating, using the machine learning model, a natural language text associated with the identified computer code samples (Para. [136], Smith shows "FIG. 8C illustrates an exemplary user interface 820 for presenting code completion options using natural language. A code completion interface component 826 is illustrated containing multiple options for code completions 828, which are presented in natural language. After receiving a selection from the user, a code snippet corresponding to the natural language description may be selected and inserted into the programmer's code. In other embodiments, after selection, another list is shown of actual code snippets corresponding to the natural language text. For example, there may be several code snippets available that accomplish the same described text. A selection may be received from the user in the new list, and the selected code snippet may be inserted." Para. [199], Smith shows "identifying or generating a natural language summary of each of the candidate code snippets; displaying the natural language summaries to the programmer; receiving a selection of one of the natural language summaries; identifying the code snippet that is associated with the selected natural language summary; inputting the associated code snippet." Examiner notes the above citations show generating natural language text alongside identified code samples.);
outputting the at least one verified identified computer code sample and the generated natural language text (Para. [155], Smith shows " identifying or generating one or more candidate code snippets; displaying the one or more candidate code snippets to the programmer; receiving a selection of one of the candidate code snippets; inputting the selected code snippet into the source code." Examiner notes in the prior citation above smith teaches ranking the code snippets by score. The above citation shows using these ranked code snippets to input into source code);
Smith does not implicitly disclose:
computing functional scores for each of the identified computer code samples
verifying at least one of the identified computer code samples based on the functional scores
and fine-tuning the trained machine learning model based on the at least one verified identified computer code sample
However, in the analogous art of automated software code modification, Prasad teaches:
computing a functional score for each of the identified computer code samples (Para. [27], Prasad shows "In some implementations, the developer system causes the modified software code to be implemented in production when a confidence score associated with identifying the portion of the software code and/or associated with modifying the portion of the software code satisfies a confidence score threshold" Examiner notes the above citation shows every piece of modified code receives a confidence score to score the functionality of the modified code before implementation after it reaches a confidence threshold);
verifying at least one of the identified computer code samples based on the functional scores (Para. [27], Prasad shows "In some implementations, the developer system causes the modified software code to be implemented in production when a confidence score associated with identifying the portion of the software code and/or associated with modifying the portion of the software code satisfies a confidence score threshold" Examiner notes the above citation shows using a confidence score to score the functionality of the modified code. Verifying that the code will work because of the confidence score behind it);
and fine-tuning the machine learning model based on the at least one verified identified computer code sample (Para. [29], Prasad shows "the one or more actions include receiving feedback associated with implementing the modified software code and updating the modified software code based on the feedback. For example, the developer system may receive feedback associated with implementing the modified software code based on providing the modified software code for display to the user. In some implementations, the feedback includes additional requirement data associated with the modified software code. The developer system may further modify the modified software code based on the additional requirement data. In some implementations, the developer system further modifies the modified software code in a manner similar to that described above." Para. [32], Prasad shows "the one or more actions include the developer system retraining the machine learning model, the code locator model, and/or the code developer model based on the modified software code. The developer system may utilize the modified software code as additional training data for retraining the machine learning model, the code locator model, and/or the code developer model, thereby increasing the quantity of training data available for training the machine learning model, the code locator model, and/or the code developer model." Examiner notes the above citation shows receiving feedback, which can be considered verifying code functionality, and using the generated modified code to retrain the model)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Prasad into the teachings of Smith to implement “computing functional scores for each of the identified computer code samples, verifying at least one of the identified computer code samples based on the functional scores, and fine-tuning the trained machine learning model based on the at least one verified identified computer code sample”. The modification would have been obvious as one of ordinary skill in the art would be motivated to conserve computing resources and train machine learning models to identify and properly correct the lines of software code (Prasad, Para. [13]).
Claim(s) 23 and 40 is/are rejected under 35 U.S.C. 103 as being unpatentable by US 20200097261 A1 (Hereinafter referred to as Smith) and US 20220308848 A1 (Hereinafter referred to as Prasad) in further view of US 20230333821 A1 (Hereinafter referred to as Cosgrove) and US 11899566 B1 (Hereinafter referred to as Singh).
Regarding claim 23, Smith teaches the independent claims as cited above, but does not disclose:
wherein each of the code samples are further verified based on at least one unit test,
the at least one unit test being generated by the machine learning model.
However, in the analogous art of automatically editing files, Cosgrove teaches:
wherein each of the code samples are further verified based on at least one unit test (Para. [87], Cosgrove shows "A first testing component 1106 verifies whether each code edit in the set 1104 correctly performs its functions, with respect to predetermined expectations of what constitutes the expected behavior of these functions. The first testing component 1106 can perform this function by applying a plurality of unit tests 1110 to each code edit in the set 1104. The first testing component 1106 concludes that a code edit under consideration correctly performs its functions when the code edit passes all of its unit tests 1110. Generally, unit testing involves: (1) identifying the operations performed by a code fragment, with the goal of breaking the code fragment into its smallest testable parts; (2) generating assertions that will be used to determine whether each identified operation produces correct or incorrect results; and (3) applying all of the unit tests to the code fragment under consideration. The first testing component 1106 produces output results 1108 that identify the subset of code edits (if any) that pass the above-described type of testing." Examiner notes the above citation shows evaluating components taken from the code by testing them),
In addition, in the analogous art of training machine learning models for automatic generation of test cases, Singh teaches:
the at least one unit test being generated by the machine learning model ([Abstract], Singh shows "Training and/or utilization of machine learning model(s) (e.g., neural network model(s)) in automatically generating test case(s) for source code.").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Cosgrove into the teachings of Smith to implement “wherein each of the code samples are further verified based on at least one unit test”. The modification would have been obvious as one of ordinary skill in the art would be motivated to identify if the identified component produces incorrect results (Cosgrove, Para. [87]).
Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Singh into the teachings of Smith to implement “the at least one unit test being generated by the machine learning model”. The modification would have been obvious as one of ordinary skill in the art would be motivated to ensure the quality of test cases and save developers time (Singh, Col. [1] lines [23-27]).
Regarding claim 40, Smith teaches:
A networked device comprising one or more processors to perform operations comprising:
receiving a docstring representing natural language text specifying a programming result (Para. [99-102], Smith shows " In some embodiments, a match of a keyword increases a score of a result and the matching of features may also lead to a higher score. Code snippets may be ranked by score with higher scores indicating a higher rank and greater prominence in the search results... The keywords associated with code snippets in code storage may be automatically extracted from codebases and code libraries.. the keywords may be generated automatically based on the code snippet and other code in the code base, without referring to external text. Moreover, keywords may also be generated based on documentation, such as docstrings, written in the source code itself." Para. [97], Smith shows “the keywords may comprise the text of a function name, such as a function definition “def getHttpResponse” being converted to keywords “get http response.” In one embodiment, the keywords received are natural language keywords such as “make HTTP request.” Examiner notes the citation above shows receiving code snippets and generating the keywords from the code snippets received which include docstrings used to generate keywords. The keywords themselves indicating toward a programming result like “get http response”);
generating, using a machine learning model and based on the docstring, computer code samples (Para. [96-97], Smith shows "the code completion system identifies relevant code snippets based on both the features of the code that were identified and also the keywords that were received from the user in step 501b. Also, rather than identifying relevant code snippets from code storage, the code completion system 342 may instead generate the code snippet on the fly by using machine learning. When the code snippet is identified or generated, both the features of the code and the keywords may be inputs to the machine learning model 200 that performs the inference and returns the result.. In one embodiment, the keywords may comprise words in code comments that are near the cursor. In one embodiment, the keywords may comprise the text of a function name, such as a function definition “def getHttpResponse” being converted to keywords “get http response.” In one embodiment, the keywords received are natural language keywords such as 'make HTTP request.' " Examiner notes the above citations show using features and keywords received in order to generate code snippets.);
causing each of the computer code samples to be executed in a testing environment associated with the machine learning model (Fig. 3; Para. [60-61], Smith shows "FIG. 3 illustrates an exemplary system for software development. Source code 310 may be provided and edited in a programming environment 300. The programming environment may allow interactive editing of the source code 310 by a user, such as a programmer. A programming environment may include an editor 302 and an interface 304. The editor 302 may provide for the developing, such as writing and editing, of source code 310. The interface 304 may present a human viewable or usable interface for using the editor 302... A compiler or interpreter 320 may compile the code 310 into executable instructions or an intermediate representation, or interpret the source code 310 for execution. The compiler/interpreter 320 may comprise a namespace 322 that can be used to store symbols, such as identifiers and types, and to allow for name resolution 330. In some embodiments, the compiler/interpreter 320 may comprise a scanner 324, parser 326, semantic checker 328, name resolver 330, and code generator 332... Code generator 332 may translate the parse tree, or other intermediate representation of the source code, into a target language. The target language may be executable instructions, such as a binary executable, or an intermediate language that may be interpreted for execution. In an execution environment 370, code may be executed, such as for testing or production." Para. [65], "Code completion may provide the programmer with one or more options to automatically insert, complete, or modify code in the programming environment. For example, as the programmer types code, the code completion system 342 may predict and present options for code snippets to insert into the source code 310 displayed in the editor 302. In response to user selection, the code completion system 342 may insert the code snippet into the source code 310 in the editor 302. In other embodiments, the code completion system 342 may predict and insert the code snippets into the source 310 automatically without user input or selection." Examiner notes in Fig. 3 it shows the code completion system executes within the execution environment which in the citation shown above can be the testing or production environment. The machine learning model is included in the co-pilot system which also executes in the execution environment),
identifying, based on a result of the executing in the testing environment, computer code samples that produce candidate results associated with the programming result (Para. [96-97], Smith shows "The keywords may be used by the machine learning model 200 for identifying code snippets to return. At step 503b, the code completion system 342 identifies relevant code snippets from code storage based on features, in the same manner as in step 503. However, in step 503b, the code completion system identifies relevant code snippets based on both the features of the code that were identified and also the keywords that were received from the user in step 501b.. In one embodiment, the keywords may comprise words in code comments that are near the cursor. In one embodiment, the keywords may comprise the text of a function name, such as a function definition “def getHttpResponse” being converted to keywords “get http response.” In one embodiment, the keywords received are natural language keywords such as 'make HTTP request.' " Para. [155], Smith shows "using the features to search for code snippets in a team or local data set or to generate code snippets based on one or more models trained using the team or local data set; identifying or generating one or more candidate code snippets" Examiner notes the above citations show identifying candidate code snippets that are generated according to keywords indicating a programming result);
outputting the at least one verified identified computer code sample (Para. [155], Smith shows "identifying or generating one or more candidate code snippets; displaying the one or more candidate code snippets to the programmer; receiving a selection of one of the candidate code snippets; inputting the selected code snippet into the source code." Examiner notes in the prior citation above smith teaches ranking the code snippets by score. The above citation shows using these ranked code snippets to output into the source code);
Smith does not disclose:
computing functional scores for each of the identified computer code samples
verifying at least one of the identified computer code samples based on the functional scores
and fine-tuning the trained machine learning model based on the at least one verified identified computer code sample
wherein each of the computer code samples are evaluated based on a unit test,
the unit test being generated by the machine learning model;
However, in the analogous art of automated software code modification, Prasad teaches:
Computing a functional score for each of the identified computer code samples (Para. [27], Prasad shows "In some implementations, the developer system causes the modified software code to be implemented in production when a confidence score associated with identifying the portion of the software code and/or associated with modifying the portion of the software code satisfies a confidence score threshold" Examiner notes the above citation shows every piece of modified code receives a confidence score to score the functionality of the modified code before implementation after it reaches a confidence threshold);
verifying at least one of the identified computer code samples based on the functional scores (Para. [27], Prasad shows "In some implementations, the developer system causes the modified software code to be implemented in production when a confidence score associated with identifying the portion of the software code and/or associated with modifying the portion of the software code satisfies a confidence score threshold" Examiner notes the above citation shows using a confidence score to score the functionality of the modified code. Verifying that the code will work because of the confidence score behind it);
and fine-tuning the machine learning model based on the at least one verified identified computer code sample (Para. [29], Prasad shows "the one or more actions include receiving feedback associated with implementing the modified software code and updating the modified software code based on the feedback. For example, the developer system may receive feedback associated with implementing the modified software code based on providing the modified software code for display to the user. In some implementations, the feedback includes additional requirement data associated with the modified software code. The developer system may further modify the modified software code based on the additional requirement data. In some implementations, the developer system further modifies the modified software code in a manner similar to that described above." Para. [32], Prasad shows "the one or more actions include the developer system retraining the machine learning model, the code locator model, and/or the code developer model based on the modified software code. The developer system may utilize the modified software code as additional training data for retraining the machine learning model, the code locator model, and/or the code developer model, thereby increasing the quantity of training data available for training the machine learning model, the code locator model, and/or the code developer model." Examiner notes the above citation shows receiving feedback, which can be considered verifying code functionality, and using the generated modified code to retrain the model)
However, in the analogous art of automatically editing files, Cosgrove teaches:
wherein each of the computer code samples are evaluated based on a unit test, (Para. [87], Cosgrove shows "A first testing component 1106 verifies whether each code edit in the set 1104 correctly performs its functions, with respect to predetermined expectations of what constitutes the expected behavior of these functions. The first testing component 1106 can perform this function by applying a plurality of unit tests 1110 to each code edit in the set 1104. The first testing component 1106 concludes that a code edit under consideration correctly performs its functions when the code edit passes all of its unit tests 1110. Generally, unit testing involves: (1) identifying the operations performed by a code fragment, with the goal of breaking the code fragment into its smallest testable parts; (2) generating assertions that will be used to determine whether each identified operation produces correct or incorrect results; and (3) applying all of the unit tests to the code fragment under consideration. The first testing component 1106 produces output results 1108 that identify the subset of code edits (if any) that pass the above-described type of testing." Examiner notes the above citation shows verifying components taken from the code by testing them),
In addition, in the analogous art of training machine learning models for automatic generation of test cases, Singh teaches:
the unit test being generated by the machine learning model; ([Abstract], Singh shows "Training and/or utilization of machine learning model(s) (e.g., neural network model(s)) in automatically generating test case(s) for source code.").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Prasad into the teachings of Smith to implement “computing functional scores for each of the identified computer code samples, verifying at least one of the identified computer code samples based on the functional scores, and fine-tuning the trained machine learning model based on the at least one verified identified computer code sample”. The modification would have been obvious as one of ordinary skill in the art would be motivated to conserve computing resources and train machine learning models to identify and properly correct the lines of software code (Prasad, Para. [13]).
Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Cosgrove into the teachings of Smith to implement “wherein each of the code samples are further verified based on at least one unit test”. The modification would have been obvious as one of ordinary skill in the art would be motivated to identify if the identified component produces incorrect results (Cosgrove, Para. [87]).
Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Singh into the teachings of Smith to implement “the at least one unit test being generated by the machine learning model”. The modification would have been obvious as one of ordinary skill in the art would be motivated to ensure the quality of test cases and save developers time (Singh, Col. [1] lines [23-27]).
Claim(s) 28 is/are rejected under 35 U.S.C. 103 as being unpatentable by US 20200097261 A1 (Hereinafter referred to as Smith) and US 20220308848 A1 (Hereinafter referred to as Prasad) in further view of US 20210182031 A1 (Hereinafter referred to as Ye).
Regarding claim 28, Singh teaches the independent claims as cited above, but does not disclose:
wherein identifying computer code samples comprises identifying at least one of the computer code samples that passes a unit test
However, in the analogous art of automatic detection of software bugs, Ye teaches:
wherein identifying computer code samples comprises identifying at least one of the computer code samples that passes a unit test (Para. [26], Ye shows "The example identifier 310 identifies correct (reference) copies of code, which refer to subset(s) of code snippets that are determined to be correct (e.g., bug-free) based on set criteria. For example, the identifier 310 determines whether a comprehensive test suite is available for at least a portion of the code snippets extracted using the extractor 305 from a code repository. For example, if the identifier 310 determines that a test suite is available, any code that passes the test suite can be marked as a reference copy (e.g., free of bugs)." Examiner notes the above citation shows testing snippets of code and marking the snippets that pass a test suite as a bug free reference.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Ye into the teachings of Smith to implement "wherein identifying computer code samples comprises identifying at least one of the computer code samples that passes a unit test". The modification would have been obvious as one of ordinary skill in the art would be motivated to increase software productivity by having automatic bug detection (Ye, Para. [15]).
Claim(s) 36 is/are rejected under 35 U.S.C. 103 as being unpatentable by US 20200097261 A1 (Hereinafter referred to as Smith) and US 20220308848 A1 (Hereinafter referred to as Prasad) in further view of US 20210182703 A1 (Hereinafter referred to as Velammal).
Regarding claim 36, Smith teaches the independent claims as cited above, but does not disclose:
wherein the machine learning model is developed by applying training data comprising annotated computer code to a precursor model, the precursor model comprising a machine learning model trained on natural language prompts.
However, in the analogous art of anti-pattern detection, Velammal teaches:
wherein the machine learning model is developed by applying training data comprising annotated computer code to a precursor model, the precursor model comprising a machine learning model trained on natural language prompts (Para. [36], Velammal shows "the output of the source code vectorization unit 206 comprising the tagged anti-patterns in the matrix form are fed to the model training unit 210 of the anti-pattern prediction unit 208. The model training unit 210 of the anti-pattern prediction unit 208 is configured to apply machine learning technique such as, but is not limited to, natural language processing (NLP) on each lexical token vector present in the matrix for determining the type of character associated with each token. The model training unit 210 is configured to train and generate multiple anti-patterns detection models utilizing supervised machine learning techniques such as, but are not limited to, three-layer deep neural network (DNN) etc. based on the fed anti-patterns in the matrix form. The models are generated utilizing an anti-pattern library, such as, but is not limited to, tensor-flow. The models generated are binary models. The models are representative of various anti-patterns associated with various application source codes. The trained models formed are thereafter utilized during assessment for detecting the anti-patterns occurring in the application source code. Further, the trained and generated anti-patterns detection models are stored in the storage unit 214 for future retrieval." Examiner notes the above citation shows feeding code that has been marked and configured for training into a model training unit which uses NLP to further process the training data for model training).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Velammal into the teachings of Smith to implement "wherein the machine learning model is developed by applying training data comprising annotated computer code to a precursor model, the precursor model comprising a machine learning model trained on natural language prompts". The modification would have been obvious as one of ordinary skill in the art would be motivated to use stored models to enhance the training of machine learning models (Velammal, Para. [36]).
Claim(s) 37 is/are rejected under 35 U.S.C. 103 as being unpatentable by US 20200097261 A1 (Hereinafter referred to as Smith) and US 20220308848 A1 (Hereinafter referred to as Prasad) in further view of US 20230412469 A1 (Hereinafter referred to as Ramaswamy).
Regarding claim 37, Smith as modified teaches
wherein the machine learning model is further trained using the generated training data. (Para. [112], Smith shows "Ranking may be used to determine a sequential order in which code snippets are presented to the user, such as top to bottom or left to right. In some embodiments, ranking is performed by assigning a score to code snippets and displaying higher scoring code snippets before lower scoring code snippets. Ranking may be implemented by applying machine learning model 200." Para. [119], Smith shows "In step 601, a machine learning system 200 is trained to detect the need for predictive edits and what edit to suggest. In an embodiment, training data is collected by monitoring programmer edits in the editor 302. As the programmer makes changes to the source code, the programmer's future changes to the source code are also monitored. These examples of initial changes to code leading to future changes in the code may comprise training examples of how an initial change of code implies changes that must be made to other parts of the code. The training examples are used to train the machine learning system to learn to identify code changes that require refactoring and to learn the appropriate refactoring that is required." Examiner notes the citations above shows creating training examples using data gathered by displaying or editing code snippets displayed to the user. All displayed code snippets are given a score prior to display deciding ranking of the snippet and order of the displayed snippets)
Smith does not explicitly disclose:
wherein the machine learning model generates training data based on a result of the computing of the functional scores
However, in the analogous art of selecting machine learning models and training, Ramaswamy teaches:
wherein the machine learning model generates training data based on a result of the computing of the functional scores (Para. [23], Ramaswamy shows " the optimization system 115 may process the network data, the image data, and the inference confidence score, with another model, to select images, from the image data, for training the machine learning model. For example, the model may determine, for each of the images, whether the network data is greater than a network threshold, whether the image data is greater than an image threshold, and whether the inference confidence score is less than a confidence threshold. If the network data is greater than the network threshold, the image data is greater than the image threshold, and the inference confidence score is less than the confidence threshold, for a particular image, the model may select the particular image as training data for the selected machine learning model." Examiner notes the above citation shows using a computed confidence score as training data for a machine learning model)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Ramaswamy into the teachings of Smith to implement "wherein the machine learning model generates training data based on a result of the computing of the functional scores". The modification would have been obvious as one of ordinary skill in the art would be motivated to conserve computing resources and not have a poorly training machine learning model (Ramaswamy, Para. [8])
Claim(s) 38 is/are rejected under 35 U.S.C. 103 as being unpatentable by US 20200097261 A1 (Hereinafter referred to as Smith) and US 20220308848 A1 (Hereinafter referred to as Prasad) in further view of US 20210182703 A1 (US 20230177309 A1 (Hereinafter referred to as Clark).
Regarding claim 38, Smith teaches the independent claims as cited above, but does not disclose:
wherein the machine learning model comprises a plurality of layers, at least one of the layers having a transformer decoder architecture.
However, in the analogous art of training neural networks, Clark teaches:
wherein the machine learning model comprises a plurality of layers, at least one of the layers having a transformer decoder architecture. (Para. [57], Clark shows "The neural network may, for example, be an (autoregressive) transformer decoder, such as one used in some known large learning models (LLM) but modified to include one or more conditional computational layers. For example, when the neural network 102 is a transformer decoder with masked self-attention" Examiner notes the above citation shows a machine learning model containing multiple layers one of them being having a transformer decoder architecture)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Clark into the teachings of Smith to implement “wherein the machine learning model comprises a plurality of layers, at least one of the layers having a transformer decoder architecture”. The modification would have been obvious as one of ordinary skill in the art would be motivated to have a machine learning model that can handle multiple inputs combined together (Clark, Para. [57]).
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 21-40 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-17 of U.S. Patent No. 12061880. Although the claims at issue are not identical, they are not patentably distinct from each other because of obviousness.
Instant Application
Patent No. 12061880
21. A computer-implemented method, comprising:
-receiving a docstring representing natural language text indicating a programming result;
-generating, using a machine learning model and based on the docstring, computer code samples; -identifying computer code samples that produce candidate results associated with the programming result;
-computing functional scores for each of the identified computer code samples;
-verifying at least one of the identified computer code samples based on the functional scores;
-outputting the at least one verified identified computer code sample;
-and fine-tuning the trained machine learning model based on the at least one verified identified computer code sample.
1. A computer-implemented method, comprising:
-receiving a docstring representing natural language text specifying a digital programming result;
-generating, using a trained machine learning model and based on the docstring, one or more computer code samples configured to produce respective candidate results;
-wherein: verifying includes computing a functional correctness score for each of the executed one or more computer code samples;
-the identifying at least one of the computer code samples is based on the functional correctness score;
-causing each of the one or more computer code samples to be executed in a testing environment associated with the trained machine learning model, wherein each of the one or more computer code samples are evaluated based on at least one unit test, the at least one unit test being generated by the machine learning model; -identifying, based on a result of the executing in the testing environment, at least one of the computer code samples which produces a particular candidate result associated with the digital programming result;
-generating, using the trained machine learning model, natural language text associated with the at least one identified computer code sample;
-verifying each of the one or more executed computer code samples;
and
-outputting the at least one identified computer code sample and the natural language text associated with the at least one identified computer code sample;
-and the trained machine learning model is fine-tuned based on verified computer code samples.
22.
wherein the verifying is performed in a testing environment associated with the machine learning model.
1.
-causing each of the one or more computer code samples to be executed in a testing environment associated with the trained machine learning model, wherein each of the one or more computer code samples are evaluated based on at least one unit test, the at least one unit test being generated by the machine learning model; identifying, based on a result of the executing in the testing environment, at least one of the computer code samples which produces a particular candidate result associated with the digital programming result;
-verifying each of the one or more executed computer code samples;
23.
wherein each of the code samples are further verified based on at least one unit test, the at least one unit test being generated by the machine learning model.
1.
-causing each of the one or more computer code samples to be executed in a testing environment associated with the trained machine learning model, wherein each of the one or more computer code samples are evaluated based on at least one unit test, the at least one unit test being generated by the machine learning model;
-verifying each of the one or more executed computer code samples;
24.
further comprising outputting natural language text with the at least one verified identified computer code sample.
1.
-verifying each of the one or more executed computer code samples;
and
-outputting the at least one identified computer code sample and the natural language text associated with the at least one identified computer code sample;
25.
wherein verifying at least one of the identified computer code samples further includes evaluating each of the identified computer code samples based on a time-related threshold
1.
-identifying, based on a result of the executing in the testing environment, at least one of the computer code samples which produces a particular candidate result associated with the digital programming result;
-causing each of the one or more computer code samples to be executed in a testing environment associated with the trained machine learning model, wherein each of the one or more computer code samples are evaluated based on at least one unit test, the at least one unit test being generated by the machine learning model;
-verifying each of the one or more executed computer code samples;
2.
-wherein each of the one or more computer code samples are evaluated based further on a time-related threshold associated with the at least one unit test
26.
wherein the machine learning model is further fine- tuned based on the evaluated computer code samples
4.
-wherein the trained machine learning model is fine-tuned based on the evaluated computer code samples.
27.
wherein the time-related threshold is used to classify each of the code samples into different categories
5.
-wherein the time-related threshold is used to classify each of the one or more computer code samples into different categories
28.
wherein identifying computer code samples comprises identifying at least one of the computer code samples that passes a unit test
3.
- wherein identifying at least one of the computer code samples comprises identifying at least one of the computer code samples that passes the at least one unit test and discarding at least one of the computer code samples that fails the at least one unit test
29.
wherein each of the generated computer code samples is associated with at least one text token or at least one whitespace token
6.
- wherein each of the one or more generated computer code samples is associated with at least one text token.
7. dependent on 6
- wherein each of the one or more generated computer code samples is further associated with at least one whitespace token.
30.
further comprising outputting the candidate results associated with each verified identified computer code sample
8.
- further comprising outputting, via the user interface, the particular candidate result of the at least one identified computer code sample.
31.
wherein the machine learning model is further fine- tuned based on at least one of a public web source or a software repository
9.
- wherein the trained machine learning model is fine-tuned based on at least one of a public web source or software repository.
32.
wherein the machine learning model is fine-tuned based on a set of training problems constructed from examples within the at least one public web source or software repository.
10.
- wherein the trained machine learning model is fine-tuned based on a set of training problems constructed from examples within the at least one public web source or software repository.
33.
wherein identifying computer code samples is based on a mean-log probability.
11.
- wherein identifying at least one of the computer code samples is further based on a mean-log probability.
34.
-compiling the verified identified computer code samples;
-transmitting the verified identified computer code samples to a recipient device;
-storing the verified identified computer code samples;
-and re-executing the verified identified computer code samples.
12.
-compiling the at least one identified computer code sample;
-transmitting the at least one identified computer code sample to a recipient device;
-storing the at least one identified computer code sample;
-and re-executing the at least one identified computer code sample.
35.
further comprising generating natural language text associated with the verified identified computer code samples, wherein the generated natural language text includes a definition of a function, method, class, or module associated with the verified identified computer code samples.
1.
-generating, using the trained machine learning model, natural language text associated with the at least one identified computer code sample;
13.
- wherein the natural language text associated with the at least one identified computer code sample includes a definition of a function, method, class, or module associated with the outputted at least one identified computer code sample.
36.
wherein the machine learning model is developed by applying training data comprising annotated computer code to a precursor model, the precursor model comprising a machine learning model trained on natural language prompts.
14.
-wherein the trained machine learning model is developed by applying training data comprising annotated computer code to a precursor model comprising a machine learning model trained on natural language prompts.
37.
wherein the machine learning model generates training data based on a result of the computing of the functional scores, wherein the machine learning model is further trained using the generated training data.
1.
-wherein: verifying includes computing a functional correctness score for each of the executed one or more computer code samples;
15.
-wherein the trained machine learning model generates training data based on the result of the executing, wherein the trained machine learning model is further trained using the generated training data.
38.
wherein the machine learning model comprises a plurality of layers, at least one of the layers having a transformer decoder architecture.
16.
-wherein the trained machine learning model comprises a plurality of layers, at least one of the layers having a transformer decoder architecture.
39.
A system comprising: at least one memory storing instructions; at least one processor configured to execute the instructions to perform operations comprising:
-receiving a docstring representing natural language text specifying a programming result;
-generating, using a machine learning model and based on the docstring, computer code samples;
-identifying computer code samples that produce candidate results associated with the programming result;
-generating, using the machine learning model, a natural language text associated with the identified computer code samples;
-computing a functional score for each of the identified computer code samples;
-outputting the at least one verified identified computer code sample and the generated natural language text;
-verifying at least one of the identified computer code samples based on the functional scores;
-and fine-tuning the machine learning model based on the at least one verified identified computer code sample.
17.
A system comprising: at least one memory storing instructions; at least one processor configured to execute the instructions to perform operations comprising:
-receiving a docstring representing natural language text specifying a digital programming result;
-generating, using a trained machine learning model and based on the docstring, one or more computer code samples configured to produce respective candidate results;
-causing each of the one or more computer code samples to be executed in a testing environment associated with the trained machine learning model, wherein each of the one or more computer code samples are evaluated based on at least one unit test, the at least one unit test being generated by the machine learning model;
-identifying, based on a result of the executing in the testing environment, at least one of the computer code samples which produces a particular candidate result associated with the digital programming result;
-generating, using the trained machine learning model, a natural language text associated with the at least one identified computer code sample;
-verifying each of the one or more executed computer code samples;
-and outputting the at least one identified computer code sample and the natural language text associated with the at least one identified computer code sample;
-wherein: verifying includes computing a functional correctness score for each of the executed one or more computer code samples;
-the identifying at least one of the computer code samples is based on the functional correctness score;
-and the trained machine learning model is fine-tuned based on verified computer code samples.
40.
A networked device comprising one or more processors to perform operations comprising:
-receiving a docstring representing natural language text specifying a programming result;
-generating, using a machine learning model and based on the docstring, computer code samples;
-causing each of the computer code samples to be executed in a testing environment associated with the machine learning model, wherein each of the computer code samples are evaluated based on a unit test, the unit test being generated by the machine learning model;
-identifying, based on a result of the executing in the testing environment, computer code samples that produce candidate results associated with the programming result;
-outputting the at least one verified identified computer code sample;
-computing functional scores for each of the identified computer code samples;
-verifying at least one of the identified computer code samples based on the functional scores;
-and fine-tuning the machine learning model based on the at least one verified identified computer code sample.
17.
A system comprising: at least one memory storing instructions; at least one processor configured to execute the instructions to perform operations comprising:
-receiving a docstring representing natural language text specifying a digital programming result;
-generating, using a trained machine learning model and based on the docstring, one or more computer code samples configured to produce respective candidate results;
-causing each of the one or more computer code samples to be executed in a testing environment associated with the trained machine learning model, wherein each of the one or more computer code samples are evaluated based on at least one unit test, the at least one unit test being generated by the machine learning model;
-identifying, based on a result of the executing in the testing environment, at least one of the computer code samples which produces a particular candidate result associated with the digital programming result;
-generating, using the trained machine learning model, a natural language text associated with the at least one identified computer code sample;
-verifying each of the one or more executed computer code samples;
-and outputting the at least one identified computer code sample and the natural language text associated with the at least one identified computer code sample;
-wherein: verifying includes computing a functional correctness score for each of the executed one or more computer code samples;
-the identifying at least one of the computer code samples is based on the functional correctness score;
-and the trained machine learning model is fine-tuned based on verified computer code samples.
Although worded differently, the conflicting claims all recite, in summary, a method or a system that, upon receiving a docstring, will generate computer code samples, identify code samples, compute functional scores, verify the samples, output the verified samples, and fine-tune the machine learning model.
For the reasons stated above claims 1-20 are rejected on the grounds of non-statutory double patenting.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20220066914 A1 - This prior art teaches generating code for unit tests
US 20200371778 A1 – This prior art teaches giving confidence scores to code snippets
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZEERICK A MALIK whose telephone number is (571)272-8110. The examiner can normally be reached Mon-Thurs, 7-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chat Do can be reached at (571) 272-3721. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Z.A.M./Examiner, Art Unit 2193
/Chat C Do/Supervisory Patent Examiner, Art Unit 2193
Prosecution Insights