Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/25/26 has been entered.
Status of Claims
This action is in response to the amendments filed 03/25/26. Claims 1, 4, 8, 11, and 15-17 have been amended, claims 2 and 9 have been cancelled. Claims 1, 3-4, 7-8, 10-11, 14-17, and 20 are currently pending.
Response to Arguments
Claim 2 and 9 have been cancelled, therefore the rejections of claim 2 and 9 no longer stand.
In light of Applicant’s amendment regarding the written description rejection, the 112(a) rejection of claims 4, 11, and 17 has been withdrawn.
Applicant’s arguments regarding the 101 rejection have been fully considered but they are not persuasive. Applicant argues that “the claims integrate the alleged judicial exception into a practical application by using the array of real value inputs comprising time to execution, code quality metrics, and the abstract syntax tree to score the unit test. . .this scoring indicates the reliability of the unit tests, thereby ensuring their robustness” and states that “when the claims are interpreted properly, this is not a close call, and the claims are directed to statutory subject matter”. Examiner respectfully disagrees and notes that while the “array of real value inputs comprising time to execution, code quality metrics, and the abstract syntax tree” are input to a discriminative model, the claims do not describe how these inputs would change how the machine learning model itself operates or what kind of “discriminative model” would be required to process these inputs. Using an improved kind of training data may result in a better result from a machine learning model, but does not necessarily change or improve the workings of the actual model. Applicant’s claims are more suitably compared to Recentive Analytics, Inc v. Fox Corp., No. 23-2437 (Fed. Cir. 2025), as the claims merely recite use of a generic machine learning model in a new environment (see page 13 of Recentive). Page 15 of Recentive further states “the claimed methods are not rendered patent eligible by the fact that (using existing machine learning technology) they perform a task previously undertaken by a human with greater speed and efficiency than could previously be achieved. Whether the issue is raised at step one or step two, the increased speed and efficiency resulting from use of computers (with no improve computer technique) do not themselves create eligibility”. Similarly, Applicant has not shown how the claims reflect an improved machine learning model or other computer technique. The 101 rejections have been updated to include the amended limitations and to clarify the reasoning given for the limitations that were not amended where necessary.
Applicant’s arguments regarding the prior art rejection have been fully considered but are moot because of the new grounds of rejection. Applicant argues that the Rao reference does not disclose “a threshold for converting pseudo-labels to binary labels”. Examiner respectfully disagrees and notes that at least figure 1 and section III A of Rao teach “‘weak’ or ‘noisy’ learning functions that are combined in a weighted manner by the generative model. Each learning function acts as a binary classifier that identifies either code search or not code search intent and abstains otherwise. These learning functions, which would contain the weak or pseudo labels, are used as input to the generative model which returns the converted binary label. This falls under the broadest reasonable interpretation of receiving, by the generative model computer program, a plurality of binary labeling functions. . . wherein each binary labeling function has a conversion threshold.
Applicant also argues that the prior art does not teach wherein “the discriminative model receives an array of real value inputs comprising time to execution, code quality metrics, and the abstract syntax tree”. Examiner notes that while at least paragraph [0021] of the Zhou reference teaches wherein an abstract syntax tree can be used as an input to a classifier, or discriminative model, that the Nadein reference has been brought in to teach time to execution and code quality metrics. The prior art rejections have been updated to include the amended limitations and to clarify the reasoning given for the limitations that were not amended where necessary.
Claim Rejections - 35 USC § 112
Claims 4, 11, and 17 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claim 4 recites the limitation “mitigating, by the unit test scoring computer program, statistical issues from the probabilistic determinative model”. Examiner notes that while Applicant’s original disclosure supports mitigation of statistical issues from a discriminative model in at least paragraphs [0008], [0014], [0020], and [0044], Applicant’s original disclosure does not describe mitigation of statistical issues for a “probabilistic determinative” model. For purposes of examination, Examiner is interpreting that statistical issues associated with a discriminative model can be mitigated.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-4, 7-8, 10-11, 14-17, and 20 are rejected under 35 U.S.C. 101. Claims 1, 3-4, and 7 are directed to a method, claims 8, 10-11, and 14 are directed to a system, and claims 15-17 and 20 are directed to a non-transitory computer readable medium; therefore, claims 1, 3-4, 7-8, 10-11, 14-17, and 20 fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter). However, claims 1, 3-4, 7-8, 10-11, 14-17, and 20 fall within the judicial exception of an abstract idea, specifically the abstract ideas of “Mental Processes” (including observation, evaluation, and opinion) and “Mathematical Concepts (including mathematical calculations and relationships)”.
Claim 1:
Claim 1 is directed to a method; therefore, the claim does fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Claim 1 recites the following abstract ideas:
parsing, by a generative model computer program, a plurality of unit tests in a repository into code snippets using an abstract syntax tree (mental step directed to observation, evaluation – a person could parse, or observe and evaluate, a plurality of unit tests in a repository into code snippets in their mind using an abstract syntax tree, potentially assisted by and paper (see MPEP 2106.04(a)(2)(III). Examiner notes that the generative model computer program is interpreted as a generic computer component used to merely apply this mental step (see MPEP 2106.05(f));
mapping, by the generative model computer program, the code snippets into a model readable format (mental step directed to observation, evaluation – a person could map observed code snippets into a model readable format, potentially assisted by and paper (see MPEP 2106.04(a)(2)(III). Examiner notes that mapping these snippets is interpreted in light of at least paragraph [0037] of Applicant’s specification, which states that this mapping is accomplished using an abstract syntax tree. Examiner also notes that the generative model computer program is interpreted as a generic computer component used to merely apply this mental step (see MPEP 2106.05(f));
creating, by the generative model computer program, a labelling matrix by applying the binary labelling functions to the parsed code snippets (mental step directed to evaluation, judgement – a person could create a labelling matrix in their mind by applying observed or determined binary labeling functions to parsed code snippets in their mind, potentially assisted by and paper (see MPEP 2106.04(a)(2)(III). Examiner notes that the generative model computer program is interpreted as a generic computer component used to merely apply this mental step (see MPEP 2106.05(f));
training, by the generative model computer program, a probabilistic generative model using the labelling matrix resulting in a vector of pseudo-labels (as neither the claims nor Applicant’s specification recite further details describing the probabilistic generative model or how this model would be trained by using the labelling matrix, the probabilistic generative model is interpreted as a generic computer component. Training this model is interpreted as generic computer activity merely implementing a mental step, as a person could use an observed or determined labelling matrix in their mind to determine a vector of pseudo-labels, potentially assisted by pen and paper. Examiner notes that the generative model computer program is interpreted as a generic computer component additionally used to merely apply this mental step (see MPEP 2106.04(a)(2)(III) and MPEP 2106.05(f));
building, by a unit test scoring computer program, a discriminative model (as neither the claims nor Applicant’s specification further define a discriminative model in such a way that requires a specific machine learning or artificial intelligence, the broadest reasonable interpretation of a discriminative model includes a mental discriminative model. Building a discriminative model is therefore interpreted as a mental step, as a person could build a mental discriminative model in their mind, potentially assisted by pen and paper. Examiner notes that the unit test scoring computer program is interpreted as a generic computer component used to merely apply this mental step (see MPEP 2106.04(a)(2)(III) and MPEP 2106.05(f));
parsing, by the unit test scoring computer program, the unit test to be scored into a plurality of code snippets to be scored using the abstract syntax tree (mental step directed to observation, evaluation – a person could parse an observed unit test into a plurality of code snippets using an abstract syntax tree in their mind, potentially assisted by pen and paper. Examiner notes that the unit test scoring computer program is interpreted as a generic computer component used to merely apply this mental step (see MPEP 2106.04(a)(2)(III) and MPEP 2106.05(f));
scoring, by the unit test scoring computer program, the unit test to be scored using the trained discriminative model (mental step directed to observation, evaluation – a person could score a unit test using an observed or determined probability threshold in their mind. Examiner notes that the unit test scoring computer program and the discriminative model are interpreted as a generic computer components used to merely apply this mental step (see MPEP 2106.05(f)).
Claim 1 recites the following additional elements:
a generative model computer program, a unit test scoring computer program; wherein each of the unit tests is configured to test a unit of an application; receiving, by the generative model computer program, a plurality of binary labelling functions from a labelling function repository, wherein each binary labeling function has its own labelling threshold for converting pseudo-labels to binary labels; wherein the discriminative model receives an array of real value inputs comprising time to execution, code quality metrics, and the abstract syntax tree; training, by the unit test scoring computer program, the discriminative model using the parsed code snippets, the vector of pseudo labels, and the abstract syntax tree; receiving, by the unit test scoring computer program, a unit test to be scored; and outputting, by the unit test scoring computer program, a score for the unit test and an actionable explanation for the score.
The broadest reasonable interpretation of a generative model computer program, a unit test scoring computer program, and a discriminative model all include interpretation as generic computer components. Wherein each of the unit tests is configured to test a unit of an application is interpreted as the intended use of the kind of unit test being parsed. Receiving a plurality of binary labeling functions from a labelling function repository is interpreted as retrieving data from memory. Wherein each binary labelling function has its own threshold is interpreted as a further description of the kind of data received over a network. Wherein a discriminative model receives an array of real value inputs comprising time to execution, code quality metrics, and the abstract syntax tree, receiving a unit test to be scored, and outputting a score for the unit test and an actionable explanation for the score are all interpreted as transmitting and receiving data over a network. As neither the claims nor Applicant’s specification recite further details describing the discriminative model or how this model would be trained by using the parsed code snippets, the vector of pseudo labels, and the abstract syntax tree, the discriminative model is interpreted as a generic computer component and training this model is interpreted as generic computer activity. These additional elements do not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea (see MPEP 2106.05(d)).
Claim 8 is a system claim and its limitation is included in claim 1. The only difference is that claim 8 requires a system. Therefore, claim 8 is rejected for the same reasons as claim 1.
Claim 15 is a non-transitory computer readable storage medium claim and its limitation is included in claim 1. The only difference is that claim 15 requires a non-transitory computer readable storage medium. Therefore, claim 15 is rejected for the same reasons as claim 1.
The independent claims are not patent eligible.
Dependent claims 3-4, 7, 10-11, 14, 16-17, and 20 when analyzed as a whole are held to be patent ineligible under 35 U.S.C. 101 because the additional recited limitations fail to establish that the claims are not directed to an abstract idea, as they recite further embellishment of the judicial exception.
Claim 3 recites wherein each pseudo label has a one-to-one relationship with a code snippet and weakly labels the code snippet as a positive test quality or a negative test quality. This limitation is interpreted as a further description of the pseudo labels created in the mental step directed to using a labelling matrix to determine a vector of pseudo-labels in claim 1. A person could use a labelling matrix in their mind, potentially assisted by pen and paper, to determine pseudo labels that have a one-to-one relationship with observed code snippets and that weakly label the observed code snippets as positive or negative.
Claim 4 recites mitigating, by the unit test scoring computer program, statistical issues from the probabilistic determinative model. Examiner notes that this claim is interpreted in light of the 112(a) rejection such that statistical issues associated with a discriminative model can be mitigated. Given this interpretation, this limitation is interpreted as a mental step directed to evaluation, judgement – a person could mitigate any statistical issues from a mental discriminative model in their mind, potentially assisted by pen and paper. Examiner notes that the unit test scoring computer program is interpreted as a generic computer component used to merely apply this mental step (see MPEP 2106.04(a)(2)(III) and MPEP 2106.05(f)).
Claim 7 recites wherein the actionable explanation comprises one or more reasons for the score. This limitation is interpreted as a further description of the kind of data transmitted over a network in claim 1 and does not integrate the abstract ideas in claim 1 (on which claim 7 depends) into a practical application or amount to significantly more than the abstract ideas in claim 1.
Claim 10 is a system claim and its limitation is included in claim 3. Claim 10 is rejected for the same reasons as claim 3.
Claim 11 is a system claim and its limitation is included in claim 4. Claim 11 is rejected for the same reasons as claim 4.
Claim 14 is a system claim and its limitation is included in claim 7. Claim 14 is rejected for the same reasons as claim 7.
Claim 16 recites wherein each pseudo label has a one-to-one relationship with a code snippet and weakly labels the code snippet as a positive test quality or a negative test quality. Wherein each pseudo label has a one-to-one relationship with a code snippet and weakly labels the code snippet as a positive test quality or a negative test quality is interpreted as a further description of the pseudo labels created in the mental step directed to using a labelling matrix to determine a vector of pseudo-labels in claim 15. A person could use a labelling matrix in their mind, potentially assisted by pen and paper, to determine pseudo labels that have a one-to-one relationship with observed code snippets and that weakly label the observed code snippets as positive or negative.
Claim 17 is a non-transitory computer readable storage medium claim and its limitation is included in claim 4. Claim 17 is rejected for the same reasons as claim 4.
Claim 20 is a non-transitory computer readable storage medium claim and its limitation is included in claim 7. Claim 20 is rejected for the same reasons as claim 7.
Viewed as a whole, these additional claim elements do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims amount to significantly more than the abstract idea itself. Therefore, the claims are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-4, 7-8, 10-11, 14-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Rao et al (“Search4Code: Code Search Intent Classification Using Weak Supervision”, herein Rao) in view of Drain et al (US 20220066747 A1, herein Drain), in further view of Zhou et al (US 20190324731 A1, herein Zhou), in further view of Nadein et al (US 20220138087 A1, herein Nadein).
Regarding claim 1, Rao teaches a method for training weakly supervised [unit test] quality scoring model (section I para. 3 recites “In this paper, we introduce a novel weak supervision based model to classify code search intent in search queries. We define a query as having code search intent if it can be sufficiently answered with a snippet of code. To the best of our knowledge, this is the first usage of weak supervision in the software engineering domain”), comprising: receiving, by the generative model computer program, a plurality of binary labelling functions from a labelling function repository (fig. 1 and section II para. 4 recite “given a set of unlabeled data points, X, the objective of weak supervision is to estimate the ground truth label by using a set of n learning functions. Each learning function has a probability to abstain and a probability to correctly label a data point as positive or negative. The learning functions are applied over m unlabeled data points to create a matrix of label outputs”. Section III A para. 2 recites “we use several ‘weak’ or ‘noisy’ learning functions, described in Table I, that are combined in a weighted manner by the generative model. We also introduce learning functions to identify patterns that indicate code examples, error codes and exceptions. Each learning function acts as a binary classifier that identifies either code search or not code search intent and abstains otherwise” (i.e., a plurality of binary labelling functions)), wherein each binary labelling function has its own labelling threshold for converting pseudo-labels to binary labels (section III A para. 2 recites “We also introduce learning functions to identify patterns that indicate code examples, error codes and exceptions. Each learning function acts as a binary classifier that identifies either code search or not code search intent and abstains otherwise. We use the label 1 for code search intent, 0 for not code search intent and -1 for abstain. The label for each learning function is chosen after manually analyzing a sample of queries. Table I provides the target label and description of heuristics used for each of the learning functions used along with a few example queries” (i.e., the binary labelling functions have a threshold of at least 1));
creating, by the generative model computer program, a labelling matrix by applying the binary labelling functions to the parsed code snippets; training, by the generative model computer program, a probabilistic generative model using the labelling matrix resulting in a vector of pseudo-labels (fig. 1 and section III A para. 3 recite “Generative Model: We apply all the individual learning functions to the data and construct a label matrix that is then fed to the generative model. The generative model then uses a weighted average of all learning functions outputs, based on the agreements and disagreements between the learning functions, to return the probability scores for each class” (i.e., creating a labelling matrix from the binary labelling functions and training a generative model with the labelling matrix to output probabilities, or pseudo-labels));
building, by a [unit test] scoring computer program, a discriminative model, wherein the discriminative model receives an array of real value inputs [comprising time to execution, code quality metrics, and the abstract syntax tree]; and training, by the [unit test] scoring computer program, the discriminative model using the parsed code snippets, the vector of pseudo labels, [and the abstract syntax tree] (fig. 1 and section III B para. 1 recite “We use the output of the generative model as the train labels (Y train) for the data we collected earlier. We then preprocess and featurize the data before passing it to the discriminative model”. Section III B para. 3 recites “Using the generated training labels (Y train) along with the featurized train data (X train) data, we train several supervised machine learning and deep learning models” (i.e., building and training a discriminative model which takes real value inputs including the parsed code snippets and the pseudo-labels output from the generative model as inputs));
and outputting, by the unit test scoring computer program, a score for [the unit] test and an actionable explanation for the score (Rao section IV C recites “To evaluate the efficacy of the various discriminative models for code search intent detection, we first train each model on the train data and compare the performance scores on the test data. Table III summarizes the performance scores of the four models. We find that the CNN model outperforms all the other models across majority of the metrics with an overall test accuracy of 77% and 76% for C# and Java respectively” (i.e., outputting a score for a test data and reasons for the score)).
However, while Rao teaches a generative model (see at least fig. 1) and using a discriminative model to score testing data (see at least section III B and section IV C), Rao does not explicitly teach parsing. . .a plurality of unit tests in a repository into code snippets for unit testing; parsing, by the unit test scoring computer program, the unit test to be scored into a plurality of code snippets to be scored; and outputting, by the unit test scoring computer program, a score for the unit test [and an actionable explanation for the score].
Drain teaches parsing. . .a plurality of unit tests in a repository into code snippets for unit testing (para. [0027] recites “The subject matter disclosed pertains to a unit test case generation system that automatically generates a unit test case for a particular method”. Para. [0039] recites “A data collection component 220 mines a source code repository 218 to obtain mappings 224 of unit test cases 226 to focal methods 222”. Para. [0066] recite “The pre-training component 206 transforms each of the selected source code files 616 into a concrete syntax tree 618. The concrete syntax tree 618 represents the source code text in a parsed form. The concrete syntax tree 618 may also be a parse tree” (i.e., unit testing for parsed code snippets from a repository));
receiving, by the unit test scoring computer program, a unit test to be scored (Drain para. [0039] recites “A data collection component 220 mines a source code repository 218 to obtain mappings 224 of unit test cases 226 to focal methods 222. A mapped unit test case 228 is formatted as a pair, mtci,= {tci, fmi}, where mtci represents a mapping of a unit test case, tci, to a focal method, fmi” (i.e., receiving a unit test to be scored));
parsing, by the unit test scoring computer program, the unit test to be scored into a plurality of code snippets to be scored [using the abstract syntax tree] (Drain para. [0027] recites “The subject matter disclosed pertains to a unit test case generation system that automatically generates a unit test case for a particular method”. Drain para. [0039] recites “A data collection component 220 mines a source code repository 218 to obtain mappings 224 of unit test cases 226 to focal methods 222”. Drain para. [0066] recite “The pre-training component 206 transforms each of the selected source code files 616 into a concrete syntax tree 618. The concrete syntax tree 618 represents the source code text in a parsed form. The concrete syntax tree 618 may also be a parse tree” (i.e., unit testing for parsed code snippets from a repository));
scoring, by the unit test scoring computer program, the unit test to be scored [using the trained discriminative model] (Drain para. [0039] recites “A data collection component 220 mines a source code repository 218 to obtain mappings 224 of unit test cases 226 to focal methods 222. A mapped unit test case 228 is formatted as a pair, mtci,= {tci, fmi}, where mtci represents a mapping of a unit test case, tci, to a focal method, fmi” (i.e., the unit test to be scored). Drain fig. 3 and para. [0056] recite “The softmax layer 336 then turns the scores of the log its vector into probabilities for each subtoken in the vocabulary which are positive and normalized” (i.e., scoring the unit test data));
and outputting, by the unit test scoring computer program, a score for the unit test [and an actionable explanation for the score] (Drain fig. 3 and para. [0056] recite “The softmax layer 336 then turns the scores of the log its vector into probabilities for each subtoken in the vocabulary which are positive and normalized” (i.e., outputting a score for a unit test)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine these teachings by applying the code parsing methods from Drain to the code search inputs from the model from Rao. Rao and Drain are both directed to code searching, or language processing methods. One of ordinary skill in the art would benefit from parsing the inputs from Rao using the tree parsing methods from Drain to mine the code sections from an input dataset such as the one described in section V of Rao to better represent code constructs in the grammar of a programming language.
However, while Drain teaches using a parse tree to parse source code snippets from a repository (see at least para. [0066]), the combination of Rao and Drain does not explicitly teach parsing a plurality of code snippets using an abstract syntax tree or mapping the code snippets into a model readable format.
Zhou teaches parsing a plurality of code snippets using an abstract syntax tree, and mapping the code snippets into a model readable format, wherein input to the model comprise the abstract syntax tree (para. [0021] recites “ML/AI models are trained using encoded subtrees, which are encoded after parsing a software program into an abstract syntax tree (AST)”. Para. [0039] recites “the example recommender system 108 is provided with the example software parser 202 to receive software from the example input determiner 106 (FIG. 1) and parse the software into subtrees of an abstract syntax tree (AST). A parser (e.g., the software parser 202) is a compiler or an interpreter component that breaks data into smaller elements for simple translation into another language. The example software parser 202 takes input data (e.g., legacy software 103 and/or new software 105) in the form of a sequence of tokens or program instructions and builds a data structure in the form of an AST. For example, the data structure may consist of several subtrees that form the AST. Additionally and/or alternatively, the example software parser 202 builds a data structure in the form of a parse tree”. Para. [0041] recites “ the subtree encoder extracts syntactic paths from within a code snippet (e.g., a subtree), maps each path to its corresponding real-valued vector representation, then concatenates each vector into a single vector that represents the path context” (i.e., parsing and mapping code snippets with an abstract syntax tree such that the abstract syntax tree comprising the parsed snippets can be input to a model. Examiner notes that mapping these snippets is interpreted in light of at least paragraph [0037] of Applicant’s specification, which states that this mapping is accomplished using the abstract syntax tree)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine these teachings by substituting the parse tree from Drain with the abstract syntax tree used by Zhou. Drain and Zhou both state that parse trees can be used to parse code into smaller elements (see at least paragraph [0066] of Drain and paragraph [0039] of Zhou). One of ordinary skill in the art would recognize that simple substitution of a known parse tree like the one taught by both Drain and Zhou for a known abstract syntax tree as taught by Zhou would obtain predictable results.
However, while the combination of Rao, Drain, and Zhou teaches inputting an abstract syntax tree to a model (see at least paragraph [0021] of Zhou), the combination of Rao, Drain, and Zhou does not explicitly teach wherein the inputs to the model comprise time to execution and code quality metrics.
Nadein teaches wherein the inputs to the model comprise time to execution and code quality metrics (para. [0011] recites “wherein the data corresponding to the plurality test results of the test runs include one or more of the following data: stack traces and execution times, types of test runners, and programming language the tests, but the disclosure is not limited thereto”. Para. [0095] recites “The ARBTM 406 (i.e., the automatic risk-based testing module) may be configured to test execution prioritization algorithm. There may be opportunities to identify tests that almost never fail. One can save significant resources if such tests are executed less frequently than other more likely to fail tests. The ARBTM 406 may be configured to understand the probability of a failure in the likely-to-fail population and prioritize execution of failure-prone tests over those that are less likely to fail, thereby greatly decreasing the lag time between code commits and test result feedback to developers. Having a general understanding of these two populations may help to distill test results data into actionable items that inform developers, while they write code, of the impact of their changes on quality”. Para. [0157] recites “The ARBTM 406 may calculate L per code base—a repository, in this example—since the code quality depends on its author's knowledge of language, familiarity with the code base, understanding of domain where the code is applied, and other factors as follows”. Para. [0132] recites “the ARBTM 406 derives the overall failure rate of an input test case for the past k test suite runs of n total runs. This metric may indicate that at a high level how often does this test signal a defect within the code base. But, it may not indicate the relative importance of those failures” (i.e., the unit testing model can incorporate at least code quality and execution time metrics)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine these teachings by utilizing the metrics taught by the testing system from Nadein to modify the unit tests taught by the combination of Rao, Drain, and Zhou. Nadein and Drain are both directed to software testing systems. One of ordinary skill in the art would be motivated to utilize the metrics from Nadein to improve the robustness of the unit tests from Drain.
Regarding claim 3, the combination of Rao, Drain, Zhou, and Nadein teaches the method of claim 1, wherein each pseudo label has a one-to-one relationship with a code snippet and weakly labels the code snippet as a positive test quality or a negative test quality (Rao section II para. 4 recites “given a set of unlabeled data points, X, the objective of weak supervision is to estimate the ground truth label by using a set of n learning functions. Each learning function has a probability to abstain and a probability to correctly label a data point as positive or negative. The learning functions are applied over m unlabeled data points to create a matrix of label outputs” (i.e., each data point, or code snippet, is weakly labelled as positive or negative)).
Regarding claim 4, the combination of Rao, Drain, Zhou, and Nadein teaches the method of claim 1, further comprising: mitigating, by the unit test scoring computer program, statistical issues from the probabilistic determinative model (Rao section II para. 6 recites “The generative model then takes ꓥ as input and returns the probability scores for each class based on the agreements and disagreements between the learning functions. The predicted label distribution output can then be used as probabilistic training labels by a discriminative classifier for a downstream classification task. We use weak supervision to generate the train labels for the code search intent classification task”. Drain para. [0086] recites “The neural transformer model with attention is tested using a validation dataset to determine the appropriate hyperparameters settings to achieve a desired goal” (i.e., validating a model to ensure that the model has not overfit, which falls under the broadest reasonable interpretation of “mitigating statistical issues”. Examiner notes that this claim is interpreted in light of the 112(a) rejection such that statistical issues associated with a discriminative model can be mitigated)).
Regarding claim 7, the combination of Rao, Drain, Zhou, and Nadein teaches the method of claim 1, wherein the actionable explanation comprises one or more reasons for the score (Drain fig. 3 and para. [0056] recite “The softmax layer 336 then turns the scores of the log its vector into probabilities for each subtoken in the vocabulary which are positive and normalized”. Rao section IV C recites “To evaluate the efficacy of the various discriminative models for code search intent detection, we first train each model on the train data and compare the performance scores on the test data. Table III summarizes the performance scores of the four models. We find that the CNN model outperforms all the other models across majority of the metrics with an overall test accuracy of 77% and 76% for C# and Java respectively” (i.e., outputting a score for a unit test and reasons for the score, such as the precision, recall, and/or F1 score of the model based on the input data)).
Claim 8 is a system claim and its limitation is included in claim 1. The only difference is that claim 8 requires a system. Therefore, claim 8 is rejected for the same reasons as claim 1.
Claim 10 is a system claim and its limitation is included in claim 3. Claim 10 is rejected for the same reasons as claim 3.
Claim 11 is a system claim and its limitation is included in claim 4. Claim 11 is rejected for the same reasons as claim 4.
Claim 14 is a system claim and its limitation is included in claim 7. Claim 14 is rejected for the same reasons as claim 7.
Claim 15 is a non-transitory computer readable storage medium claim and its limitation is included in claim 1. The only difference is that claim 15 requires a non-transitory computer readable storage medium. Therefore, claim 15 is rejected for the same reasons as claim 1.
Regarding claim 16, the combination of Rao, Drain, Zhou, and Nadein teaches the non-transitory computer readable storage medium of claim 15, wherein each pseudo label has a one-to-one relationship with a code snippet and weakly labels the code snippet as a positive test quality or a negative test quality (Rao section II para. 4 recites “given a set of unlabeled data points, X, the objective of weak supervision is to estimate the ground truth label by using a set of n learning functions. Each learning function has a probability to abstain and a probability to correctly label a data point as positive or negative. The learning functions are applied over m unlabeled data points to create a matrix of label outputs” (i.e., each data point, or code snippet, is weakly labelled as positive or negative)).
Claim 17 is a non-transitory computer readable storage medium claim and its limitation is included in claim 4. Claim 17 is rejected for the same reasons as claim 4.
Claim 20 is a non-transitory computer readable storage medium claim and its limitation is included in claim 7. Claim 20 is rejected for the same reasons as claim 7.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
“A Unified Scheme of Some Nonhomogeneous Poisson Process Models for Software Reliability Estimation” (Huang et al) teaches an overview of different software reliability testing metrics including execution time and the quality of the software test.
US 20210192321 A1 (Zhang) teaches a method for learning and utilizing mappings between source code changes and regions of latent space associated with code change intents.
US 20210149793 A1 (Shao et al) teaches a method for determining a cognitive code coverage weight for code snippets located in a portion of code.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEAH M FEITL whose telephone number is (571) 272-8350. The examiner can normally be reached on M-F 0900-1700 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached on (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/L.M.F./ Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
Prosecution Insights