Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-33 are presented for examination.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “wherein the test selection system is configured to…” in claim 10, “wherein the test selection system is further configured to” in claim 11-12, “wherein the test selection system is configured to …” Claim 21, “wherein the test selection system is further configured to …”.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-12, 15-23 and 29-33 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding Claim 1, the claim recites in limitation 1a “one or more characteristics relating to an importance of said code, an importance of each of said plurality of tests, or both.” This creates an internal inconsistency: if only one of code importance or test importance is selected via the disjunctive “or” in limitation 1a, then one of limitations 1d or 1e cannot be satisfied, rendering the scope of the claim unclear. Appropriate correction is required.
Dependent Claims 2–9 and 32-33 are also rejected under 35 U.S.C. 112(b) as depending from rejected Claim 1.
Regarding Claim 9, the claim recites “wherein the user determines a rule for test importance or running the selected test.” There is insufficient antecedent basis for the term “the user” in the claim. Claim 1, from which Claim 9 depends, does not introduce “a user.” Appropriate correction is required.
Regarding Claim 10, the claim recites in limitation 10a “execute a set of a plurality of tests on a first set of program code, wherein execution of each respective test generates data indicative of code coverage of the respective test.” Limitation 10b then recites “store a mapping of the tests to the first set of program code based on the code coverage data.” The term “the code coverage data” lacks proper antecedent basis because limitation 10a introduces “data indicative of code coverage,” not “code coverage data.” It is unclear whether “the code coverage data” refers to the same “data indicative of code coverage” or to different data. Appropriate correction is required.
Dependent Claims 11–12 are also rejected under 35 U.S.C. 112(b) as depending from rejected Claim 10.
Regarding Claim 12, the claim recites “determine, based on the code coverage data,” which inherits the same antecedent basis deficiency from Claim 10 as discussed above. Appropriate correction is required.
Regarding Claim 15, the claim recites in limitation 15c “receiving data indicating modifications to one or more portions of the first version of the code for a second version of the program code.” The term “the code” lacks proper antecedent basis. Limitation 15a introduces “program code” (i.e., “a first version of program code”), not simply “the code.” It is unclear whether “the code” refers to the same “program code” or something different. Appropriate correction is required.
Additionally, limitation 15d(ii) recites “the data indicating modified portions.” However, limitation 15c introduces “data indicating modifications to one or more portions,” not “data indicating modified portions.” The shift in terminology from “modifications to portions” to “modified portions” renders the antecedent unclear.
Claims 16–17 are also rejected under 35 U.S.C. 112(b) as depending from rejected Claim 15.
Regarding Claim 17, the claim recites in limitation 17c “moving the test from the deprecated suite to the suite of tests.” There is insufficient antecedent basis for “the suite of tests.” Claim 15, the parent claim, introduces “tests” generically but never establishes “a suite of tests.” Claim 16 introduces “a suite of deprecated tests,” but “the suite of tests” in Claim 17c appears to reference an active (non-deprecated) suite that was never introduced. Appropriate correction is required.
Regarding Claim 18, the claim recites “determining a subset of the tests that are likely to exercise a second set of program code that comprises one or more modified portions of the first set of program code, based at least in part on the mapping and the data indicative of the one or more modified portions.” There is insufficient antecedent basis for “the data indicative of the one or more modified portions.” The claim introduces “data indicative of code coverage” but never separately introduces data indicative of modified portions with an indefinite article. The “one or more modified portions” is introduced as a description of the second set of program code, but “the data indicative of” those portions is never independently established. Appropriate correction is required.
Claims 19–20 are also rejected under 35 U.S.C. 112(b) as depending from rejected Claim 18.
Regarding Claim 21, the claim recites “wherein said mapping is performed per test executed on code in a selected environment, and also according to at least one previous mapping for said code in a different environment.” The term “said code” is ambiguous. The claim introduces “a first set of program code” and “a second set of program code,” and it is unclear which of these “said code” refers to. Appropriate correction is required.
Claims 22–23 are also rejected under 35 U.S.C. 112(b) as depending from rejected Claim 21.
Regarding Claim 29, the claim recites “updating a test selection system based on the identified changes to select tests relevant to the new build version.” There is insufficient antecedent basis for “the new build version.” The claim introduces “a build version of running components” and references “changes in the build version,” but never introduces “a new build version.” It is unclear whether “the new build version” refers to the originally detected build version, a subsequent build version, or the build version as modified by the identified changes. Appropriate correction is required.
Claims 30–31 are also rejected under 35 U.S.C. 112(b) as depending from rejected Claim 29.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 15 recites a computer-readable storage medium. However, the specification did not define what type of medium is included in a computer storage medium. According to MPEP 2111, examiner obliged to give the terms or phrases their broadest interpretation definition awarded by one of an ordinary skill in the art unless applicant has provided some indication of the definition of the claimed terms or phrases. Therefore, examiner interprets the a computer-readable storage medium including any type of medium which includes carrier medium such as signals. Signals are directed to a non-statutory subject matter. Thus, claim 15 is rejected under 35 U.S.C. 101 for directing to a non-statutory subject matter. Applicant is advised to amend to “a non-transitory computer-readable storage medium” to overcome this rejection.
Dependent Claims 16-17 are rejected for failing to cure the deficiency from their respective parent claim by dependency.
Claims 1-9, 13-14, 29-33 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Claims 1 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “determining a relative importance of a selected test in a plurality of tests for testing code”, “determine the relative importance of the selected test”, “determining a relationship between the selected test and the code that is tested according to a statistical analysis, a machine learning algorithm or a combination thereof”, “wherein said characteristics relate to an importance of said code and are selected from the group consisting of interactions between a section of the code and other sections, an importance of the code overall for execution in production, code with many changes or new commits, and otherwise modified code; and e. wherein said characteristics relate to an importance of said plurality of tests and are selected from the group consisting of tests that apply to code that is deemed important, tests that have failed previously, a new test, and a test that is testing modified code” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
This judicial exception is not integrated into a practical application. The claims recites the following additional elements “a computational device,” “said computational device comprising a memory for storing instructions and a processor for executing said stored instructions,” “an analyzer executed by said processor,” “a machine learning algorithm” and “receiving one or more characteristics relating to an importance of said code, an importance of each of said plurality of tests, or both”. The additional elements “a computational device,” “said computational device comprising a memory for storing instructions and a processor for executing said stored instructions,” “an analyzer executed by said processor,” “a machine learning algorithm” are merely instructions to implement an abstract idea on a computer, or merely using a generic computer or computer components as a tool to perform the abstract idea. See MPEP 2106.05(f). The additional element “receiving one or more characteristics relating to an importance of said code, an importance of each of said plurality of tests, or both” does nothing more than add insignificant extra solution activity to the judicial exception, such as data gathering and outputting the results of the abstract idea to perform a task. See MPEP 2106.05(g). Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea, thus fail to integrate the abstract idea into a practical application.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element “a computational device,” “said computational device comprising a memory for storing instructions and a processor for executing said stored instructions,” “an analyzer executed by said processor,” “a machine learning algorithm” are generic computer components and instructions used as the tools to perform the abstract idea. See MPEP 2106.05(f). As to the additional element “receiving one or more characteristics relating to an importance of said code, an importance of each of said plurality of tests, or both” the courts have identified gathering data and displaying the output of the abstract idea is well-understood, routine, conventional activity. See MPEP 2106.05(d). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claims are not patent eligible.
Claim 2 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein an importance of the selected test in relation to said code is determined at least partially according to a change in environment, environment variables and characteristics, a change in a configuration file, or a change in an asset” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 3 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein said statistical analysis determines a statistical relationship between said selected test and the code according to a timing of execution of said selected test in relation to a time of execution of a particular section of code” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 4 recites the following additional elements “analyzer further comprises a plurality of machine learning algorithms and an ensemble learning algorithm for determining said relationship” which are merely instructions to implement an abstract idea on a computer, or merely using a generic computer or computer components as a tool to perform the abstract idea. See MPEP 2106.05(f). Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea, thus fail to integrate the abstract idea into a practical application.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element are generic computer components and instructions used as the tools to perform the abstract idea. See MPEP 2106.05(f). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claims are not patent eligible.
Claim 5 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “a mapper for determining a relationship between one or more code components and said plurality of tests to be executed” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 6 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein said mapper determines said relationship according to a previous history of execution of said plurality of tests or according to an analysis of a relationship between said plurality of tests and execution of said code” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 7 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein said relationship is determined according to a timing of execution of said code and a timing margin, wherein said timing margin comprises an additional amount of time for accepting a test execution as being related to said code execution” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 8 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein said relationship is further determined according to a relative degree of sensitivity versus accuracy” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 9 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein the user determines a rule for test importance or running the selected test” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claims 13 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “determining a subset of the tests likely to exercise a second set of program code comprising one or more modified portions of the first set of program code, based on:i. the mapping, ii. data indicating the one or more modified portions, and iii. a determination of a relationship between the mapping of the tests to the one or more modified portions” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
This judicial exception is not integrated into a practical application. The claims recites the following additional elements “a computer-implemented” and “receiving a mapping of tests to a first set of program code, wherein the mapping is based on code coverage data generated from execution of the tests on the first set of program code and indicates portions of the first set exercised by respective tests”. The additional elements “a computer-implemented” are merely instructions to implement an abstract idea on a computer, or merely using a generic computer or computer components as a tool to perform the abstract idea. See MPEP 2106.05(f). The additional element “receiving a mapping of tests to a first set of program code, wherein the mapping is based on code coverage data generated from execution of the tests on the first set of program code and indicates portions of the first set exercised by respective tests” does nothing more than add insignificant extra solution activity to the judicial exception, such as data gathering and outputting the results of the abstract idea to perform a task. See MPEP 2106.05(g). Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea, thus fail to integrate the abstract idea into a practical application.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element “a computer-implemented” are generic computer components and instructions used as the tools to perform the abstract idea. See MPEP 2106.05(f). As to the additional element “receiving a mapping of tests to a first set of program code, wherein the mapping is based on code coverage data generated from execution of the tests on the first set of program code and indicates portions of the first set exercised by respective tests” the courts have identified gathering data and displaying the output of the abstract idea is well-understood, routine, conventional activity. See MPEP 2106.05(d). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claims are not patent eligible.
Claims 14 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “determining, based on the code coverage data, that a particular test does not exercise the first set of program code; and b. moving the particular test to a suite of deprecated tests excluded from the subset likely to exercise the second set of program code” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claims 29 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “monitoring a distributed testing environment to detect a build version of running components; automatically identifying changes in the build version; and updating a test selection system based on the identified changes to select tests relevant to the new build version” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
This judicial exception is not integrated into a practical application. The claims recites the following additional elements “automatic build discovery in a CI/CT/CD system,” which are merely instructions to implement an abstract idea on a computer, or merely using a generic computer or computer components as a tool to perform the abstract idea. See MPEP 2106.05(f). Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea, thus fail to integrate the abstract idea into a practical application.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element are generic computer components and instructions used as the tools to perform the abstract idea. See MPEP 2106.05(f). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claims are not patent eligible.
Claims 30 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “analyzing messages regarding test execution to determine which server or service is associated with a particular test” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claims 31 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein the automatic identification of changes includes analyzing build-diff data and matching with test footprints” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claims 32 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “mapping a test method to a list of methods or functions executed upon execution of the test; analyzing the mapping to determine a relative likelihood of failure for the test; and prioritizing the test based on the determined relative likelihood of failure” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claims 33 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “mapping a test method to a list of methods or functions executed upon execution of the test; analyzing the mapping to determine a relative likelihood of failure for the test; and prioritizing the test based on the determined relative likelihood of failure” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Belur (US 2016/0259713 A1) in view of Boshernitsan (US 2015/0007140 A1) further in view of Deng (US 8,276,123 B1).
Regarding Claim 1, Belur (US 2016/0259713 A1) teaches
A system for determining a relative importance of a selected test in a plurality of tests for testing code, comprising:
a. a computational device for receiving one or more characteristics relating to an importance of said code, an importance of each of said plurality of tests, or both (Para[0023], "Testing application 141 may then query database 130 to retrieve the list of relevant tests which map back to the source code, as well as code coverage associated with those tests. Testing application 141 may instead query for relevant tests matching any feasible criteria, such as having some desired code coverage or cyclomatic dependency score") Examiner Comments: Belur’s testing application is a computational device that receives test relevance characteristics (code coverage and cyclomatic dependency scores) relating to the importance of the tests with respect to the code;
b. said computational device comprising a memory for storing instructions and a processor for executing said stored instructions to determine the relative importance of the selected test (Para[0029]–[00331], "system 400 includes, without limitation, a central processing unit (CPU) 410... a memory 460" and "CPU 410 retrieves and executes programming instructions stored in the memory 460") Examiner Comments: Belur explicitly teaches a computing system with CPU and memory that executes instructions
wherein said characteristics relate to an importance of said plurality of tests and are selected from the group consisting of tests that apply to code that is deemed important, tests that have failed previously, a new test, and a test that is testing modified code (Belur, Para[0020]–[0021], "new tests or modified tests may be learned at a delta level" and "testing application 141 automatically retrieves a list of tests relevant to the source code file") Examiner Comments: Belur teaches identifying tests relevant to modified source code and recognizing new/modified tests, which are characteristics relating to test importance from the claimed group.
Belur did not specifically teach
c. an analyzer executed by said processor for determining a relationship between the selected test and the code that is tested according to a statistical analysis, a machine learning algorithm or a combination thereof
wherein said characteristics relate to an importance of said code and are selected from the group consisting of interactions between a section of the code and other sections, an importance of the code overall for execution in production, code with many changes or new commits, and otherwise modified code.
However, Boshernitsan teaches
wherein said characteristics relate to an importance of said code and are selected from the group consisting of interactions between a section of the code and other sections, an importance of the code overall for execution in production, code with many changes or new commits, and otherwise modified code (Para[0063], "weights tests... according to the number of functions having property changes in the changed code") Examiner Comments: Boshernitsan teaches weighting based on property changes in modified code, together teaching characteristics from the claimed group;
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Belur and Boshernitsan did not specifically teach
an analyzer executed by said processor for determining a relationship between the selected test and the code that is tested according to a statistical analysis, a machine learning algorithm or a combination thereof.
However, Deng (US 8,276,123 B1) teaches
an analyzer executed by said processor for determining a relationship between the selected test and the code that is tested according to a statistical analysis, a machine learning algorithm or a combination thereof (Col. 16, line 21–31, "Intelligent RTS module 22 may implement an adaptive-combo test selection algorithm that may adaptively apply one or more, e.g., a combination or combo of, test selection algorithms to select the subset of test cases") Examiner Comments: Deng’s intelligent RTS module serves as an analyzer that uses adaptive algorithms (analogous to statistical analysis) to determine relationships between tests and code based on coverage, dependency, and code change data.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur and Boshernitsan’s teaching with Deng’s in order to apply adaptive and statistical analysis algorithms to the test-to-code mapping data, as Deng’s intelligent RTS module provides a well-known approach for refining test selection beyond simple coverage mapping by adaptively combining multiple selection algorithms.
Regarding Claim 2, Belur, Boshernitsan and Deng teach
The system of Claim 1.
Boshernitsan further teaches
Wherein an importance of the selected test in relation to said code is determined at least partially according to a change in environment, environment variables and characteristics, a change in a configuration file, or a change in an asset (Boshernitsan, Para[0063], "weights tests... according to the number of functions having property changes in the changed code") Examiner Comments: Property changes in code functions are analogous to changes in configuration or assets that affect test importance.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 3, Belur, Boshernitsan and Deng teach
The system of Claim 1.
Belur further teaches
Wherein said statistical analysis determines a statistical relationship between said selected test and the code according to a timing of execution of said selected test in relation to a time of execution of a particular section of code (Belur, Para[0012], "the instrumented instructions provide, e.g., counters indicating the number of times particular lines of code, methods, etc. have been executed")Examiner Comments: Belur’s agent correlates test execution with code execution through instrumentation counters that track when particular code sections execute during test runs, establishing a timing-based statistical relationship.
Regarding Claim 4, Belur, Boshernitsan and Deng teach
The system of Claim 1.
Deng further teaches
Wherein said analyzer further comprises a plurality of machine learning algorithms and an ensemble learning algorithm for determining said relationship (Deng, Col. 16, line 21–31, "Intelligent RTS module 22 may implement an adaptive-combo test selection algorithm that may adaptively apply one or more, e.g., a combination or combo of, test selection algorithms to select the subset of test cases") Examiner Comments: Deng’s adaptive-combo algorithm combines multiple test selection algorithms, which is analogous to an ensemble learning approach.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur and Boshernitsan’s teaching with Deng’s in order to apply adaptive and statistical analysis algorithms to the test-to-code mapping data, as Deng’s intelligent RTS module provides a well-known approach for refining test selection beyond simple coverage mapping by adaptively combining multiple selection algorithms.
Regarding Claim 5, Belur, Boshernitsan and Deng teach
The system of Claim 1.
Belur further teaches
Further comprising a mapper for determining a relationship between one or more code components and said plurality of tests to be executed (Belur, Para[0013], "agent 122 maps portions of the source code indicated by the code coverage instrumentation to the test and reports the mapping to testing application 141"; Para[0023], "the mappings being at, e.g., the source code file, method, or source code line level of granularity") Examiner Comments: Belur’s agent/testing application serves as a mapper determining the relationship between code components (files, methods, lines) and tests.
Regarding Claim 6, Belur, Boshernitsan and Deng teach
The system of Claim 5.
Belur further teaches
Wherein said mapper determines said relationship according to a previous history of execution of said plurality of tests or according to an analysis of a relationship between said plurality of tests and execution of said code (Belur, Para[0013], "the mapping need not be relearned each time the test is run on the source code, and testing application 142 may instead monitor learned tests and only relearn tests that have changed") Examiner Comments: Belur’s system builds and maintains a history of test-to-code mappings from previous test executions and uses this history to determine test relevance.
Regarding Claim 7, Belur, Boshernitsan and Deng teach
The system of Claim 6.
Belur further teaches
Wherein said relationship is determined according to a timing of execution of said code and a timing margin, wherein said timing margin comprises an additional amount of time for accepting a test execution as being related to said code execution (Belur, Para[0012], "the instrumented instructions provide, e.g., counters indicating the number of times particular lines of code, methods, etc. have been executed") Examiner Comments: Belur’s runtime instrumentation correlates test execution with code execution through timing; it would have been obvious to incorporate a timing margin for asynchronous execution, as this is well-known in dynamic code analysis.
Regarding Claim 8, Belur, Boshernitsan and Deng teach
The system of Claim 7.
Deng further teaches
Wherein said relationship is further determined according to a relative degree of sensitivity versus accuracy (Deng, Abstract, "a computing device may include a user interface with which a user interacts to specify a safety level … adaptively applying these algorithms in order of decreasing safety assurances and increasing precision") Examiner Comments: Deng’s safety level specification and adaptive application of algorithms trading off safety assurances versus precision directly teaches determining a relationship according to sensitivity versus accuracy.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur and Boshernitsan’s teaching with Deng’s in order to apply adaptive and statistical analysis algorithms to the test-to-code mapping data, as Deng’s intelligent RTS module provides a well-known approach for refining test selection beyond simple coverage mapping by adaptively combining multiple selection algorithms.
Regarding Claim 9, Belur, Boshernitsan and Deng teach
The system of Claim 1.
Belur further teaches
Wherein the user determines a rule for test importance or running the selected test (Belur, Para[0016], "a blacklists and/or whitelist may be employed to limit or increase the number of tests that are run"; Para [0027], "the blacklist may be added to configuration file 110 by a user") Examiner Comments: Belur teaches that the user configures blacklists/whitelists that define rules for which tests to run or exclude, constituting user-determined rules for test importance.
Claim(s) 10-33 is/are rejected under 35 U.S.C. 103 as being unpatentable over Belur (US 2016/0259713 A1) in view of Boshernitsan (US 2015/0007140 A1).
Regarding Claim 10, Belur (US 2016/0259713 A1) teaches
A test selection system comprising one or more computing devices, wherein the test selection system is configured to:
a. execute a set of a plurality of tests on a first set of program code, wherein execution of each respective test generates data indicative of code coverage of the respective test (Abstract, "During a training phase, the agent intercepts tests being run on source code and provides a dump of the interception results after each test is executed"; Para[0013], "agent 122 maps portions of the source code indicated by the code coverage instrumentation to the test and reports the mapping to testing application 141. For example, if test A were run on source code file X, agent 122 may generate a dump of code coverage instrumentation results showing that test A affected particular lines or methods in source code file X") Examiner Comments: Belur’s training phase, where an agent intercepts tests and generates code coverage results for each test, teaches executing tests on a first set of code where each test generates code coverage data;
b. store a mapping of the tests to the first set of program code based on the code coverage data, wherein the mapping comprises data indicating portions of the first set of program code exercised by said respective tests (Para[0013], "Testing application 141 may then store the mapping of test A to file X, or the particular lines or methods in file X, and the associated code coverage information in database 130") Examiner Comments: Belur explicitly teaches storing mappings of tests to source code in a database, where the mappings include which lines or methods are exercised by respective tests;
c. receive a second set of program code, comprising modifications to said first set of program code, associated with data indicating one or more modified portions (Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system) to determine if a new version of a source code file is being checked in. Such a source code file may be a new file or an existing file that has been modified by a developer") Examiner Comments: Belur teaches detecting when a modified version of source code is checked in, where the modifications constitute the one or more modified portions;
d. select a subset of the tests likely to exercise the second set of program code based on (i) the mapping of the tests to the first set of program code, and (ii) the data indicating the one or more modified portions (Para[0014], "testing application 141 automatically retrieves a list of tests relevant to the source code file by querying database 130 for tests which map to the source code file, if any. That is, testing application 141 reverse maps the source code file (or lines of code) to relevant tests, based on the mappings stored in the database 130") Examiner Comments: Belur’s reverse mapping from modified source code file to stored test mappings teaches selecting a subset based on the mapping and the modified portions.
Belur did not specifically teach
(iii) a determination of a relationship between the mapping of the tests to the one or more modified portions
cause the subset of tests to be executed on the second set of program code.
However, Boshernitsan (US 2015/0007140 A1) teaches
a determination of a relationship between the mapping of the tests to the one or more modified portions (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover"; Para[0079], "the static analysis module 212 applies the second filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions having property changes in the changed code") Examiner Comments: Boshernitsan teaches analyzing the relationship between test mappings and modified portions of code by weighting tests according to how many changed functions they cover, supplementing Belur’s coverage-based reverse mapping with a finer-grained relationship analysis.
e. cause the subset of tests to be executed on the second set of program code (Para[0024], " A dynamic test process 116 runs multiple tests on the Src0 cod ") Examiner Comments: Belur explicitly teaches running the determined subset of tests on the modified source code.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 11, Belur and Boshernitsan teach
The system of Claim 10.
Boshernitsan further teaches
Wherein the test selection system is further configured to determine an ordered sequence for the subset of tests based on an estimated likelihood of failure or estimated execution time (Para[0078], "the greater the coverage score, the higher the priority assigned to the test. The prioritization module 294, therefore, configures the computer to apply Rule 1 which prioritizes the tests in order demo1, followed by demo2, followed by demo3") Examiner Comments: Boshernitsan teaches determining an ordered sequence for tests based on coverage scores, which correlate with estimated likelihood of failure since tests covering more changed code are more likely to detect faults.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 12, Belur and Boshernitsan teach
The system of Claim 11.
Belur further teaches
Wherein the test selection is further configured to: a. determine, based on the code coverage data, that a particular test does not exercise the first set of program code; and b. move the particular test to a suite of deprecated tests excluded from the subset likely to exercise the second set of program code (Para[0009], "If too many relevant tests are identified, then adequate code coverage may be obtained without running all relevant tests, and the testing application may limit the number of tests to run according to a blacklist"; Para[0009], "if no relevant tests or too few relevant tests are identified, then the testing application may add one or more default tests specified in a whitelist to improve code coverage") Examiner Comments: Belur’s system identifies tests that do or do not map to particular source code based on code coverage and excludes irrelevant tests via blacklisting, which teaches deprecating tests that do not exercise the first set of program code.
Regarding Claim 13, Belur (US 2016/0259713 A1) teaches
A computer-implemented method, comprising:
receiving a mapping of tests to a first set of program code, wherein the mapping is based on code coverage data generated from execution of the tests on the first set of program code and indicates portions of the first set exercised by respective tests (Para[0013], "Testing application 141 may then store the mapping of test A to file X, or the particular lines or methods in file X, and the associated code coverage information in database 130"; Abstract, "Using such results, mappings of the tests to the source code is created and stored") Examiner Comments: Belur teaches storing and thus making available (receiving) mappings of tests to a first set of program code based on code coverage data, where the mapping indicates particular lines or methods exercised by respective tests.
determining a subset of the tests likely to exercise a second set of program code comprising one or more modified portions of the first set of program code, based on:
(i) the mapping (Para[0014], "testing application 141 automatically retrieves a list of tests relevant to the source code file by querying database 130 for tests which map to the source code file, if any. That is, testing application 141 reverse maps the source code file (or lines of code) to relevant tests, based on the mappings stored in the database 130") Examiner Comments: Belur’s reverse mapping of modified source code to stored test mappings teaches determining a subset based on the mapping;
(ii) data indicating the one or more modified portions (Para[0014], "Such a source code file may be a new file or an existing file that has been modified by a developer") Examiner Comments: Belur identifies that the checked-in source code is modified, and uses this modification data to drive test selection.
Belur did not specifically teach
(iii) a determination of a relationship between the mapping of the tests to the one or more modified portions.
However, Boshernitsan (US 2015/0007140 A1) teaches
(iii) a determination of a relationship between the mapping of the tests to the one or more modified portions (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover"; Para[0079], "the static analysis module 212 applies the second filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions having property changes in the changed code") Examiner Comments: Boshernitsan teaches determining a relationship between the test-to-code mapping and the modified portions by weighting tests according to how many changed functions they cover.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 14, is a method claim corresponding to the system claim above (Claim 13) and, therefore, is rejected for the same reasons set forth in the rejection of claim 13.
Regarding Claim 15, is a computer-readable storage medium claim corresponding to the system claim above (Claim 10) and, therefore, is rejected for the same reasons set forth in the rejection of claim 10.
Regarding Claim 16, is a computer-readable storage medium claim corresponding to the system claim above (Claim 12) and, therefore, is rejected for the same reasons set forth in the rejection of claim 12.
Regarding Claim 17, Belur and Boshernitsan teach
The computer-readable medium of Claim 16.
Belur further teaches
Wherein the instructions are further executable to perform: a. storing data indicating portions of the first version exercised by the test; b. determining the portions are reverted to a prior version for which the test was exercised; and c. moving the test from the deprecated suite to the suite of tests (Belur, Para[0013], "the mapping need not be relearned each time the test is run on the source code, and testing application 142 may instead monitor learned tests and only relearn tests that have changed. That is, new tests or modified tests may be learned at a delta level") Examiner Comments: Belur teaches monitoring test mappings and relearning when changes occur; it would have been obvious to restore a previously deprecated test when code reverts to a version the test covered, since the code coverage data would again show the test exercises the code.
Regarding Claim 18, Belur (US 2016/0259713 A1) teaches
A computer-implemented method for continuous integration, continuous testing, and continuous delivery (CI/CT/CD) of software, the method comprising:
receiving a mapping of a suite of tests to a first set of program code, wherein data indicative of code coverage is generated using execution of the tests on the first set of program code (Abstract, "During a training phase, the agent intercepts tests being run on source code and provides a dump of the interception results after each test is executed"; Para[0013], "Testing application 141 may then store the mapping of test A to file X, or the particular lines or methods in file X, and the associated code coverage information in database 130") Examiner Comments: Belur teaches intercepting tests during a training phase and generating code coverage data, then storing the resulting test-to-code mappings, which teaches receiving a mapping of tests based on code coverage data;
determining a subset of the tests that are likely to exercise a second set of program code that comprises one or more modified portions of the first set of program code, based at least in part on the mapping and the data indicative of the one or more modified portions (Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system) to determine if a new version of a source code file is being checked in. Such a source code file may be a new file or an existing file that has been modified by a developer … testing application 141 automatically retrieves a list of tests relevant to the source code file by querying database 130 for tests which map to the source code file") Examiner Comments: Belur teaches detecting modified code in a CI system and selecting a subset of relevant tests by reverse-mapping from the modified code to stored test mappings.
executing the subset of the tests on the second set of program code (Para[0027], "testing application 141 runs the determined tests") Examiner Comments: Belur explicitly teaches running the determined subset of tests on the modified source code.
Belur did not specifically teach
a determination of a relationship between the mapping of the tests to the one or more modified portions.
However, Boshernitsan (US 2015/0007140 A1) teaches
a determination of a relationship between the mapping of the tests to the one or more modified portions (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover") Examiner Comments: Boshernitsan teaches determining a relationship between test mappings and modified code portions by weighting tests according to how many changed functions they cover;
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 19, is a method claim corresponding to the system claim above (Claim 12) and, therefore, is rejected for the same reasons set forth in the rejection of claim 12.
Regarding Claim 20, Belur and Boshernitsan teach
The method of Claim 19.
Belur, further teaches
Wherein the subset of the tests is determined based at least in part on user input representing a selection of tests for prior versions of the second set of program code (Belur, Para[0016], "a blacklists and/or whitelist may be employed to limit or increase the number of tests that are run, respectively"; Para[0027], "the blacklist may be added to configuration file 110 by a user") Examiner Comments: Belur teaches that a user configures blacklists/whitelists that define rules for which tests to run, constituting user input representing a test selection.
Regarding Claim 21, Belur (US 2016/0259713 A1) teaches
A system, comprising: one or more computing devices configured to implement a test selection system, wherein the test selection system is configured to:
execute a suite of tests on a first set of program code, wherein execution of a respective test from the suite generates data indicative of code coverage of the respective test with respect to the first set of program code (Abstract, "During a training phase, the agent intercepts tests being run on source code and provides a dump of the interception results after each test is executed"; Para[0013], "agent 122 maps portions of the source code indicated by the code coverage instrumentation to the test and reports the mapping to testing application 141. For example, if test A were run on source code file X, agent 122 may generate a dump of code coverage instrumentation results showing that test A affected particular lines or methods in source code file X") Examiner Comments: Belur’s training phase where the agent intercepts test runs and generates code coverage results for each test teaches executing a suite of tests that generates code coverage data;
store a mapping of the tests to the first set of program code, wherein the mapping is determined based at least in part on the data indicative of code coverage (Para[0013], "Testing application 141 may then store the mapping of test A to file X, or the particular lines or methods in file X, and the associated code coverage information in database 130") Examiner Comments: Belur explicitly teaches storing test-to-source-code mappings in a database based on collected code coverage data;
receive a second set of program code, wherein the second set of program code is associated with data indicative of one or more modified portions of the first set of program code (Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system) to determine if a new version of a source code file is being checked in. Such a source code file may be a new file or an existing file that has been modified by a developer") Examiner Comments: Belur teaches receiving a new/modified version of source code via CI system polling, where the modification data is inherently associated with the checked-in code;
select a subset of the tests that are likely to exercise the second set of program code based at least in part on the mapping and the data indicative of the one or more modified portions of the first set of program code (Para[0014], "testing application 141 automatically retrieves a list of tests relevant to the source code file by querying database 130 for tests which map to the source code file, if any. That is, testing application 141 reverse maps the source code file (or lines of code) to relevant tests, based on the mappings stored in the database 130") Examiner Comments: Belur teaches selecting a subset of tests by reverse-mapping from the modified source code to the stored test-to-code mappings;
cause the subset of the tests to be executed on the second set of program code (Para[0027], "testing application 141 runs the determined tests") Examiner Comments: Belur explicitly teaches running the determined subset of tests on the modified code;
further comprising a test execution module configured to execute selected tests across different environments and testing tools (Para[0012], "agent 122 is able to interpret instrumentation by a variety of code coverage instrumentation tools, with the particular type of code coverage instrumentation to interpret being specified in configuration file 110 … different types of agents 122 capable of interpreting respective coverage instrumentations may be copied to the IP address of testing system 120, based on the code coverage instrumentation tool type specified in configuration file 110") Examiner Comments: Belur’s support for multiple instrumentation types across different testing systems teaches executing tests across different environments; it would have been obvious to use mappings from one environment to inform test selection in another environment to improve overall coverage, as cross-environment testing is well-known in CI/CD.
Belur did not specifically teach
wherein said mapping is performed per test executed on code in a selected environment, and also according to at least one previous mapping for said code in a different environment.
However, Boshernitsan (US 2015/0007140 A1) teaches
wherein said mapping is performed per test executed on code in a selected environment, and also according to at least one previous mapping for said code in a different environment (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 22, Belur and Boshernitsan teach
The system of Claim 21. Claim 22 recites deprecated test limitations. Taught by Belur (Para[0009]) as discussed for Claim 12 above.
Regarding Claim 23, Belur and Boshernitsan teach
The system of Claim 22. Claim 23 recites user input test selection limitations. Taught by Belur (Para[0016] and [0027]) as discussed for Claim 20 above.
Regarding Claim 24, Belur (US 2016/0259713 A1) teaches
A non-transitory computer-readable storage medium storing program instructions computer-executable to perform:
executing a suite of tests on a first version of program code, wherein execution of a respective test from the suite generates data indicative of code coverage of the respective test with respect to the program code (Abstract, "During a training phase, the agent intercepts tests being run on source code and provides a dump of the interception results after each test is executed"; Para[0013], "agent 122 may generate a dump of code coverage instrumentation results showing that test A affected particular lines or methods in source code file X") Examiner Comments: Belur’s training phase teaches executing tests on source code where each test generates code coverage data for the respective test;
storing a mapping of the tests to the program code, wherein the mapping is determined based at least in part on the data indicative of code coverage (Para[0013], "Testing application 141 may then store the mapping of test A to file X, or the particular lines or methods in file X, and the associated code coverage information in database 130") Examiner Comments: Belur explicitly teaches storing mappings of tests to source code in a database based on the collected code coverage information;
receiving data indicative of one or more modified portions of a second version of the program code, wherein said one or more modified portions are modified from said first version of the program code (Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system) to determine if a new version of a source code file is being checked in. Such a source code file may be a new file or an existing file that has been modified by a developer") Examiner Comments: Belur teaches receiving a modified version of source code via CI polling, where the new version inherently carries data indicating which portions were modified from the prior version;
determining a subset of the tests that are likely to exercise the second version of the program code (Para[0014], "testing application 141 automatically retrieves a list of tests relevant to the source code file by querying database 130 for tests which map to the source code file, if any. That is, testing application 141 reverse maps the source code file (or lines of code) to relevant tests, based on the mappings stored in the database 130"; Para[0023], "database 130 is populated with mappings of source code to tests and associated code coverage during a learning phase, with the mappings being at, e.g., the source code file, method, or source code line level of granularity") Examiner Comments: Belur teaches determining a subset of relevant tests by querying stored mappings based on the modified source code, using both the mapping data and the identification of modified portions;
initiating execution of the subset of the tests on the second version of the program code (Para[0027], "testing application 141 runs the determined tests") Examiner Comments: Belur explicitly teaches initiating execution of the selected subset of tests on the modified source code.
Belur did not specifically teach
based at least in part on the mapping and the data indicative of the one or more modified portions of the second version of the program code.
However, Boshernitsan (US 2015/0007140 A1) teaches
based at least in part on the mapping and the data indicative of the one or more modified portions of the second version of the program code (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 25, Belur and Boshernitsan teach
The Computer-readable storage medium of Claim 24. Claim 25 recites deprecated test limitations. Taught by Belur (Para[0009]) as discussed for Claim 12 above.
Regarding Claim 26, Belur and Boshernitsan teach
The Computer-readable storage medium of Claim 25. Claim 26 recites reversion logic (same as Claim 17). Taught by Belur (Para[0013]) as discussed for Claim 17 above.
Regarding Claim 27, Belur and Boshernitsan teach
The Computer-readable storage medium of Claim 25.
Belur further teaches
Wherein in determining the subset of the tests that are likely to exercise the second version of the program code, the program instructions are further computer executable to perform determining confidence scores for the tests, wherein the confidence scores indicate a likelihood of exercising the second version of the program code; and determining that the confidence scores for the subset of the tests meet a confidence threshold and that the confidence scores for other ones of the tests do not meet the confidence threshold (Belur, Para[0024], "Testing application 141 may instead query for relevant tests matching any feasible criteria, such as having some desired code coverage or cyclomatic dependency score"; Para[0016], "the blacklist may specify that when a file is touched by greater than 80% of the total tests, then a subset of those test needed to reach 100% coverage or some fixed number (e.g., 10) of tests with the greatest code coverage should be used") Examiner Comments: Belur teaches filtering tests based on code coverage scores (confidence scores) and applying thresholds (80%, 100% coverage, top 10) to determine which tests make it into the subset, directly teaching the confidence score and threshold limitation.
Regarding Claim 28, Belur and Boshernitsan teach
The Computer-readable storage medium of Claim 25.
Belur further teaches
Wherein the subset of the tests are executed using a continuous integration system that tests the second version of the program code (Belur, Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system)") Examiner Comments: Belur explicitly teaches integration with a continuous integration system.
Regarding Claim 29, Belur (US 2016/0259713 A1) teaches
A method for automatic build discovery in a CI/CT/CD system, the method comprising:
monitoring a distributed testing environment to detect a build version of running components (Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system) to determine if a new version of a source code file is being checked in") Examiner Comments: Belur teaches polling a CI system to detect when new versions of code are checked in, which constitutes monitoring a testing environment to detect build versions;
automatically identifying changes in the build version (Para[0014], "Such a source code file may be a new file or an existing file that has been modified by a developer") Examiner Comments: Belur’s system automatically identifies that the checked-in code is new or modified;
Belur did not specifically teach
updating a test selection system based on the identified changes to select tests relevant to the new build version.
However, Boshernitsan (US 2015/0007140 A1) teaches
updating a test selection system based on the identified changes to select tests relevant to the new build version (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover"; Para[0079], "the static analysis module 212 applies the second filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions having property changes in the changed code")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 30, Belur and Boshernitsan teach
The method of Claim 29.
Belur further teaches
Wherein the automatic identification of changes includes analyzing messages regarding test execution to determine which server or service is associated with a particular test (Belur, Para[0015]–[0016], Para[0018], "agent 122 sends a signal (e.g., a TCP/IP message) reporting the gathered information and the test run to testing application 141"; Para [0011], "Configuration file 110 further includes a pointer 111 to an Internet Protocol (IP) address of testing system 120").
Regarding Claim 31, Belur and Boshernitsan teach
The method of Claim 30.
the automatic identification of changes includes analyzing build-diff data and matching with test footprints (Belur, Para[0021]–[0023]; Boshernitsan, Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover") Examiner Comments: Belur identifies code changes and matches them to test mappings (footprints), and Boshernitsan teaches analyzing changed functions and matching test coverage to those changes.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 32, Belur teaches
A method for statistically determining a relative importance of running a particular test within a suite of tests, the method comprising performing the following functions with regard to the system according to claim 1:
mapping a test method to a list of methods or functions executed upon execution of the test (Belur, Para[0013], "agent 122 may generate a dump of code coverage instrumentation results showing that test A affected particular lines or methods in source code file X") Examiner Comments: Belur teaches mapping a test to the methods/functions of the code that are executed when the test runs.
Belur did not specificialy teach
analyzing the mapping to determine a relative likelihood of failure for the test
prioritizing the test based on the determined relative likelihood of failure.
However, Boshernitsan teaches
analyzing the mapping to determine a relative likelihood of failure for the test (Boshernitsan, Para[0077], "weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover") Examiner Comments: Boshernitsan’s analysis of how many changed functions a test covers directly determines a relative likelihood that the test will find failures;
prioritizing the test based on the determined relative likelihood of failure (Boshernitsan, Para[0078], "the greater the coverage score, the higher the priority assigned to the test") Examiner Comments: Boshernitsan explicitly teaches prioritizing tests based on coverage scores correlating with likelihood of detecting failures.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 33, Belur, and Boshernitsan teach
The method of Claim 32.
Boshernitsan further teaches
Wherein the prioritization includes determining whether the test is to be run at a particular time or in a particular order within the suite of tests (Boshernitsan, Para[0078], "The prioritization module 294, therefore, configures the computer to apply Rule 1 which prioritizes the tests in order demo1, followed by demo2, followed by demo3") Examiner Comments: Boshernitsan explicitly teaches determining a particular order in which tests should be run within the suite.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMIR SOLTANZADEH whose telephone number is (571)272-3451. The examiner can normally be reached M-F, 9am - 5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wei Mui can be reached at (571) 272-3708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMIR SOLTANZADEH/Examiner, Art Unit 2191
/WEI Y MUI/Supervisory Patent Examiner, Art Unit 2191
Prosecution Insights