Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-33 are presented for examination.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “wherein the test selection system is configured to…” in claim 10, “wherein the test selection system is further configured to” in claim 11-12, “wherein the test selection system is configured to …” Claim 21, “wherein the test selection system is further configured to …”.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9, 13-14, 29-33 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Claims 1 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “determining a relative importance of a selected test in a plurality of tests for testing code”, “determine the relative importance of the selected test”, “determining a relationship between the selected test and the code that is tested according to a statistical analysis, a machine learning algorithm or a combination thereof”, “wherein when said characteristics relate to an importance of said code said characteristics are selected from the group consisting of interactions between a section of the code and other sections, an importance of the code overall for execution in production, code with many changes or new commits, and otherwise modified code; and e. wherein when said characteristics relate to an importance of said plurality of tests, said characteristics are selected from the group consisting of tests that apply to code that is deemed important, tests that have failed previously, a new test, and a test that is testing modified code” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
This judicial exception is not integrated into a practical application. The claims recites the following additional elements “a computational device,” “said computational device comprising a memory for storing instructions and a processor for executing said stored instructions,” “an analyzer executed by said processor,” “a machine learning algorithm” and “receiving one or more characteristics relating to an importance of said code, an importance of each of said plurality of tests, or both”. The additional elements “a computational device,” “said computational device comprising a memory for storing instructions and a processor for executing said stored instructions,” “an analyzer executed by said processor,” “a machine learning algorithm” are merely instructions to implement an abstract idea on a computer, or merely using a generic computer or computer components as a tool to perform the abstract idea. See MPEP 2106.05(f). The additional element “receiving one or more characteristics relating to an importance of said code, an importance of each of said plurality of tests, or both” does nothing more than add insignificant extra solution activity to the judicial exception, such as data gathering and outputting the results of the abstract idea to perform a task. See MPEP 2106.05(g). Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea, thus fail to integrate the abstract idea into a practical application.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element “a computational device,” “said computational device comprising a memory for storing instructions and a processor for executing said stored instructions,” “an analyzer executed by said processor,” “a machine learning algorithm” are generic computer components and instructions used as the tools to perform the abstract idea. See MPEP 2106.05(f). As to the additional element “receiving one or more characteristics relating to an importance of said code, an importance of each of said plurality of tests, or both” the courts have identified gathering data and displaying the output of the abstract idea is well-understood, routine, conventional activity. See MPEP 2106.05(d). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claims are not patent eligible.
Claim 2 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein an importance of the selected test in relation to said code is determined at least partially according to a change in environment, environment variables and characteristics, a change in a configuration file, or a change in an asset” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 3 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein said statistical analysis determines a statistical relationship between said selected test and the code according to a timing of execution of said selected test in relation to a time of execution of a particular section of code” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 4 recites the following additional elements “analyzer further comprises a plurality of machine learning algorithms and an ensemble learning algorithm for determining said relationship” which are merely instructions to implement an abstract idea on a computer, or merely using a generic computer or computer components as a tool to perform the abstract idea. See MPEP 2106.05(f). Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea, thus fail to integrate the abstract idea into a practical application.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element are generic computer components and instructions used as the tools to perform the abstract idea. See MPEP 2106.05(f). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claims are not patent eligible.
Claim 5 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “a mapper for determining a relationship between one or more code components and said plurality of tests to be executed” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 6 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein said mapper determines said relationship according to a previous history of execution of said plurality of tests or according to an analysis of a relationship between said plurality of tests and execution of said code” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 7 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein said relationship is determined according to a timing of execution of said code and a timing margin, wherein said timing margin comprises an additional amount of time for accepting a test execution as being related to said code execution” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 8 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein said relationship is further determined according to a relative degree of sensitivity versus accuracy” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim 9 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein the user determines a rule for test importance or running the selected test” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claims 13 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “determining a subset of the tests likely to exercise a second set of program code comprising one or more modified portions of the first set of program code, based on:i. the mapping, ii. data indicating the one or more modified portions, and iii. a determination of a relationship between the mapping of the tests to the one or more modified portions” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
This judicial exception is not integrated into a practical application. The claims recites the following additional elements “a computer-implemented” and “receiving a mapping of tests to a first set of program code, wherein the mapping is based on code coverage data generated from execution of the tests on the first set of program code and indicates portions of the first set exercised by respective tests”. The additional elements “a computer-implemented” are merely instructions to implement an abstract idea on a computer, or merely using a generic computer or computer components as a tool to perform the abstract idea. See MPEP 2106.05(f). The additional element “receiving a mapping of tests to a first set of program code, wherein the mapping is based on code coverage data generated from execution of the tests on the first set of program code and indicates portions of the first set exercised by respective tests” does nothing more than add insignificant extra solution activity to the judicial exception, such as data gathering and outputting the results of the abstract idea to perform a task. See MPEP 2106.05(g). Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea, thus fail to integrate the abstract idea into a practical application.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element “a computer-implemented” are generic computer components and instructions used as the tools to perform the abstract idea. See MPEP 2106.05(f). As to the additional element “receiving a mapping of tests to a first set of program code, wherein the mapping is based on code coverage data generated from execution of the tests on the first set of program code and indicates portions of the first set exercised by respective tests” the courts have identified gathering data and displaying the output of the abstract idea is well-understood, routine, conventional activity. See MPEP 2106.05(d). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claims are not patent eligible.
Claims 14 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “determining, based on the code coverage data, that a particular test does not exercise the first set of program code; and b. moving the particular test to a suite of deprecated tests excluded from the subset likely to exercise the second set of program code” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claims 29 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “monitoring a distributed testing environment to detect a build version of running components; automatically identifying changes in the build version; and updating a test selection system based on the identified changes to select tests relevant to the new build version” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
This judicial exception is not integrated into a practical application. The claims recites the following additional elements “automatic build discovery in a CI/CT/CD system,” which are merely instructions to implement an abstract idea on a computer, or merely using a generic computer or computer components as a tool to perform the abstract idea. See MPEP 2106.05(f). Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea, thus fail to integrate the abstract idea into a practical application.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element are generic computer components and instructions used as the tools to perform the abstract idea. See MPEP 2106.05(f). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claims are not patent eligible.
Claims 30 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “analyzing messages regarding test execution to determine which server or service is associated with a particular test” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claims 31 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “wherein the automatic identification of changes includes analyzing build-diff data and matching with test footprints” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claims 32 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “mapping a test method to a list of methods or functions executed upon execution of the test; analyzing the mapping to determine a relative likelihood of failure for the test; and prioritizing the test based on the determined relative likelihood of failure” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claims 33 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitation “mapping a test method to a list of methods or functions executed upon execution of the test; analyzing the mapping to determine a relative likelihood of failure for the test; and prioritizing the test based on the determined relative likelihood of failure” as drafted, is a process that, under its broadest reasonable interpretation, recite the abstract idea of mental processes. These limitations encompass a human mind carrying out these functions through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Belur (US 2016/0259713 A1) in view of Boshernitsan (US 2015/0007140 A1) further in view of Deng (US 8,276,123 B1).
Regarding Claim 1, Belur (US 2016/0259713 A1) teaches
A system for determining a relative importance of a selected test in a plurality of tests for testing code, comprising:
a. a computational device for receiving one or more characteristics relating to an importance of said code, an importance of each of said plurality of tests, or both (Para[0023], "Testing application 141 may then query database 130 to retrieve the list of relevant tests which map back to the source code, as well as code coverage associated with those tests. Testing application 141 may instead query for relevant tests matching any feasible criteria, such as having some desired code coverage or cyclomatic dependency score") Examiner Comments: Belur’s testing application is a computational device that receives test relevance characteristics (code coverage and cyclomatic dependency scores) relating to the importance of the tests with respect to the code;
b. said computational device comprising a memory for storing instructions and a processor for executing said stored instructions to determine the relative importance of the selected test (Para[0029]–[00331], "system 400 includes, without limitation, a central processing unit (CPU) 410... a memory 460" and "CPU 410 retrieves and executes programming instructions stored in the memory 460") Examiner Comments: Belur explicitly teaches a computing system with CPU and memory that executes instructions
wherein when said characteristics relate to an importance of said plurality of tests, said characteristics are selected from the group consisting of tests that apply to code that is deemed important, tests that have failed previously, a new test, and a test that is testing modified code (Belur, Para[0020]–[0021], "new tests or modified tests may be learned at a delta level" and "testing application 141 automatically retrieves a list of tests relevant to the source code file") Examiner Comments: Belur teaches identifying tests relevant to modified source code and recognizing new/modified tests, which are characteristics relating to test importance from the claimed group.
Belur did not specifically teach
c. an analyzer executed by said processor for determining a relationship between the selected test and the code that is tested according to a statistical analysis, a machine learning algorithm or a combination thereof
wherein when said characteristics relate to an importance of said code, said characteristics are selected from the group consisting of interactions between a section of the code and other sections, an importance of the code overall for execution in production, code with many changes or new commits, and otherwise modified code.
However, Boshernitsan teaches
wherein when said characteristics relate to an importance of said code, said characteristics are selected from the group consisting of interactions between a section of the code and other sections, an importance of the code overall for execution in production, code with many changes or new commits, and otherwise modified code (Para[0063], "weights tests... according to the number of functions having property changes in the changed code") Examiner Comments: Boshernitsan teaches weighting based on property changes in modified code, together teaching characteristics from the claimed group;
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Belur and Boshernitsan did not specifically teach
an analyzer executed by said processor for determining a relationship between the selected test and the code that is tested according to a statistical analysis, a machine learning algorithm or a combination thereof.
However, Deng (US 8,276,123 B1) teaches
an analyzer executed by said processor for determining a relationship between the selected test and the code that is tested according to a statistical analysis, a machine learning algorithm or a combination thereof (Col. 16, line 21–31, "Intelligent RTS module 22 may implement an adaptive-combo test selection algorithm that may adaptively apply one or more, e.g., a combination or combo of, test selection algorithms to select the subset of test cases") Examiner Comments: Deng’s intelligent RTS module serves as an analyzer that uses adaptive algorithms (analogous to statistical analysis) to determine relationships between tests and code based on coverage, dependency, and code change data.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur and Boshernitsan’s teaching with Deng’s in order to apply adaptive and statistical analysis algorithms to the test-to-code mapping data, as Deng’s intelligent RTS module provides a well-known approach for refining test selection beyond simple coverage mapping by adaptively combining multiple selection algorithms.
Regarding Claim 2, Belur, Boshernitsan and Deng teach
The system of Claim 1.
Boshernitsan further teaches
Wherein an importance of the selected test in relation to said code is determined at least partially according to a change in environment, environment variables and characteristics, a change in a configuration file, or a change in an asset (Boshernitsan, Para[0063], "weights tests... according to the number of functions having property changes in the changed code") Examiner Comments: Property changes in code functions are analogous to changes in configuration or assets that affect test importance.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 3, Belur, Boshernitsan and Deng teach
The system of Claim 1.
Belur further teaches
Wherein said statistical analysis determines a statistical relationship between said selected test and the code according to a timing of execution of said selected test in relation to a time of execution of a particular section of code (Belur, Para[0012], "the instrumented instructions provide, e.g., counters indicating the number of times particular lines of code, methods, etc. have been executed")Examiner Comments: Belur’s agent correlates test execution with code execution through instrumentation counters that track when particular code sections execute during test runs, establishing a timing-based statistical relationship.
Regarding Claim 4, Belur, Boshernitsan and Deng teach
The system of Claim 1.
Deng further teaches
Wherein said analyzer further comprises a plurality of machine learning algorithms and an ensemble learning algorithm for determining said relationship (Deng, Col. 16, line 21–31, "Intelligent RTS module 22 may implement an adaptive-combo test selection algorithm that may adaptively apply one or more, e.g., a combination or combo of, test selection algorithms to select the subset of test cases") Examiner Comments: Deng’s adaptive-combo algorithm combines multiple test selection algorithms, which is analogous to an ensemble learning approach.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur and Boshernitsan’s teaching with Deng’s in order to apply adaptive and statistical analysis algorithms to the test-to-code mapping data, as Deng’s intelligent RTS module provides a well-known approach for refining test selection beyond simple coverage mapping by adaptively combining multiple selection algorithms.
Regarding Claim 5, Belur, Boshernitsan and Deng teach
The system of Claim 1.
Belur further teaches
Further comprising a mapper for determining a relationship between one or more code components and said plurality of tests to be executed (Belur, Para[0013], "agent 122 maps portions of the source code indicated by the code coverage instrumentation to the test and reports the mapping to testing application 141"; Para[0023], "the mappings being at, e.g., the source code file, method, or source code line level of granularity") Examiner Comments: Belur’s agent/testing application serves as a mapper determining the relationship between code components (files, methods, lines) and tests.
Regarding Claim 6, Belur, Boshernitsan and Deng teach
The system of Claim 5.
Belur further teaches
Wherein said mapper determines said relationship according to a previous history of execution of said plurality of tests or according to an analysis of a relationship between said plurality of tests and execution of said code (Belur, Para[0013], "the mapping need not be relearned each time the test is run on the source code, and testing application 142 may instead monitor learned tests and only relearn tests that have changed") Examiner Comments: Belur’s system builds and maintains a history of test-to-code mappings from previous test executions and uses this history to determine test relevance.
Regarding Claim 7, Belur, Boshernitsan and Deng teach
The system of Claim 6.
Belur further teaches
Wherein said relationship is determined according to a timing of execution of said code and a timing margin, wherein said timing margin comprises an additional amount of time for accepting a test execution as being related to said code execution (Belur, Para[0012], "the instrumented instructions provide, e.g., counters indicating the number of times particular lines of code, methods, etc. have been executed") Examiner Comments: Belur’s runtime instrumentation correlates test execution with code execution through timing; it would have been obvious to incorporate a timing margin for asynchronous execution, as this is well-known in dynamic code analysis.
Regarding Claim 8, Belur, Boshernitsan and Deng teach
The system of Claim 7.
Deng further teaches
Wherein said relationship is further determined according to a relative degree of sensitivity versus accuracy (Deng, Abstract, "a computing device may include a user interface with which a user interacts to specify a safety level … adaptively applying these algorithms in order of decreasing safety assurances and increasing precision") Examiner Comments: Deng’s safety level specification and adaptive application of algorithms trading off safety assurances versus precision directly teaches determining a relationship according to sensitivity versus accuracy.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur and Boshernitsan’s teaching with Deng’s in order to apply adaptive and statistical analysis algorithms to the test-to-code mapping data, as Deng’s intelligent RTS module provides a well-known approach for refining test selection beyond simple coverage mapping by adaptively combining multiple selection algorithms.
Regarding Claim 9, Belur, Boshernitsan and Deng teach
The system of Claim 1.
Belur further teaches
Wherein a user determines a rule for test importance or running the selected test (Belur, Para[0016], "a blacklists and/or whitelist may be employed to limit or increase the number of tests that are run"; Para [0027], "the blacklist may be added to configuration file 110 by a user") Examiner Comments: Belur teaches that the user configures blacklists/whitelists that define rules for which tests to run or exclude, constituting user-determined rules for test importance.
Claim(s) 10-33 is/are rejected under 35 U.S.C. 103 as being unpatentable over Belur (US 2016/0259713 A1) in view of Boshernitsan (US 2015/0007140 A1).
Regarding Claim 10, Belur (US 2016/0259713 A1) teaches
A test selection system comprising one or more computing devices, wherein the test selection system is configured to:
a. execute a set of a plurality of tests on a first set of program code, wherein execution of each respective test generates data indicative of code coverage of the respective test (Abstract, "During a training phase, the agent intercepts tests being run on source code and provides a dump of the interception results after each test is executed"; Para[0013], "agent 122 maps portions of the source code indicated by the code coverage instrumentation to the test and reports the mapping to testing application 141. For example, if test A were run on source code file X, agent 122 may generate a dump of code coverage instrumentation results showing that test A affected particular lines or methods in source code file X") Examiner Comments: Belur’s training phase, where an agent intercepts tests and generates code coverage results for each test, teaches executing tests on a first set of code where each test generates code coverage data;
b. store a mapping of the tests to the first set of program code based on the data indicative of code coverage, wherein the mapping comprises data indicating portions of the first set of program code exercised by said respective tests (Para[0013], "Testing application 141 may then store the mapping of test A to file X, or the particular lines or methods in file X, and the associated code coverage information in database 130") Examiner Comments: Belur explicitly teaches storing mappings of tests to source code in a database, where the mappings include which lines or methods are exercised by respective tests;
c. receive a second set of program code, comprising modifications to said first set of program code, associated with data indicating one or more modified portions (Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system) to determine if a new version of a source code file is being checked in. Such a source code file may be a new file or an existing file that has been modified by a developer") Examiner Comments: Belur teaches detecting when a modified version of source code is checked in, where the modifications constitute the one or more modified portions;
d. select a subset of the tests likely to exercise the second set of program code based on (i) the mapping of the tests to the first set of program code, and (ii) the data indicating the one or more modified portions (Para[0014], "testing application 141 automatically retrieves a list of tests relevant to the source code file by querying database 130 for tests which map to the source code file, if any. That is, testing application 141 reverse maps the source code file (or lines of code) to relevant tests, based on the mappings stored in the database 130") Examiner Comments: Belur’s reverse mapping from modified source code file to stored test mappings teaches selecting a subset based on the mapping and the modified portions.
Belur did not specifically teach
(iii) a determination of a relationship between the mapping of the tests to the one or more modified portions
cause the subset of tests to be executed on the second set of program code.
However, Boshernitsan (US 2015/0007140 A1) teaches
a determination of a relationship between the mapping of the tests to the one or more modified portions (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover"; Para[0079], "the static analysis module 212 applies the second filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions having property changes in the changed code") Examiner Comments: Boshernitsan teaches analyzing the relationship between test mappings and modified portions of code by weighting tests according to how many changed functions they cover, supplementing Belur’s coverage-based reverse mapping with a finer-grained relationship analysis.
e. cause the subset of tests to be executed on the second set of program code (Para[0024], " A dynamic test process 116 runs multiple tests on the Src0 cod ") Examiner Comments: Belur explicitly teaches running the determined subset of tests on the modified source code.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 11, Belur and Boshernitsan teach
The system of Claim 10.
Boshernitsan further teaches
Wherein the test selection system is further configured to determine an ordered sequence for the subset of tests based on an estimated likelihood of failure or estimated execution time (Para[0078], "the greater the coverage score, the higher the priority assigned to the test. The prioritization module 294, therefore, configures the computer to apply Rule 1 which prioritizes the tests in order demo1, followed by demo2, followed by demo3") Examiner Comments: Boshernitsan teaches determining an ordered sequence for tests based on coverage scores, which correlate with estimated likelihood of failure since tests covering more changed code are more likely to detect faults.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 12, Belur and Boshernitsan teach
The system of Claim 11.
Belur further teaches
Wherein the test selection is further configured to: a. determine, based on the data indicative of code coverage data, that a particular test does not exercise the first set of program code; and b. move the particular test to a suite of deprecated tests excluded from the subset likely to exercise the second set of program code (Para[0009], "If too many relevant tests are identified, then adequate code coverage may be obtained without running all relevant tests, and the testing application may limit the number of tests to run according to a blacklist"; Para[0009], "if no relevant tests or too few relevant tests are identified, then the testing application may add one or more default tests specified in a whitelist to improve code coverage") Examiner Comments: Belur’s system identifies tests that do or do not map to particular source code based on code coverage and excludes irrelevant tests via blacklisting, which teaches deprecating tests that do not exercise the first set of program code.
Regarding Claim 13, Belur (US 2016/0259713 A1) teaches
A computer-implemented method, comprising:
receiving a mapping of tests to a first set of program code, wherein the mapping is based on code coverage data generated from execution of the tests on the first set of program code and indicates portions of the first set exercised by respective tests (Para[0013], "Testing application 141 may then store the mapping of test A to file X, or the particular lines or methods in file X, and the associated code coverage information in database 130"; Abstract, "Using such results, mappings of the tests to the source code is created and stored") Examiner Comments: Belur teaches storing and thus making available (receiving) mappings of tests to a first set of program code based on code coverage data, where the mapping indicates particular lines or methods exercised by respective tests.
determining a subset of the tests likely to exercise a second set of program code comprising one or more modified portions of the first set of program code, based on:
(i) the mapping (Para[0014], "testing application 141 automatically retrieves a list of tests relevant to the source code file by querying database 130 for tests which map to the source code file, if any. That is, testing application 141 reverse maps the source code file (or lines of code) to relevant tests, based on the mappings stored in the database 130") Examiner Comments: Belur’s reverse mapping of modified source code to stored test mappings teaches determining a subset based on the mapping;
(ii) data indicating the one or more modified portions (Para[0014], "Such a source code file may be a new file or an existing file that has been modified by a developer") Examiner Comments: Belur identifies that the checked-in source code is modified, and uses this modification data to drive test selection.
Belur did not specifically teach
(iii) a determination of a relationship between the mapping of the tests to the one or more modified portions.
However, Boshernitsan (US 2015/0007140 A1) teaches
(iii) a determination of a relationship between the mapping of the tests to the one or more modified portions (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover"; Para[0079], "the static analysis module 212 applies the second filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions having property changes in the changed code") Examiner Comments: Boshernitsan teaches determining a relationship between the test-to-code mapping and the modified portions by weighting tests according to how many changed functions they cover.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 14, is a method claim corresponding to the system claim above (Claim 13) and, therefore, is rejected for the same reasons set forth in the rejection of claim 13.
Regarding Claim 15, is a computer-readable storage medium claim corresponding to the system claim above (Claim 10) and, therefore, is rejected for the same reasons set forth in the rejection of claim 10.
Regarding Claim 16, is a computer-readable storage medium claim corresponding to the system claim above (Claim 12) and, therefore, is rejected for the same reasons set forth in the rejection of claim 12.
Regarding Claim 17, Belur and Boshernitsan teach
The computer-readable medium of Claim 16.
Belur further teaches
Wherein the instructions are further executable to perform: a. storing data indicating portions of the first version exercised by the test; b. determining the portions are reverted to a prior version for which the test was exercised; and c. moving the test from the deprecated suite to the suite of tests (Belur, Para[0013], "the mapping need not be relearned each time the test is run on the source code, and testing application 142 may instead monitor learned tests and only relearn tests that have changed. That is, new tests or modified tests may be learned at a delta level") Examiner Comments: Belur teaches monitoring test mappings and relearning when changes occur; it would have been obvious to restore a previously deprecated test when code reverts to a version the test covered, since the code coverage data would again show the test exercises the code.
Regarding Claim 18, Belur (US 2016/0259713 A1) teaches
A computer-implemented method for continuous integration, continuous testing, and continuous delivery (CI/CT/CD) of software, the method comprising:
receiving a mapping of a suite of tests to a first set of program code, wherein data indicative of code coverage is generated using execution of the tests on the first set of program code (Abstract, "During a training phase, the agent intercepts tests being run on source code and provides a dump of the interception results after each test is executed"; Para[0013], "Testing application 141 may then store the mapping of test A to file X, or the particular lines or methods in file X, and the associated code coverage information in database 130") Examiner Comments: Belur teaches intercepting tests during a training phase and generating code coverage data, then storing the resulting test-to-code mappings, which teaches receiving a mapping of tests based on code coverage data;
determining a subset of the tests that are likely to exercise a second set of program code that comprises one or more modified portions of the first set of program code, based at least in part on the mapping and the data indicative of the one or more modified portions (Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system) to determine if a new version of a source code file is being checked in. Such a source code file may be a new file or an existing file that has been modified by a developer … testing application 141 automatically retrieves a list of tests relevant to the source code file by querying database 130 for tests which map to the source code file") Examiner Comments: Belur teaches detecting modified code in a CI system and selecting a subset of relevant tests by reverse-mapping from the modified code to stored test mappings.
executing the subset of the tests on the second set of program code (Para[0027], "testing application 141 runs the determined tests") Examiner Comments: Belur explicitly teaches running the determined subset of tests on the modified source code.
Belur did not specifically teach
a determination of a relationship between the mapping of the tests to the one or more modified portions.
However, Boshernitsan (US 2015/0007140 A1) teaches
a determination of a relationship between the mapping of the tests to the one or more modified portions (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover") Examiner Comments: Boshernitsan teaches determining a relationship between test mappings and modified code portions by weighting tests according to how many changed functions they cover;
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 19, is a method claim corresponding to the system claim above (Claim 12) and, therefore, is rejected for the same reasons set forth in the rejection of claim 12.
Regarding Claim 20, Belur and Boshernitsan teach
The method of Claim 19.
Belur, further teaches
Wherein the subset of the tests is determined based at least in part on user input representing a selection of tests for prior versions of the second set of program code (Belur, Para[0016], "a blacklists and/or whitelist may be employed to limit or increase the number of tests that are run, respectively"; Para[0027], "the blacklist may be added to configuration file 110 by a user") Examiner Comments: Belur teaches that a user configures blacklists/whitelists that define rules for which tests to run, constituting user input representing a test selection.
Regarding Claim 21, Belur (US 2016/0259713 A1) teaches
A system, comprising: one or more computing devices configured to implement a test selection system, wherein the test selection system is configured to:
execute a suite of tests on a first set of program code, wherein execution of a respective test from the suite generates data indicative of code coverage of the respective test with respect to the first set of program code (Abstract, "During a training phase, the agent intercepts tests being run on source code and provides a dump of the interception results after each test is executed"; Para[0013], "agent 122 maps portions of the source code indicated by the code coverage instrumentation to the test and reports the mapping to testing application 141. For example, if test A were run on source code file X, agent 122 may generate a dump of code coverage instrumentation results showing that test A affected particular lines or methods in source code file X") Examiner Comments: Belur’s training phase where the agent intercepts test runs and generates code coverage results for each test teaches executing a suite of tests that generates code coverage data;
store a mapping of the tests to the first set of program code, wherein the mapping is determined based at least in part on the data indicative of code coverage (Para[0013], "Testing application 141 may then store the mapping of test A to file X, or the particular lines or methods in file X, and the associated code coverage information in database 130") Examiner Comments: Belur explicitly teaches storing test-to-source-code mappings in a database based on collected code coverage data;
receive a second set of program code, wherein the second set of program code is associated with data indicative of one or more modified portions of the first set of program code (Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system) to determine if a new version of a source code file is being checked in. Such a source code file may be a new file or an existing file that has been modified by a developer") Examiner Comments: Belur teaches receiving a new/modified version of source code via CI system polling, where the modification data is inherently associated with the checked-in code;
select a subset of the tests that are likely to exercise the second set of program code based at least in part on the mapping and the data indicative of the one or more modified portions of the first set of program code (Para[0014], "testing application 141 automatically retrieves a list of tests relevant to the source code file by querying database 130 for tests which map to the source code file, if any. That is, testing application 141 reverse maps the source code file (or lines of code) to relevant tests, based on the mappings stored in the database 130") Examiner Comments: Belur teaches selecting a subset of tests by reverse-mapping from the modified source code to the stored test-to-code mappings;
cause the subset of the tests to be executed on the second set of program code (Para[0027], "testing application 141 runs the determined tests") Examiner Comments: Belur explicitly teaches running the determined subset of tests on the modified code;
further comprising a test execution module configured to execute selected tests across different environments and testing tools (Para[0012], "agent 122 is able to interpret instrumentation by a variety of code coverage instrumentation tools, with the particular type of code coverage instrumentation to interpret being specified in configuration file 110 … different types of agents 122 capable of interpreting respective coverage instrumentations may be copied to the IP address of testing system 120, based on the code coverage instrumentation tool type specified in configuration file 110") Examiner Comments: Belur’s support for multiple instrumentation types across different testing systems teaches executing tests across different environments; it would have been obvious to use mappings from one environment to inform test selection in another environment to improve overall coverage, as cross-environment testing is well-known in CI/CD.
Belur did not specifically teach
wherein said mapping is performed per test executed on the second set of program code in a selected environment, and also according to at least one previous mapping for said second set of program code in a different environment.
However, Boshernitsan (US 2015/0007140 A1) teaches
wherein said mapping is performed per test executed on the second set of program code in a selected environment, and also according to at least one previous mapping for said second set of program code in a different environment (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 22, Belur and Boshernitsan teach
The system of Claim 21. Claim 22 recites deprecated test limitations. Taught by Belur (Para[0009]) as discussed for Claim 12 above.
Regarding Claim 23, Belur and Boshernitsan teach
The system of Claim 22. Claim 23 recites user input test selection limitations. Taught by Belur (Para[0016] and [0027]) as discussed for Claim 20 above.
Regarding Claim 24, Belur (US 2016/0259713 A1) teaches
A non-transitory computer-readable storage medium storing program instructions computer-executable to perform:
executing a suite of tests on a first version of program code, wherein execution of a respective test from the suite generates data indicative of code coverage of the respective test with respect to the program code (Abstract, "During a training phase, the agent intercepts tests being run on source code and provides a dump of the interception results after each test is executed"; Para[0013], "agent 122 may generate a dump of code coverage instrumentation results showing that test A affected particular lines or methods in source code file X") Examiner Comments: Belur’s training phase teaches executing tests on source code where each test generates code coverage data for the respective test;
storing a mapping of the tests to the program code, wherein the mapping is determined based at least in part on the data indicative of code coverage (Para[0013], "Testing application 141 may then store the mapping of test A to file X, or the particular lines or methods in file X, and the associated code coverage information in database 130") Examiner Comments: Belur explicitly teaches storing mappings of tests to source code in a database based on the collected code coverage information;
receiving data indicative of one or more modified portions of a second version of the program code, wherein said one or more modified portions are modified from said first version of the program code (Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system) to determine if a new version of a source code file is being checked in. Such a source code file may be a new file or an existing file that has been modified by a developer") Examiner Comments: Belur teaches receiving a modified version of source code via CI polling, where the new version inherently carries data indicating which portions were modified from the prior version;
determining a subset of the tests that are likely to exercise the second version of the program code (Para[0014], "testing application 141 automatically retrieves a list of tests relevant to the source code file by querying database 130 for tests which map to the source code file, if any. That is, testing application 141 reverse maps the source code file (or lines of code) to relevant tests, based on the mappings stored in the database 130"; Para[0023], "database 130 is populated with mappings of source code to tests and associated code coverage during a learning phase, with the mappings being at, e.g., the source code file, method, or source code line level of granularity") Examiner Comments: Belur teaches determining a subset of relevant tests by querying stored mappings based on the modified source code, using both the mapping data and the identification of modified portions;
initiating execution of the subset of the tests on the second version of the program code (Para[0027], "testing application 141 runs the determined tests") Examiner Comments: Belur explicitly teaches initiating execution of the selected subset of tests on the modified source code.
Belur did not specifically teach
based at least in part on the mapping and the data indicative of the one or more modified portions of the second version of the program code.
However, Boshernitsan (US 2015/0007140 A1) teaches
based at least in part on the mapping and the data indicative of the one or more modified portions of the second version of the program code (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 25, Belur and Boshernitsan teach
The Computer-readable storage medium of Claim 24. Claim 25 recites deprecated test limitations. Taught by Belur (Para[0009]) as discussed for Claim 12 above.
Regarding Claim 26, Belur and Boshernitsan teach
The Computer-readable storage medium of Claim 25. Claim 26 recites reversion logic (same as Claim 17). Taught by Belur (Para[0013]) as discussed for Claim 17 above.
Regarding Claim 27, Belur and Boshernitsan teach
The Computer-readable storage medium of Claim 25.
Belur further teaches
Wherein in determining the subset of the tests that are likely to exercise the second version of the program code, the program instructions are further computer executable to perform determining confidence scores for the tests, wherein the confidence scores indicate a likelihood of exercising the second version of the program code; and determining that the confidence scores for the subset of the tests meet a confidence threshold and that the confidence scores for other ones of the tests do not meet the confidence threshold (Belur, Para[0024], "Testing application 141 may instead query for relevant tests matching any feasible criteria, such as having some desired code coverage or cyclomatic dependency score"; Para[0016], "the blacklist may specify that when a file is touched by greater than 80% of the total tests, then a subset of those test needed to reach 100% coverage or some fixed number (e.g., 10) of tests with the greatest code coverage should be used") Examiner Comments: Belur teaches filtering tests based on code coverage scores (confidence scores) and applying thresholds (80%, 100% coverage, top 10) to determine which tests make it into the subset, directly teaching the confidence score and threshold limitation.
Regarding Claim 28, Belur and Boshernitsan teach
The Computer-readable storage medium of Claim 25.
Belur further teaches
Wherein the subset of the tests are executed using a continuous integration system that tests the second version of the program code (Belur, Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system)") Examiner Comments: Belur explicitly teaches integration with a continuous integration system.
Regarding Claim 29, Belur (US 2016/0259713 A1) teaches
A method for automatic build discovery in a CI/CT/CD system, the method comprising:
monitoring a distributed testing environment to detect a build version of running components (Para[0014], "testing application 141 polls a continuous integration system (e.g., Perforce® revision control system) to determine if a new version of a source code file is being checked in") Examiner Comments: Belur teaches polling a CI system to detect when new versions of code are checked in, which constitutes monitoring a testing environment to detect build versions;
automatically identifying changes in the build version (Para[0014], "Such a source code file may be a new file or an existing file that has been modified by a developer") Examiner Comments: Belur’s system automatically identifies that the checked-in code is new or modified;
Belur did not specifically teach
updating a test selection system based on the identified changes to select tests relevant to the new build version.
However, Boshernitsan (US 2015/0007140 A1) teaches
updating a test selection system based on the identified changes to select tests relevant to the new build version (Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover"; Para[0079], "the static analysis module 212 applies the second filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions having property changes in the changed code")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 30, Belur and Boshernitsan teach
The method of Claim 29.
Belur further teaches
Wherein the automatic identification of changes includes analyzing messages regarding test execution to determine which server or service is associated with a particular test (Belur, Para[0015]–[0016], Para[0018], "agent 122 sends a signal (e.g., a TCP/IP message) reporting the gathered information and the test run to testing application 141"; Para [0011], "Configuration file 110 further includes a pointer 111 to an Internet Protocol (IP) address of testing system 120").
Regarding Claim 31, Belur and Boshernitsan teach
The method of Claim 30.
the automatic identification of changes includes analyzing build-diff data and matching with test footprints (Belur, Para[0021]–[0023]; Boshernitsan, Para[0077], "the static analysis module 212 applies the first filter, which produces filter results 280 that weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover") Examiner Comments: Belur identifies code changes and matches them to test mappings (footprints), and Boshernitsan teaches analyzing changed functions and matching test coverage to those changes.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 32, Belur teaches
A method for statistically determining a relative importance of running a particular test within a suite of tests, the method comprising performing the following functions with regard to the system according to claim 1:
mapping a test method to a list of methods or functions executed upon execution of the test (Belur, Para[0013], "agent 122 may generate a dump of code coverage instrumentation results showing that test A affected particular lines or methods in source code file X") Examiner Comments: Belur teaches mapping a test to the methods/functions of the code that are executed when the test runs.
Belur did not specificialy teach
analyzing the mapping to determine a relative likelihood of failure for the test
prioritizing the test based on the determined relative likelihood of failure.
However, Boshernitsan teaches
analyzing the mapping to determine a relative likelihood of failure for the test (Boshernitsan, Para[0077], "weights tests demo1, demo2 and demo3 according to the number of functions in the changed code that they cover") Examiner Comments: Boshernitsan’s analysis of how many changed functions a test covers directly determines a relative likelihood that the test will find failures;
prioritizing the test based on the determined relative likelihood of failure (Boshernitsan, Para[0078], "the greater the coverage score, the higher the priority assigned to the test") Examiner Comments: Boshernitsan explicitly teaches prioritizing tests based on coverage scores correlating with likelihood of detecting failures.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Regarding Claim 33, Belur, and Boshernitsan teach
The method of Claim 32.
Boshernitsan further teaches
Wherein the prioritization includes determining whether the test is to be run at a particular time or in a particular order within the suite of tests (Boshernitsan, Para[0078], "The prioritization module 294, therefore, configures the computer to apply Rule 1 which prioritizes the tests in order demo1, followed by demo2, followed by demo3") Examiner Comments: Boshernitsan explicitly teaches determining a particular order in which tests should be run within the suite.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Belur’s teaching with Boshernitsan’s in order to improve test selection precision by weighting tests based on how many changed functions they cover, as Belur teaches selecting relevant tests for modified code but does not detail how to analyze the quality of the relationship when many tests map to the same file, and Boshernitsan’s multi-filter prioritization provides a well-known mechanism to refine test selection, yielding predictable improved results.
Response to Arguments
Applicant argues, “the claims are not directed to an abstract idea… [the] characterization [of] the claim features as mental processes… improperly oversimplifies the claimed invention,” and that the claims recite “complex computational techniques that require specialized hardware and software to perform, and cannot reasonably be performed in the human mind or with pen and paper” because they operate “on scales fundamentally beyond mental capacity” and employ deep belief networks, convolutional neural networks, and ensemble learning.
Examiner respectfully disagrees. The deep belief networks, convolutional neural networks, and ensemble-learning algorithms on which Applicant relies appear in the specification ( [0276], [0278], [0280]; FIGS. 17A–18), not in the claims. Independent claim 1 recites only “a statistical analysis, a machine learning algorithm or a combination thereof,” and independent claims 13 and 29 recite no neural-network or ensemble limitation at all. It is improper to import these unclaimed embodiments from the specification into the claims to confer eligibility; the claims are evaluated on what they actually recite. See MPEP 2111.01. As recited, determining a relative importance of a test, determining a relationship between a test and code, and selecting tests according to characteristics such as “tests that have failed previously” or “otherwise modified code” are acts of observation, evaluation, and judgment that fall within the “Mental Processes” grouping of MPEP 2106.04a. A claim that recites a mental process does not become non-abstract merely because the recited steps are applied to many items or are performed quickly by a machine; no particular “scale” is recited in the claims, and even if it were, using a generic computer as a tool to perform a mental evaluation faster or on more data does not remove the claim from the mental-processes grouping. See MPEP 2106.04a. The recited “machine learning algorithm” and “statistical analysis,” stated functionally and at a high level of generality, are not a specific technical implementation that removes the claim from the realm of abstract ideas.
Applicant further argues that, “even assuming, arguendo, that the claims recite an abstract idea, the claims are integrated into a practical application under Step 2A, Prong Two, because specific claim elements produce concrete improvements in the functioning of CI/CT/CD computing systems,” namely “faster feedback to developers, reduced infrastructure costs from unnecessary test execution, and higher defect detection rates,” and that the claims are “like the claims in Enfish and McRO.”
Examiner respectfully disagrees. The asserted benefits—faster feedback, reduced infrastructure cost, and higher defect detection are improvements to the test-selection process itself, i.e., to the abstract idea, not improvements to the functioning of a computer or to any technology. See MPEP 2106.05(a). Running fewer tests more quickly is the natural consequence of performing the abstract idea (intelligently determining which tests are important); it does not improve how the computer operates. The additional elements “a computational device,” a “processor,” a “memory,” an “analyzer,” a generic “machine learning algorithm,” and a “continuous integration system” are recited at a high level of generality and amount to no more than applying the abstract idea using generic computer components as tools. See MPEP 2106.05(f). Reciting that the determination occurs in a “CI/CT/CD” environment is a field-of-use limitation that does not impose a meaningful limit. See MPEP 2106.05(h). Enfish and McRO are distinguishable: in Enfish, the claims recited a specific improved data structure (a self-referential table) that improved database functionality, and in McRO, the claims recited a specific type of rule that produced a technological improvement in computer animation. Here, the claims recite the result of the abstract idea (selecting and prioritizing important tests) without reciting a specific technical mechanism that improves the computer itself. Under Step 2B, the additional elements, considered individually and in combination, do not amount to significantly more than the judicial exception; they are generic computer components and well-understood, routine, conventional activities such as receiving data and outputting a result. See MPEP 2106.05(d), (f), (g). The rejection is therefore maintained.
Applicant argues that “Deng’s adaptive-combo algorithm is fundamentally different from the claimed machine learning algorithm… Deng’s system adaptively applies three predetermined, heuristic test selection algorithms (‘FULL-SET,’ ‘COMBO-SET(x),’ and ‘MIN-SET’)… None of these algorithms involves training a model, learning from historical data, or adapting based on outcomes,” and that “Deng does not teach or suggest using a machine learning algorithm to determine the relationship between a test and code.”
Examiner respectfully disagrees. Claim 1 recites the analyzer as determining the relationship “according to a statistical analysis, a machine learning algorithm or a combination thereof.” This limitation is recited in the alternative. To render it obvious, the prior art need teach only one of the recited alternatives. Applicant’s argument is directed to the “machine learning algorithm” alternative and does not address the “statistical analysis” alternative on which the rejection relies. The rejection mapped Deng’s intelligent RTS module / adaptive-combo algorithm to a statistical analysis: Deng quantitatively analyzes coverage data, dependency data, and code-change data and selects and weights test cases per changed or impacted function according to numeric constraints (Dmax, Nmax, Xmax) and metrics such as average test-execution time (Deng, Col. 16, line 21–37; Col. 4, line. 42–Col. 5, line. 17). Whether Deng constitutes “machine learning” is therefore not dispositive, because Deng need only teach “a statistical analysis,” which it does. Furthermore, the neural-network and ensemble embodiments of the specification are not recited in claim 1 and are not read into it under the broadest reasonable interpretation.
To the extent Applicant attacks Deng individually, the rejection is based on the combination of Belur, Boshernitsan, and Deng, and one cannot establish nonobviousness by attacking references individually where the rejection is based on a combination. Belur supplies the computing device with CPU and memory and the stored test-to-code mapping; Boshernitsan supplies the weighting of tests according to changed code; and Deng supplies the analyzer that determines the test-to-code relationship by statistical analysis. The motivation to combine was articulated in the Office Action: applying Deng’s adaptive and statistical analysis to the test-to-code mapping data refines test selection beyond simple coverage mapping, and reflects a combination of known elements according to known methods to yield predictable results.
Applicant argues that “Boshernitsan’s weighting of tests by function-count coverage does not teach the claimed ‘determination of a relationship between the mapping of the tests to the one or more modified portions’… It simply counts, for each pre-existing test, how many changed functions that test covers… This is a unidirectional scoring of tests against changed functions; it does not determine or analyze any relationship between the test-to-code mapping and the modifications as the claims require,” and that Belur is “a straightforward reverse lookup” that “does not determine or analyze a relationship between the mapping and the modified portions.”
Examiner respectfully disagrees. Claim 10 recites “a determination of a relationship between the mapping of the tests to the one or more modified portions.” Under the broadest reasonable interpretation, this requires determining a relationship between (a) the stored mapping of tests to code and (b) the modified portions. The claim does not require analyzing the mapping “as a whole,” nor any “bidirectional” or holistic analysis; those qualifiers appear nowhere in the claim and cannot be read into it. Boshernitsan determines precisely the recited relationship. During the baseline run, Boshernitsan establishes and stores a history of which functions each test covers, the mapping of tests to code. During the partial run, the static analysis module weights each test “according to the number of functions in the changed code that they cover” (Boshernitsan,[0077]) and “according to the number of functions having property changes in the changed code” ([0079]). By Boshernitsan’s own definition, that weight is “indicative of a number of items within the second source code version that are covered by the associated test according to the first history” (Abstract). Computing that weight necessarily intersects the stored test-to-code mapping (the first history) with the modified portions (the changed functions); that intersection is the very “relationship between the mapping of the tests to the one or more modified portions” the claim recites. Applicant’s characterization of this as mere “unidirectional scoring” that does not analyze a relationship is not commensurate with Boshernitsan’s disclosure.
Applicant’s separate attack on Belur is likewise unpersuasive because the rejection is based on the combination. Belur teaches the stored test-to-code mapping ([0013]) and the reverse-mapping of a modified file to relevant tests, while Boshernitsan teaches the relationship determination of limitation. The “reverse lookup” Applicant criticizes is Belur’s contribution to limitations (i) and (ii), not to limitation (iii). Combining the two yields the predictable benefit of improved test-selection precision when many tests map to the same file, as the Office Action articulated.
Regarding amended claim 21, Applicant argues that “Boshernitsan operates entirely within a single baseline-to-partial-run paradigm… [and] does not teach or in any way contemplate maintaining separate mappings per environment, let alone performing a mapping for a test executed on code in a selected environment ‘according to at least one previous mapping for said code in a different environment.’”
Examiner respectfully disagrees. Belur expressly teaches testing across different environments and tools: “agent 122 is able to interpret instrumentation by a variety of code coverage instrumentation tools… different types of agents 122 capable of interpreting respective coverage instrumentations may be copied to the IP address of testing system 120, based on the code coverage instrumentation tool type specified in configuration file 110” (Belur,[0012]). Belur thus discloses generating test-to-code mappings for the same code under different instrumentation environments and testing systems. It would have been obvious to one of ordinary skill in the art before the effective filing date to perform a mapping for a test executed on code in a selected environment according to at least one previous mapping for that same code in a different environment, because reusing mapping information generated in one environment to inform test selection in another is a predictable extension of Belur’s multi-environment instrumentation support and improves overall test coverage across the distributed CI/CD environments, a combination of known techniques yielding predictable results.
Regarding claims 29–31, Applicant argues that “Belur’s polling of a revision control system to determine ‘if a new version of a source code file is being checked in’ is not equivalent to the recited monitoring a distributed testing environment to detect a build version of running components… Belur monitors a source control system for code check-ins, not a testing environment for running build components.”
Examiner respectfully disagrees. Under the broadest reasonable interpretation consistent with the specification, which describes automatic build discovery by monitoring a continuous-integration environment, Belur’s testing system reads on the recited monitoring. Belur’s testing application “polls a continuous integration system… to determine if a new version of a source code file is being checked in” (Belur,[0014]) within a distributed arrangement of agents and testing systems that communicate over IP (Belur, [0011]–[0012]). The continuous-integration system constitutes a distributed testing environment, the newly checked-in version constitutes a new build version, and Belur’s determination that the code is “a new file or an existing file that has been modified” ([0014]) reads on automatically identifying changes in the new build version from a previous build version. Boshernitsan teaches updating the test selection based on the identified changes ([0077], [0079]). Dependent claims 30–31 are addressed in the Office Action. The rejection is maintained.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMIR SOLTANZADEH whose telephone number is (571)272-3451. The examiner can normally be reached M-F, 9am - 5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wei Mui can be reached at (571) 272-3708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMIR SOLTANZADEH/Examiner, Art Unit 2191
/WEI Y MUI/Supervisory Patent Examiner, Art Unit 2191
Prosecution Insights