Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are presented for examination.
Allowable Subject Matter
Claim 2-3 would be allowable if rewritten to overcome the Double patenting rejection(s) set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the "right to exclude" granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 11,163,675 B1 (Pearson et al., issued from parent application 17/224,873). Although the claims at issue are not identical, they are not patentably distinct from each other because the conflicting claims, which share a common specification and inventive entity with the present application, are directed to overlapping subject matter, mutation testing in parallel threads of a software application by modifying mutatable source code class files, compiling the modifications to bytecode, generating application mutants by swapping previously-compiled bytecode in memory without recompiling other bytecode, and executing test cases against the application mutants. For illustration purposes, the rejection of Claim 1 of the present application over Claim 1 of U.S. Patent No. 11,163,675 is provided in the table below:
Current Application 18/948,219
U.S. Patent No. 11,163,675 B1
1. A computer-implemented method, comprising:
executing, by a computing system comprising a processor, test cases against an unmodified version of a software application;
determining, by the computing system, and based on execution of the test cases, code coverage metrics corresponding to a plurality of source code files associated with the software application;
identifying, by the computing system, and based on the code coverage metrics, mutatable files, of the plurality of source code files, that: correspond to at least one mutation configuration, and have at least a threshold code coverage level;
generating, by the computing system, mutated source code files by modifying the mutatable files based on the at least one mutation configuration;
compiling, by the computing system, the mutated source code files into bytecode files; and
executing, by the computing system, and in parallel, the test cases against application mutants that include one or more of the bytecode files.
1. A computer-implemented method, comprising:
initiating, by one or more processors of one or more computing devices, a plurality of parallel threads;
identifying, by the one or more processors, and in different threads of the plurality of parallel threads, mutatable source code class files associated with a software application;
modifying, by the one or more processors, and in the different threads, the mutatable source code class files based on at least one mutation configuration;
compiling, by the one or more processors, and in the different threads, the mutatable source code class files into compiled bytecode;
generating, by the one or more processors, and in the different threads, application mutants that include the compiled bytecode and first previously-compiled bytecode associated with unmodified source code files, by swapping, in memory associated with the different threads, second previously-compiled bytecode with the compiled bytecode, without recompiling the first previously-compiled bytecode; and
executing, by the one or more processors, and in the different threads, a plurality of test cases against the application mutants.
Claim 1 of U.S. Patent No. 11,163,675 does not explicitly recite (i) "executing… test cases against an unmodified version of a software application [and] determining… code coverage metrics", or (ii) "identifying… mutatable files… that have at least a threshold code coverage level." However, the common specification shared by U.S. Patent No. 11,163,675 and the present application expressly discloses both (see, e.g., the disclosure corresponding to FIG. 2, block 202, describing "generate a test report by executing the test cases 106 against an unmodified version of the software application" to obtain "code coverage metrics associated with individual class files"; and the disclosure corresponding to FIG. 2, block 212, describing identifying mutatable classes that "have at least a threshold percentage of those source code lines covered by the test cases"). It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate these features into Claim 1 of U.S. Patent No. 11,163,675 in order to avoid mutating, compiling, and testing source code class files for which the test cases provide insufficient coverage, thereby reducing wasted computational resources and improving mutation testing efficiency. Accordingly, the differences between present Claim 1 and Claim 1 of U.S. Patent No. 11,163,675 are not patentably distinct, and present Claim 1 would have been obvious over Claim 1 of U.S. Patent No. 11,163,675.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 11,797,429 B2 (Pearson et al., issued October 24, 2023 from parent application 17/486,682). Although the claims at issue are not identical, they are not patentably distinct from each other because the conflicting claims, which share a common specification and inventive entity with the present application, are directed to overlapping subject matter, mutation testing in parallel threads by identifying mutatable source code class files, modifying and recompiling those files into bytecode, and executing test cases against the resulting application mutants. For illustration purposes, the rejection of Claim 16 of the present application over Claim 15 of U.S. Patent No. 11,797,429 is provided in the table below:
Current Application 18/948,219
U.S. Patent No. 11,797,429 B2
16. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to:
execute a set of test cases against an unmodified version of a software application;
determine, based on executing the set of test cases, code coverage metrics corresponding to a plurality of source code files associated with the software application;
identify, based on the code coverage metrics, mutatable files, of the plurality of source code files, that: correspond to at least one mutation configuration, and have at least a threshold code coverage level;
generate mutated source code files by modifying the mutatable files based on the at least one mutation configuration;
compile the mutated source code files into mutated bytecode files; and
perform mutation testing by executing, in parallel, test cases of the set of test cases against different application mutants that comprise different sets of the mutated bytecode files.
15. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to:
initiate a first parallel thread and a second parallel thread; and
perform, substantially concurrently:
first parallel thread operations, associated with the first parallel thread, comprising: identifying a first mutatable source code class file associated with a software application;
generating a first mutated source code class file by modifying the first mutatable source code class file based on at least one mutation configuration;
compiling the first mutated source code class file into two or more first compiled bytecode class files;
and executing a plurality of test cases against a first application mutant that includes the two or more first compiled bytecode class files; and
second parallel thread operations, associated with the second parallel thread, comprising: identifying a second mutatable source code class file associated with the software application; generating a second mutated source code class file by modifying the second mutatable source code class file based on the at least one mutation configuration; compiling the second mutated source code class file into two or more second compiled bytecode class files; and executing the plurality of test cases against a second application mutant that includes the two or more second compiled bytecode class files.
Claim 15 of U.S. Patent No. 11,797,429 recites substantially identical subject matter to present Claim 16, including identifying mutatable source code class files, modifying and recompiling them based on a mutation configuration, generating application mutants comprising the resulting bytecode, and executing test cases against multiple different application mutants concurrently in parallel threads. Claim 15 of U.S. Patent No. 11,797,429 does not explicitly recite (i) "execute a set of test cases against an unmodified version of a software application" and "determine… code coverage metrics", or (ii) "identify… mutatable files… that… have at least a threshold code coverage level". However, the common specification expressly discloses both features (see, e.g., the disclosure corresponding to FIG. 2, blocks 202 and 212), and it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate these features into Claim 15 of U.S. Patent No. 11,797,429 in order to avoid wasting computational resources on mutating, compiling, and testing source code class files for which the existing test suite provides insufficient line coverage.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 12,174,733 B2 (Pearson et al., issued from parent application 18/466,756). Although the claims at issue are not identical, they are not patentably distinct from each other because the conflicting claims, which share a common specification and inventive entity with the present application, are directed to substantially the same subject matter, mutation testing performed by modifying mutatable source code class files based on a mutation configuration, compiling the resulting mutated source into bytecode having a different structure than the source, generating application mutants by swapping previously-compiled bytecode in memory without recompiling other bytecode, and executing test cases against the application mutants in parallel. For illustration purposes, the rejection of Claim 1 of the present application over Claims 1 of U.S. Patent No. 12,174,733 is provided in the table below:
Current Application 18/948,219
U.S. Patent No. 12,174,733 B2
1. A computer-implemented method, comprising:
executing, by a computing system comprising a processor, test cases against an unmodified version of a software application;
determining, by the computing system, and based on execution of the test cases, code coverage metrics corresponding to a plurality of source code files associated with the software application;
identifying, by the computing system, and based on the code coverage metrics, mutatable files, of the plurality of source code files, that: correspond to at least one mutation configuration, and have at least a threshold code coverage level;
generating, by the computing system, mutated source code files by modifying the mutatable files based on the at least one mutation configuration;
compiling, by the computing system, the mutated source code files into bytecode files; and
executing, by the computing system, and in parallel, the test cases against application mutants that include one or more of the bytecode files.
1. A computer-implemented method, comprising:
identifying, by one or more processors, mutatable source code class files associated with a software application;
generating, by the one or more processors, mutated source code class files by modifying the mutatable source code class files based on at least one mutation configuration;
compiling, by the one or more processors, the mutated source code class files into compiled bytecode class files that have a different structure than an original structure of the mutated source code class files;
generating, by the one or more processors, application mutants by swapping first previously-compiled bytecode files with one or more of the compiled bytecode class files in memory, without recompiling additional second previously-compiled bytecode files; and
executing, by the one or more processors, and in parallel, test cases against the application mutants.
7. The computer-implemented method of claim 1, wherein identifying the mutatable source code class files comprises: determining, by the one or more processors, code coverage metrics associated with a plurality of source class files by executing the test cases against an unmodified version of the software application; and identifying, by the one or more processors, and based on the code coverage metrics, source class files that have at least a threshold code coverage level.
Claims 1 and 7 of U.S. Patent No. 12,174,733, read together, recite every limitation of present Claim 1 in substantially the same words: identifying mutatable source code class files by determining code coverage metrics from executing test cases against an unmodified version of the software application and selecting source class files having at least a threshold code coverage level (Claim 7); generating mutated source code class files based on a mutation configuration (Claim 1); compiling the mutated source into bytecode (Claim 1); and executing test cases against application mutants in parallel (Claim 1). To the extent any minor wording differences exist between Claims 1 and 7 of U.S. Patent No. 12,174,733 and present Claim 1, those differences would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, in view of the common specification, because the same disclosure supports both claim sets. Accordingly, present Claim 1 is not patentably distinct from Claims 1of U.S. Patent No. 12,174,733.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 4-6, 8-9, 13, 16, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Abhishek (US 2020/0073790 A1) in view of Dern (US 8,276,021 B2).
Regarding Claim 1, Abhishek (US 2020/0073790 A1) teaches
A computer-implemented method, comprising:
executing, by a computing system comprising a processor, test cases against an unmodified version of a software application; (Para [0014], "the test suite 137 of all test cases for the input source code 131 may be run to get the line coverage details for each source code file using any suitable code coverage library tool, such as JaCoCo, Sonar, EclEmma, Cobertura, CodeCover, Coverage.py, Gretel, JCov, and the like") Examiner Comments: Abhishek discloses running the test suite against the input (unmodified) source code in order to obtain line coverage details, which corresponds to executing test cases against an unmodified version of the software application.
determining, by the computing system, and based on execution of the test cases, code coverage metrics corresponding to a plurality of source code files associated with the software application; (Para [0014], "the test suite 137 of all test cases for the input source code 131 may be run to get the line coverage details for each source code file using any suitable code coverage library tool, such as JaCoCo, Sonar, EclEmma, Cobertura, CodeCover, Coverage.py, Gretel, JCov, and the like") Examiner Comments: Abhishek determines, based on running the test suite, line coverage details for each input source code file, which corresponds to determining code coverage metrics for a plurality of source code files.
identifying, by the computing system, and based on the code coverage metrics, mutatable files, of the plurality of source code files, that: correspond to at least one mutation configuration, (Para [0012], "the mutation generator 114 may define a plurality of mutators, such as the Increments_Mutator, Void_Method_Call_Mutator, Return_Vals_Mutator, Math_Mutator, Negate_Conditionals_Mutator, Invert_Negs_Mutator, or Conditionals_Boundary_Mutator. Using inputs from the mutation generator 113, the source code modification module 115 may dynamically change one or more application code segments 132, 133 in the input source code files 131, thereby generating mutated source code files 134") Examiner Comments: The plurality of mutators defined by the mutation generator corresponds to the claimed mutation configuration, and Abhishek identifies application code segments (i.e., mutatable files) within the input source code files that correspond to those mutators.
and have at least a threshold code coverage level; (Para [0002], "the quality of a test suite must be evaluated to determine not only the code coverage for the test suite (e.g., what percentage of the execution paths in the computer program have been exercised during automated testing), but also the test accuracy and user case completeness of the test suite") Examiner Comments: Abhishek's use of line coverage details (Para [0014]) per source code file, combined with the recognized objective of evaluating what percentage of execution paths have been exercised, would lead one of ordinary skill in the art to identify files meeting at least a threshold coverage level for mutation, in order to avoid wastefully mutating files that are not covered by tests.
generating, by the computing system, mutated source code files by modifying the mutatable files based on the at least one mutation configuration; (Para [0012], "the source code modification module 115 may dynamically change one or more application code segments 132, 133 in the input source code files 131, thereby generating mutated source code files 134 having one or more mutated application code segments 135, 136 which are generated for the purpose of causing test failures in the mutated source code 134") Examiner Comments: Abhishek expressly discloses generating mutated source code files by modifying the input source code files using the defined mutators (i.e., the mutation configuration).
compiling, by the computing system, the mutated source code files into bytecode files; (Para [0013], "the test engine 117 may compile and build the modified source code 134 and run one or more test generator modules 118 to generate one or more test suites 137") Examiner Comments: Abhishek expressly discloses that the test engine compiles and builds the mutated source code, which produces compiled (bytecode) files that the test suite is run against.
Abhishek did not specifically teach
executing, by the computing system, and in parallel, the test cases against application mutants that include one or more of the bytecode files.
However, Dern (US 8,276,021 B2) teaches
executing, by the computing system, and in parallel, the test cases against application mutants that include one or more of the bytecode files. (Col 3, lines 27-50, "FIG. 1 illustrates that the implementation source code 102 may be iteratively modified to modified instances of the implementation source code 102. For example, as illustrated, four iterations of the implementation source code have been created 102', 102", 102'", 102"". Iterations as used herein do not necessarily include a time component. For example, iteration 102' does not necessarily need to be created and tested before iteration 102". Rather the iterations can be created and tested in any time order or simultaneously") Examiner Comments: Dern expressly discloses that multiple modified iterations (i.e., application mutants) of the source code can be created and tested by the test suite simultaneously, which corresponds to executing the test cases in parallel against application mutants.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the mutation testing system of Abhishek with the simultaneous (parallel) testing of multiple modified source iterations as taught by Dern in order to reduce overall testing time and improve the effectiveness of test suites for detecting concurrency problems and product faults by enabling iterative source code modifications to be created and tested simultaneously, which provides earlier feedback on test adequacy and product code faults (Dern, Col 1, lines 23-35; Col 3, lines 27-44).
Regarding Claim 4, Abhishek and Dern teach
The method of Claim 1.
Dern further teaches
wherein instances of at least one of generating the mutated source code files or compiling the mutated source code files are performed in parallel. (Col 3, lines 27-50, "iteration 102' does not necessarily need to be created and tested before iteration 102". Rather the iterations can be created and tested in any time order or simultaneously") Examiner Comments: Because Dern teaches that iterations of the modified source code can be created and tested simultaneously, this includes performing instances of generating (creating) the modified source code in parallel.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the mutation testing system of Abhishek with the simultaneous (parallel) testing of multiple modified source iterations as taught by Dern in order to reduce overall testing time and improve the effectiveness of test suites for detecting concurrency problems and product faults by enabling iterative source code modifications to be created and tested simultaneously, which provides earlier feedback on test adequacy and product code faults (Dern, Col 1, lines 23-35; Col 3, lines 27-44).
Regarding Claim 5, Abhishek and Dern teach
The method of Claim 1.
Dern further teaches
wherein executing the test cases against the application mutants comprises executing, in parallel, the test cases against different application mutants, the different application mutants comprising different sets of the bytecode files that are characterized by different modifications to the mutatable files. (Col 3, lines 24-50, "each iteration removes or disables a synchronization primitive. For example iteration 102' disables or removes the synchronization primitive 106-1, iteration 102" disables or removes the synchronization primitive 106-2, iteration 102'" disables or removes the synchronization primitive 106-3, and iteration 102"" disables or removes the synchronization primitive 106-4 … Rather the iterations can be created and tested in any time order or simultaneously") Examiner Comments: Dern's multiple iterations each represent a different modification of the source code, and Dern expressly teaches running them simultaneously, which corresponds to executing test cases in parallel against different application mutants that comprise different sets of files characterized by different modifications.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the mutation testing system of Abhishek with the simultaneous (parallel) testing of multiple modified source iterations as taught by Dern in order to reduce overall testing time and improve the effectiveness of test suites for detecting concurrency problems and product faults by enabling iterative source code modifications to be created and tested simultaneously, which provides earlier feedback on test adequacy and product code faults (Dern, Col 1, lines 23-35; Col 3, lines 27-44).
Regarding Claim 6, Abhishek and Dern teach
The method of Claim 1.
Dern further teaches
wherein executing the test cases against the application mutants comprises executing different sets of test cases, in parallel, against a particular application mutant of the application mutants. (Col 3, lines 50-67, "the iterations 102', 102", 102'", 102"", of the source code 102 are provided to a multi-thread system 104 which is running a test suite 108. The test suite 108 includes a number of diagnostic modules which can detect various runtime problems") Examiner Comments: Dern teaches running multiple diagnostic modules of the test suite against a particular iteration (i.e., particular application mutant) using a multi-thread system, which corresponds to executing different sets of test cases in parallel against a particular application mutant.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the mutation testing system of Abhishek with the simultaneous (parallel) testing of multiple modified source iterations as taught by Dern in order to reduce overall testing time and improve the effectiveness of test suites for detecting concurrency problems and product faults by enabling iterative source code modifications to be created and tested simultaneously, which provides earlier feedback on test adequacy and product code faults (Dern, Col 1, lines 23-35; Col 3, lines 27-44).
Regarding Claim 8, Abhishek and Dern teach
The method of Claim 1.
Abhishek further teaches,
logging, by the computing system, mutation test results associated with executing the test cases against the application mutants in parallel; and (Abhishek, Para [0013]-[0014], "if the test suite 137 does not detect a change caused by a mutated source code file (e.g., no test fails), then the mutated source code file is said to "survive," and may be collected with the survived mutations 138 (processing step 123)") Examiner Comments: Abhishek discloses collecting and logging the survived mutations resulting from running the test suite, which corresponds to logging mutation test results.
aggregating, by the computing system, the mutation test results into an aggregated test result report. (Abhishek, Para [0014], "With an assembled listing of blamed test cases/classes 139, the program developer/tester can generate new tests 140") Examiner Comments: Abhishek discloses assembling an aggregated listing of test results (blamed test cases/classes 139) corresponding to the survived mutations, which corresponds to aggregating the mutation test results into an aggregated test result report.
Regarding Claim 9, Abhishek (US 2020/0073790 A1) teaches
A computing system, comprising: one or more processors; and memory storing computer-executable instructions that, when executed by the one or more processors, cause the computing system to:
identify mutatable files, of a plurality of source code files associated with a software application, that: correspond to at least one mutation configuration, (Para [0012], "the mutation generator 114 may define a plurality of mutators, such as the Increments_Mutator, Void_Method_Call_Mutator, Return_Vals_Mutator, Math_Mutator, Negate_Conditionals_Mutator, Invert_Negs_Mutator, or Conditionals_Boundary_Mutator. Using inputs from the mutation generator 113, the source code modification module 115 may dynamically change one or more application code segments 132, 133 in the input source code files 131") Examiner Comments: The plurality of mutators corresponds to the mutation configuration, and Abhishek identifies application code segments within the input source code files (i.e., mutatable files) that correspond to those mutators.
and have at least a threshold percentage of source code lines covered by a set of test cases; (Para [0002]; Para [0014], "the quality of a test suite must be evaluated to determine not only the code coverage for the test suite (e.g., what percentage of the execution paths in the computer program have been exercised during automated testing) … the test suite 137 of all test cases for the input source code 131 may be run to get the line coverage details for each source code file") Examiner Comments: Abhishek's recognition that what is evaluated is the percentage of execution paths exercised by tests, combined with obtaining line coverage details per source code file, teaches identifying source code files with at least a threshold percentage of source code lines covered.
generate mutated source code files by modifying the mutatable files based on the at least one mutation configuration; (Para [0012], "the source code modification module 115 may dynamically change one or more application code segments 132, 133 in the input source code files 131, thereby generating mutated source code files 134") Examiner Comments: Abhishek expressly discloses generating mutated source code files by modifying the input source code files using the defined mutators (i.e., mutation configuration).
compile the mutated source code files into bytecode files; (Para [0013], "the test engine 117 may compile and build the modified source code 134") Examiner Comments: Abhishek discloses that the test engine compiles and builds the modified source code, producing compiled (bytecode) files.
Abhishek did not specifically teach
execute, in parallel, test cases of the set of test cases against application mutants comprising one or more of the bytecode files.
However, Dern teaches
execute, in parallel, test cases of the set of test cases against application mutants comprising one or more of the bytecode files. (Col 3, lines 27-50, "Iterations as used herein do not necessarily include a time component. For example, iteration 102' does not necessarily need to be created and tested before iteration 102". Rather the iterations can be created and tested in any time order or simultaneously") Examiner Comments: Dern expressly teaches that multiple modified iterations (i.e., application mutants) of the source code can be tested simultaneously, which corresponds to executing test cases in parallel against application mutants.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the mutation testing system of Abhishek with the simultaneous (parallel) testing of multiple modified source iterations as taught by Dern in order to reduce overall testing time and improve the effectiveness of test suites for detecting concurrency problems and product faults by enabling iterative source code modifications to be created and tested simultaneously, which provides earlier feedback on test adequacy and product code faults (Dern, Col 1, lines 23-35; Col 3, lines 27-44).
Regarding Claim 13, Abhishek and Dern teach
The computing system of Claim 9.
Dern further teaches, wherein the computer-executable instructions cause the computing system to perform at least one of generating the mutated source code files or compiling the mutated source code files in parallel. (Dern, Col 3, lines 36-44, "iteration 102' does not necessarily need to be created and tested before iteration 102". Rather the iterations can be created and tested in any time order or simultaneously") Examiner Comments: Dern's teaching that iterations can be created (i.e., generated) and tested simultaneously encompasses performing at least one of generating or compiling the mutated source code files in parallel. The motivation to combine is the same as set forth above for Claim 9.
Regarding Claim 16, Abhishek (US 2020/0073790 A1) teaches
One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to:
execute a set of test cases against an unmodified version of a software application; (Para [0014], "the test suite 137 of all test cases for the input source code 131 may be run to get the line coverage details for each source code file") Examiner Comments: Abhishek discloses running the test suite against the input source code 131, which corresponds to executing the set of test cases against an unmodified version of the software application.
determine, based on executing the set of test cases, code coverage metrics corresponding to a plurality of source code files associated with the software application; (Para [0014], "the test suite 137 of all test cases for the input source code 131 may be run to get the line coverage details for each source code file using any suitable code coverage library tool, such as JaCoCo, Sonar, EclEmma, Cobertura, CodeCover, Coverage.py, Gretel, JCov, and the like") Examiner Comments: Abhishek determines line coverage details for each input source code file based on running the test suite, which corresponds to determining code coverage metrics.
identify, based on the code coverage metrics, mutatable files, of the plurality of source code files, that: correspond to at least one mutation configuration, and have at least a threshold code coverage level; (Paras [0002] and [0012], "the quality of a test suite must be evaluated to determine not only the code coverage for the test suite (e.g., what percentage of the execution paths in the computer program have been exercised during automated testing)… the mutation generator 114 may define a plurality of mutators, such as the Increments_Mutator, Void_Method_Call_Mutator, Return_Vals_Mutator, Math_Mutator, Negate_Conditionals_Mutator, Invert_Negs_Mutator, or Conditionals_Boundary_Mutator") Examiner Comments: Abhishek's plurality of mutators corresponds to the mutation configuration, and Abhishek identifies code segments / files that correspond to the mutators while evaluating the percentage of execution paths exercised by tests, which teaches identifying mutatable files having at least a threshold code coverage level corresponding to the mutation configuration.
generate mutated source code files by modifying the mutatable files based on the at least one mutation configuration; (Para [0012], "the source code modification module 115 may dynamically change one or more application code segments 132, 133 in the input source code files 131, thereby generating mutated source code files 134") Examiner Comments: Abhishek expressly discloses generating mutated source code files by modifying the input source code files using the defined mutators (i.e., mutation configuration).
compile the mutated source code files into mutated bytecode files; (Para [0013], "the test engine 117 may compile and build the modified source code 134") Examiner Comments: Abhishek discloses that the test engine compiles and builds the modified source code, producing compiled (mutated bytecode) files.
Abhishek did not specifically teach
perform mutation testing by executing, in parallel, test cases of the set of test cases against different application mutants that comprise different sets of the mutated bytecode files.
However, Dern teaches
perform mutation testing by executing, in parallel, test cases of the set of test cases against different application mutants that comprise different sets of the mutated bytecode files. (Col 3, lines 27-50, "each iteration removes or disables a synchronization primitive. For example iteration 102' disables or removes the synchronization primitive 106-1, iteration 102" disables or removes the synchronization primitive 106-2 … Rather the iterations can be created and tested in any time order or simultaneously") Examiner Comments: Dern teaches that multiple different modified iterations (i.e., different application mutants) of the source code are each created by modifying a different portion, and can be tested simultaneously, which corresponds to executing test cases in parallel against different application mutants comprising different sets of mutated bytecode files.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the mutation testing system of Abhishek with the simultaneous (parallel) testing of multiple modified source iterations as taught by Dern in order to reduce overall testing time and improve the effectiveness of test suites for detecting concurrency problems and product faults by enabling iterative source code modifications to be created and tested simultaneously, which provides earlier feedback on test adequacy and product code faults (Dern, Col 1, lines 23-35; Col 3, lines 27-44).
Regarding Claim 19, Abhishek and Dern teach
The one or more non-transitory computer-readable media of Claim 16.
Dern further teaches, wherein the computer-executable instructions cause the one or more processors to perform at least one of generating the mutated source code files or compiling the mutated source code files in parallel. (Dern, Col 3, lines 27-50, "iteration 102' does not necessarily need to be created and tested before iteration 102". Rather the iterations can be created and tested in any time order or simultaneously") Examiner Comments: Dern's teaching that iterations can be created (i.e., generated) and tested simultaneously encompasses performing at least one of generating or compiling the mutated source code files in parallel. The motivation to combine is the same as set forth above for Claim 16.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the mutation testing system of Abhishek with the simultaneous (parallel) testing of multiple modified source iterations as taught by Dern in order to reduce overall testing time and improve the effectiveness of test suites for detecting concurrency problems and product faults by enabling iterative source code modifications to be created and tested simultaneously, which provides earlier feedback on test adequacy and product code faults (Dern, Col 1, lines 23-35; Col 3, lines 27-44).
Claim(s) 10-12, 17 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Abhishek (US 2020/0073790 A1) in view of Dern et al. (US 8,276,021 B2) and further in view of Gregersen (US 2015/0100951 A1).
Regarding Claim 10, Abhishek and Dern teach
The computing system of Claim 9.
Abhishek and Dern did not specifically teach
wherein the bytecode files have a different structure than an original structure of the mutated source code files.
However, Gregersen teaches
wherein the bytecode files have a different structure than an original structure of the mutated source code files. (Abstract; Para [0007], "A system and method for transforming name synthesized classes in response to dynamic class updates to existing classes, featuring the ability to provide replacement synthetic names for reloaded name synthesized classes") Examiner Comments: Gregersen discloses that compiled bytecode contains synthetically-generated classes that do not appear in the original source code, which corresponds to the claimed bytecode files having a different structure than the original structure of the source code files.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Abhishek and Dern with Gregersen in order to extend mutation testing to programming environments in which compilers produce a different number of bytecode class files than the number of original source code class files (e.g., due to inner classes, anonymous classes, or synthetically-generated classes), thereby providing a system and method capable of transforming and reloading name-synthesized classes that result from such compilers (Gregersen, Abstract).
Regarding Claim 11, Abhishek and Dern teach
The computing system of Claim 10.
Abhishek and Dern did not specifically teach
wherein: the original structure of the mutated source code files is characterized by a first number of files or classes, and the different structure of the bytecode files is characterized by a second number of files or classes that is different from the first number.
However, Gregersen teaches
wherein: the original structure of the mutated source code files is characterized by a first number of files or classes, and the different structure of the bytecode files is characterized by a second number of files or classes that is different from the first number. (Abstract; Para [0007]; Fig. 4, "A system and method for transforming name synthesized classes in response to dynamic class updates to existing classes, featuring the ability to provide replacement synthetic names for reloaded name synthesized classes") Examiner Comments: The synthetically-generated (e.g., anonymous/inner) classes addressed by Gregersen result in a number of compiled bytecode classes/files that is different from the number of original source code classes/files, corresponding to the claimed mismatch between first and second numbers.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Abhishek and Dern with Gregersen in order to extend mutation testing to programming environments in which compilers produce a different number of bytecode class files than the number of original source code class files (e.g., due to inner classes, anonymous classes, or synthetically-generated classes), thereby providing a system and method capable of transforming and reloading name-synthesized classes that result from such compilers (Gregersen, Abstract).
Regarding Claim 12, Abhishek and Dern teach
The computing system of Claim 9.
Abhishek and Dern did not specifically teach
wherein: a first mutated source code file, of the mutated source code files, comprises a single mutated class, and the bytecode files comprise two or more classes that correspond to the single mutated class.
However, Gregersen teaches
wherein: a first mutated source code file, of the mutated source code files, comprises a single mutated class, and the bytecode files comprise two or more classes that correspond to the single mutated class. (Para [0088]-[0092]; Fig. 4 (steps 624-628), "is current class an enclosing class? … Are there any more inner classes? … Identify next inner class within enclosing class … is current inner class an anonymous inner class?") Examiner Comments: Gregersen discloses that an enclosing source class compiles into multiple bytecode classes (the enclosing class plus its inner/anonymous classes), which corresponds to the claimed single mutated class (in source) resulting in two or more bytecode classes that correspond to that single mutated class.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Abhishek and Dern with Gregersen in order to extend mutation testing to programming environments in which compilers produce a different number of bytecode class files than the number of original source code class files (e.g., due to inner classes, anonymous classes, or synthetically-generated classes), thereby providing a system and method capable of transforming and reloading name-synthesized classes that result from such compilers (Gregersen, Abstract).
Regarding Claim 17, Abhishek and Dern teach
The one or more non-transitory computer-readable media of Claim 16.
Abhishek and Dern did not specifically teach
wherein the mutated bytecode files have a different structure than an original structure of the mutated source code files.
However, Gregersen teaches
wherein the mutated bytecode files have a different structure than an original structure of the mutated source code files. (Abstract, "A system and method for transforming name synthesized classes in response to dynamic class updates to existing classes, featuring the ability to provide replacement synthetic names for reloaded name synthesized classes") Examiner Comments: Gregersen discloses that compiled bytecode contains synthetically-generated classes that do not appear in the original source code, which corresponds to the claimed bytecode files having a different structure than the original structure of the source code files.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Abhishek and Dern with Gregersen in order to extend mutation testing to programming environments in which compilers produce a different number of bytecode class files than the number of original source code class files (e.g., due to inner classes, anonymous classes, or synthetically-generated classes), thereby providing a system and method capable of transforming and reloading name-synthesized classes that result from such compilers (Gregersen, Abstract).
Regarding Claim 18, Abhishek and Dern teach
The one or more non-transitory computer-readable media of Claim 17.
Abhishek and Dern did not specifically teach
wherein: the original structure of the mutated source code files is characterized by a first number of files or classes, and the different structure of the mutated bytecode files is characterized by a second number of files or classes that is different from the first number.
However, Gregersen teaches
wherein: the original structure of the mutated source code files is characterized by a first number of files or classes, and the different structure of the mutated bytecode files is characterized by a second number of files or classes that is different from the first number. (Abstract; Para [0007]; Fig. 4, "A system and method for transforming name synthesized classes in response to dynamic class updates to existing classes, featuring the ability to provide replacement synthetic names for reloaded name synthesized classes") Examiner Comments: The synthetically-generated classes addressed by Gregersen result in a number of compiled bytecode classes/files different from the number of original source code classes/files, corresponding to the claimed mismatch between the first and second numbers.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Abhishek and Dern with Gregersen in order to extend mutation testing to programming environments in which compilers produce a different number of bytecode class files than the number of original source code class files (e.g., due to inner classes, anonymous classes, or synthetically-generated classes), thereby providing a system and method capable of transforming and reloading name-synthesized classes that result from such compilers (Gregersen, Abstract).
Claim(s) 7, 14, 15, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Abhishek (US 2020/0073790 A1) in view of Dern et al. (US 8,276,021 B2) and further in view of Susarla et al. (US 6,915,511 B2).
Regarding Claim 7, Abhishek and Dern teach
The method of Claim 1.
Abhishek and Dern did not specifically teach
further comprising generating the application mutants by swapping, in memory, first previously-compiled bytecode files with the one or more of the bytecode files that correspond to the mutated source code files, without recompiling additional second previously-compiled bytecode files.
However, Susarla et al. (US 6,915,511 B2) teaches
further comprising generating the application mutants by swapping, in memory, first previously-compiled bytecode files with the one or more of the bytecode files that correspond to the mutated source code files, without recompiling additional second previously-compiled bytecode files. (Col 8, lines 1-4, "Using the dynamic class reloading mechanism, only a changed class and its dependent classes are reloaded, thus limiting the number of files that are affected on the application server") Examiner Comments: Susarla discloses replacing, in memory, the previously-loaded (previously-compiled) class with the changed class while the application is executing, and reloads only the changed class and its dependent classes without reloading or replacing the other previously-loaded classes, which corresponds to swapping in memory the first previously-compiled bytecode files with the mutated bytecode files without recompiling additional second previously-compiled bytecode files.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the in-memory class-reloading mechanism of Susarla into the mutation testing system of Abhishek and Dern in order to generate each application mutant by swapping only the recompiled mutated class into memory in place of the previously-compiled class, without recompiling or reloading the unchanged classes, thereby limiting the number of files that are affected and avoiding the overhead of restarting or fully recompiling the application for each mutant (Susarla, Col 8, lines 1-4), which further reduces the per-mutant testing overhead sought by Abhishek and Dern.
Regarding Claim 14, Abhishek, and Dern teach
The computing system of Claim 9.
Abhishek and Dern did not specifically teach
wherein the computer-executable instructions cause the computing system to generate the application mutants by swapping, in the memory or a second memory, first previously-compiled bytecode files with the one or more of the bytecode files corresponding to the mutated source code files, without recompiling additional second previously-compiled bytecode files.
However, Susarla teaches
wherein the computer-executable instructions cause the computing system to generate the application mutants by swapping, in the memory or a second memory, first previously-compiled bytecode files with the one or more of the bytecode files corresponding to the mutated source code files, without recompiling additional second previously-compiled bytecode files. (Col 8, lines 1-4, "Using the dynamic class reloading mechanism, only a changed class and its dependent classes are reloaded, thus limiting the number of files that are affected on the application server") Examiner Comments: Susarla discloses replacing the previously-loaded class with the changed class in memory while the application is executing and reloading only the changed class and its dependents, which corresponds to swapping, in the memory, the first previously-compiled bytecode files with the mutated bytecode files without recompiling the other previously-compiled bytecode files.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the in-memory class-reloading mechanism of Susarla into the mutation testing system of Abhishek and Dern in order to generate each application mutant by swapping only the recompiled mutated class into memory in place of the previously-compiled class, without recompiling or reloading the unchanged classes, thereby limiting the number of files that are affected and avoiding the overhead of restarting or fully recompiling the application for each mutant (Susarla, Col 8, lines 1-4), which further reduces the per-mutant testing overhead sought by Abhishek and Dern.
Regarding Claim 15, Abhishek, and Dern teach
The computing system of Claim 14.
Abhishek and Dern did not specifically teach
wherein different application mutants are generated by swapping different first previously-compiled bytecode files with different bytecode files.
However, Susarla teaches
wherein different application mutants are generated by swapping different first previously-compiled bytecode files with different bytecode files. (Col 8, lines 1-4, "Using the dynamic class reloading mechanism, only a changed class and its dependent classes are reloaded, thus limiting the number of files that are affected on the application server") Examiner Comments: Susarla discloses replacing different previously-loaded classes with their respective changed classes in memory depending on which class changed, which corresponds to generating different application mutants by swapping different first previously-compiled bytecode files with different mutated bytecode files.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the in-memory class-reloading mechanism of Susarla into the mutation testing system of Abhishek and Dern in order to generate each application mutant by swapping only the recompiled mutated class into memory in place of the previously-compiled class, without recompiling or reloading the unchanged classes, thereby limiting the number of files that are affected and avoiding the overhead of restarting or fully recompiling the application for each mutant (Susarla, Col 8, lines 1-4), which further reduces the per-mutant testing overhead sought by Abhishek and Dern.
Regarding Claim 20, Abhishek, and Dern teach
The one or more non-transitory computer-readable media of Claim 16.
Abhishek and Dern did not specifically teach
wherein the computer-executable instructions cause the one or more processors to generate the different application mutants by swapping first sets of previously-compiled bytecode files with the different sets of the mutated bytecode files, without recompiling other previously-compiled bytecode files.
However, Susarla teaches
wherein the computer-executable instructions cause the one or more processors to generate the different application mutants by swapping first sets of previously-compiled bytecode files with the different sets of the mutated bytecode files, without recompiling other previously-compiled bytecode files. (Col 8, lines 1-4, "Using the dynamic class reloading mechanism, only a changed class and its dependent classes are reloaded, thus limiting the number of files that are affected on the application server") Examiner Comments: Susarla discloses swapping, in memory, the set of previously-loaded classes affected by a change (the changed class and its dependents) with the recompiled versions while leaving the other previously-compiled classes unchanged, which corresponds to swapping first sets of previously-compiled bytecode files with the different sets of mutated bytecode files without recompiling other previously-compiled bytecode files.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the in-memory class-reloading mechanism of Susarla into the mutation testing system of Abhishek and Dern in order to generate each application mutant by swapping only the recompiled mutated class into memory in place of the previously-compiled class, without recompiling or reloading the unchanged classes, thereby limiting the number of files that are affected and avoiding the overhead of restarting or fully recompiling the application for each mutant (Susarla, Col 8, lines 1-4), which further reduces the per-mutant testing overhead sought by Abhishek and Dern.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMIR SOLTANZADEH whose telephone number is (571)272-3451. The examiner can normally be reached M-F, 9am - 5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wei Mui can be reached at (571) 272-3708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMIR SOLTANZADEH/Examiner, Art Unit 2191
/WEI Y MUI/Supervisory Patent Examiner, Art Unit 2191
Prosecution Insights