CTNF 18/678,587 CTNF 78924 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. This Office Action is in response to the filing date of 5/30/2024. Claims 1-20 are pending and have been considered below. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because the claimed invention is directed to an abstract idea without significantly more. Per claim 1: Under Prong 1 Step 2A , the claim recites “comparing the first stack trace data and flaky test case data…” and “based at least in part on the comparing, determining that the first test case is a flaky test case” under its broadest reasonable interpretation, cover performance of the limitations in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas. Under Prong 2 Step 2A , the judicial exception is not integrated into a practical application. The claim recites additional elements of “a system comprising at least one processor” are merely instructions to implement the abstract idea on a computer, or merely uses a computer, with instructions, as a tool to perform the abstract idea according to MPEP 2106.06(f). Thus, not indicative of an integration into a practical application. The claim further recites additional elements of “accessing first stack trace data…” and “providing, to a user, an indication that the first test case is a flaky test case” which are merely insignificant extra solution activity of gathering data and displaying data, which do not integrate the judicial exception into a practical application. Accordingly, the additional elements do not integrate the recited judicial exception into a practical application, and the claim is therefore directed to the judicial exception. See MPEP 2106.05(g). Under Step 2B , the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements “a system comprising at least one processor” are generic computer components used as tools to perform the abstract idea. As to the additional elements of “accessing first stack trace data…” and “providing, to a user, an indication that the first test case is a flaky test case” the courts have identified that receiving data and displaying/presenting the output of the abstract idea is well-understood, routine, and conventional activity. See MPEP 2106.05(d). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claim is not patent eligible. Per claims 2, 11, and 20: Under Prong 1 Step 2A , the claim recites “comparing the second stack trace data to flaky test case data…”, “determining that the comparing does not indicate that the second test case is a flaky test case”, “performing a set of additional executions of the second test case”, “determining that the software application passed at least one of the set of additional executions of the second test case”, and “updating the flaky test case data based at least in part on the second stack trace data” under its broadest reasonable interpretation, cover performance of the limitations in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. Thus, these limitations recite and fall within the “Mental Processes” grouping of abstract ideas. Under Prong 2 Step 2A , the judicial exception is not integrated into a practical application. The claim recites additional elements of “accessing second stack trace data…” merely insignificant extra solution activity of gathering data, which does not integrate the judicial exception into a practical application. Accordingly, the additional element does not integrate the recited judicial exception into a practical application, and the claim is therefore directed to the judicial exception. See MPEP 2106.05(g). Under Step 2B , the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application regarding the additional element of “accessing first stack trace data…”, the courts have identified that receiving data is well-understood, routine, and conventional activity. See MPEP 2106.05(d). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claim is not patent eligible. Per claims 3-7, 9, 12-16, and 18: These claims recite limitations that under its broadest reasonable interpretation cover performance of the limitations in the mind. Per claims 8 and 17: These claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception Per claim 10: Under Prong 1 Step 2A , the claim recites “comparing the first stack trace data and flaky test case data…” and “based at least in part on the comparing, determining that the first test case is a flaky test case” under its broadest reasonable interpretation, cover performance of the limitations in the mind but for the recitation of generic computer components. Nothing in the claim element precludes these steps from practically being performed in the mind through observation, evaluation, judgment and/or opinion. Under Prong 2 Step 2A , the judicial exception is not integrated into a practical application. The claim recites additional elements of “accessing first stack trace data…” and “providing, to a user, an indication that the first test case is a flaky test case” which are merely insignificant extra solution activity of gathering data and displaying data, which do not integrate the judicial exception into a practical application. Accordingly, the additional elements do not integrate the recited judicial exception into a practical application, and the claim is therefore directed to the judicial exception. See MPEP 2106.05(g). Under Step 2B , the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application regarding the additional elements of “accessing first stack trace data…” and “providing, to a user, an indication that the first test case is a flaky test case” the courts have identified that receiving data and displaying/presenting the output of the abstract idea is well-understood, routine, and conventional activity. See MPEP 2106.05(d). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claim is not patent eligible. Per claim 19: Under Prong 1 Step 2A , the claim recites “comparing the first stack trace data and flaky test case data…” and “based at least in part on the comparing, determining that the first test case is a flaky test case” under its broadest reasonable interpretation, cover performance of the limitations in the mind but for the recitation of generic computer components. Nothing in the claim element precludes these steps from practically being performed in the mind through observation, evaluation, judgment and/or opinion. Under Prong 2 Step 2A , the judicial exception is not integrated into a practical application. The claim recites additional elements of “a non-transitory machine medium” and “at least one processor” are merely instructions to implement the abstract idea on a computer, or merely uses a computer, with instructions, as a tool to perform the abstract idea according to MPEP 2106.06(f). Thus, not indicative of an integration into a practical application. The claim further recites additional elements of “accessing first stack trace data…” and “providing, to a user, an indication that the first test case is a flaky test case” which are merely insignificant extra solution activity of gathering data and displaying data, which do not integrate the judicial exception into a practical application. Accordingly, the additional elements do not integrate the recited judicial exception into a practical application, and the claim is therefore directed to the judicial exception. See MPEP 2106.05(g). Under Step 2B , the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements “a non-transitory machine medium” and “at least one processor” are generic computer components used as tools to perform the abstract idea. As to the additional elements of “accessing first stack trace data…” and “providing, to a user, an indication that the first test case is a flaky test case” the courts have identified that receiving data and displaying/presenting the output of the abstract idea is well-understood, routine, and conventional activity. See MPEP 2106.05(d). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. Thus, the claim is not patent eligible. Claim Rejections - 35 USC § 102 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-15 AIA Claim s 1-20 are rejected under 35 U.S.C. 102( a)(1 ) as being anticipated by “Reducing Effort for Flaky Test Detection through Dynamic Program Analysis” to Aaron Tacke . Per claims 1, 10, and 19, Aaron Tacke teaches a system for debugging a software application, comprising: at least one processor programmed to perform operations comprising: accessing first stack trace data, the first stack trace data describing a plurality of function calls made by the software application during a failed execution of a first test case (see at least page 16, 3 rd paragraph “… a CSV file for the tracer output is created that enables easy further processing. Every function call entry, together with its number of calls and the identifying test information (obtained during test preparation), is written to this file …”; see also Figure 4.2 ); comparing the first stack trace data and flaky test case data, the flaky test case data describing at least one function call made by the software application during execution of at least one flaky test case (see at least page 19, “ Algorithm 1” ), the at least one flaky test case comprising a first flaky test case that the software application passed during one execution of the first flaky test case and failed during another execution of the first flaky test case (see at least page 4, 2 nd paragraph “ Definition 1 An NOD flaky test is a test that can both pass and fail when executed multiple times without changing the execution environment, i.e., the order of tests, the executed test code, or the CUT …”); based at least in part on the comparing, determining that the first test case is a flaky test case (see at least page 19, 1 st paragraph “… We construct a binary classifier based on the functions C that classifies tests as flaky if and only if their function call trace includes at least one member of C. Thus, functions C characterize the flaky tests T well, if the corresponding classifier has the maximum recall (percentage of flaky tests that are identified) and the maximum precision (percentage of flaky tests among all identified ones) …”); and providing, to a user, an indication that the first test case is a flaky test case (see at least page18, paragraph 5 “… Given an objective function that quantifies the correlation between a set of functions and a set of tests, we can optimize characteristic functions for NOD flaky tests using this optimizer …”). Per claims 2, 11, and 20, Aaron Tacke further teaches: accessing second stack trace data, the second stack trace data describing a plurality of function calls made by the software application during a failed execution of a second test case; comparing the second stack trace data to flaky test case data; determining that the comparing does not indicate that the second test case is a flaky test case; performing a set of additional executions of the second test case; determining that the software application passed at least one of the set of additional executions of the second test case; and updating the flaky test case data based at least in part on the second stack trace data (see at least page 19, “ Algorithm 1 ”). Per claims 3 and 12, Aaron Tacke further teaches: accessing first error message data describing the failed execution of the first test case; and comparing the first error message data and the flaky test case data, the determining that the first test case is a flaky test case also being based at least in part on the comparing of the first error message data and the flaky test case data (see at least page 22, paragraphs 2-6 “ During rerunning, a test is only said to be flaky if a passing and a failing execution occurs. Therefore, no false positives (non-flaky tests claimed to be flaky) are possible…we require the probability Ut(n) for unveiling flakiness after n reruns for a test t. In accordance with Gruber et al., we assume that different runs are independent from each other. Given the probability Pt(pass) that t passes and the probability Pt( f ail) that t fails, we can calculate …”). Per claims 4 and 13, Aaron Tacke further teaches: the flaky test case data describing a plurality of function calls made by the software application during execution of the at least one flaky test case (see at least page 3, paragraph 2 “ The term flaky tests describe tests with non-deterministic outcomes [2]. This means, that multiple executions of such tests may produce different outcomes, without changing the test code or CUT ”), the determining that the first test case is a flaky test case comprising determining that a number of common function calls described by both the first stack trace data and the flaky test case data meets a threshold (see at least page 19, “ Algorithm 1 ”) Per claims 5 and 14, Aaron Tacke further teaches the flaky test case data describing a plurality of function calls made by the software application during execution of the at least one flaky test case (see at least page 3, paragraph 2 “ The term flaky tests describe tests with non-deterministic outcomes [2]. This means, that multiple executions of such tests may produce different outcomes, without changing the test code or CUT ”), the determining that the first test case is a flaky test case comprising determining that the plurality of function calls made by the software application during the failed execution of the first test case were also made by a threshold number of the at least one flaky test case (see at least page 22, paragraph 2 “… a test is only said to be flaky if a passing and a failing execution occurs. Therefore, no false positives (non-flaky tests claimed to be flaky) are possible. The efficiency of rerunning a given set of tests to detect flakiness therefore only depends on the number of detected flaky tests and the cost of execution …”). Per claims 6 and 15, Aaron Tacke further teaches the determining that the first test case is a flaky test case further comprising determining that a threshold number of error messages described by first error message data describing the failed execution of the first test case match at least one error message described by the flaky test case data (see at least page 22, paragraphs 2-6 “ During rerunning, a test is only said to be flaky if a passing and a failing execution occurs. Therefore, no false positives (non-flaky tests claimed to be flaky) are possible…we require the probability Ut(n) for unveiling flakiness after n reruns for a test t. In accordance with Gruber et al., we assume that different runs are independent from each other. Given the probability Pt(pass) that t passes and the probability Pt(fail) that t fails, we can calculate …”). Per claims 7 and 16, Aaron Tacke further teaches: before the comparing, filtering the first stack trace data to remove at least a portion of the plurality of function calls (see page 17, paragraph 5 “… The manual selection of characteristic functions serves as a baseline for comparisons with automatically generated results. We use existing literature, the executed code (Test Code, CUT and information about library functions), as well as the dataset (see Section 4.1.1) with the results of our function call tracer …”; see also page 18, paragraph 5 “ For our automatic selection, we decide to use a greedy algorithm. Since our manual procedure was mostly greedy (in terms of selecting functions successively) and yielded useful results, this seemed more promising than some alternatives: We have too little labeled data for machine learning approaches but too many traced functions for computing the optimal subset of characteristic functions. Our optimizer starts with an empty set and adds functions (from a freely selectable set of functions) one after another (see Algorithm 1). Each time, the function that maximizes a given objective function is added, until no improvement (a higher value of the objective function) is possible. Given an objective function that quantifies the correlation between a set of functions and a set of tests, we can optimize characteristic functions for NOD flaky tests using this optimizer …”; see also at least page 34, paragraph 4 “ After removing the functions denoted as Solo and Cryptic, the remaining characteristic functions of the optimization on the root causes only expose 10.7% of all tests containing 27.4% of NOD flaky tests …”). Per claims 8 and 17, Aaron Tacke further teaches: the at least a portion of the plurality of function calls comprising at least one function call not associated with the first test case (see at least page 17, paragraph 5 “… The manual selection of characteristic functions serves as a baseline for comparisons with automatically generated results. We use existing literature, the executed code (Test Code, CUT and information about library functions), as well as the dataset (see Section 4.1.1) with the results of our function call tracer …”). Per claims 9 and 18, Aaron Tacke further teaches: before the comparing, removing at least a portion of numerical values of the first stack trace data (see at least page 34, paragraph 4 “ After removing the functions denoted as Solo and Cryptic, the remaining characteristic functions of the optimization on the root causes only expose 10.7% of all tests containing 27.4% of NOD flaky tests …”) . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US20200183821 relates to flaky testing. US20230229586 relates to flaky testing. Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHILLIP H NGUYEN whose telephone number is (571)270-1070. The examiner can normally be reached Monday-Friday 9:00AM-5:00PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wei Zhen can be reached at (571) 272-3708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /PHILLIP H NGUYEN/Primary Examiner, Art Unit 2191 Application/Control Number: 18/678,587 Page 2 Art Unit: 2191 Application/Control Number: 18/678,587 Page 3 Art Unit: 2191 Application/Control Number: 18/678,587 Page 4 Art Unit: 2191 Application/Control Number: 18/678,587 Page 5 Art Unit: 2191 Application/Control Number: 18/678,587 Page 6 Art Unit: 2191 Application/Control Number: 18/678,587 Page 7 Art Unit: 2191 Application/Control Number: 18/678,587 Page 8 Art Unit: 2191 Application/Control Number: 18/678,587 Page 9 Art Unit: 2191 Application/Control Number: 18/678,587 Page 10 Art Unit: 2191 Application/Control Number: 18/678,587 Page 11 Art Unit: 2191 Application/Control Number: 18/678,587 Page 12 Art Unit: 2191 Application/Control Number: 18/678,587 Page 13 Art Unit: 2191 Application/Control Number: 18/678,587 Page 14 Art Unit: 2191
Prosecution Insights