DETAILED ACTION
This action is responsive to the application filed on July 24, 2024.
Claims 1-20 are pending and presented to examination.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Foreign Priority
The foreign priority date considered for this application is February 23, 2024.
Drawings
The drawings filed on July 15, 2024 and July 24, 2024 are acceptable for examination purposes.
Information Disclosure Statement
As required by M.P.E.P. 609, the applicant’s submission of the Information Disclosure Statement dated July 24, 2024 is acknowledged by the examiner and the cited references have been considered in the examination of the claims now pending.
Claim Objections
Claims 14-20 are objected to because of the following informalities: Claim 14 recites the limitation “determine, from among the programming elements, a fault element, [[,]] the determining based on a result of combining the rule-based first ranking with the machine learning-based second ranking, wherein the combining is based on a fault feature of a fault occurring in the execution of the program.” in lines 10-13 . Appropriate correction is required. Please amend the claim language as indicated in bold.
Dependent claims 15-20 do not overcome the deficiency of the base claim and, therefore, are objected for the same reasons as the base claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention recites a judicial exception, is directed to that judicial exception, an abstract idea, as it has not been integrated into practical application and the claims further do not recite significantly more than the judicial exception. Examiner has evaluated the claims under the framework provided in the 2019 Patent Eligibility Guidance published in the Federal Register 01/07/2019 and has provided such analysis below.
Step 1: Claims 1-13 are directed to methods and fall within the statutory category of processes; and Claims 14-20 are directed to devices and fall within the statutory category of machines. Therefore, “Are the claims to a process, machine, manufacture or composition of matter?” Yes.
In order to evaluate the Step 2A inquiry “Is the claim directed to a law of nature, a natural phenomenon or an abstract idea?” we must determine, at Step 2A Prong 1, whether the claim recites a law of nature, a natural phenomenon or an abstract idea and further whether the claim recites additional elements that integrate the judicial exception into a practical application.
Step 2A Prong 1:
Claims 1 and 14 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components, also covers mathematical concepts. That is, the limitations:
a) “determining a rule-based first ranking of fault probability of the respective programming elements, based on whether the test case passes and based on whether each programming element is executed;”. This limitation recites the abstract idea as a combination of mathematical concept and mental process. The recited “rule-based first ranking of fault probability” is computed via the SBFL (spectrum-based fault-localization) suspiciousness formulas set out in the present specification at paragraphs [0095]–[0102] and Equations 1, 2, 6, and 8 — closed-form mathematical relationships between the counts of failed/passed tests that execute each element and a probability score. The recited evaluation — that programming elements more correlated with failed test outcomes are more likely faulty — is a judgment / evaluation an ordinarily-skilled human debugger can perform with pen and paper for a small enough program. The sorting of elements by score is itself a mental and mathematical operation. b) “determining a machine learning-based second ranking of the programming elements based on a result of applying a machine learning model to element features of the respective programming elements;”. This limitation likewise recites the abstract idea as a combination of mathematical concept and mental process. The recited “machine learning model”, as understood by one of ordinary skill in the art, is a parameterized mathematical function from an input feature vector to an output score; computing such a function and sorting by the output is mathematical. The underlying evaluation — predicting which programming elements are most likely faulty based on observed features — is a pattern-recognition judgment a human can perform.
c) “determining, from among the programming elements, a fault element, the determining based on a result of combining the rule-based first ranking with the machine learning-based second ranking, wherein the combining is based on a fault feature of a fault occurring in the execution of the program.”. This limitation likewise recites the abstract idea as a combination of mathematical concept and mental process. The recited combining of two rankings is a weighted mathematical aggregation. The recited fault-feature-dependent parameterization of the weights is a mathematical parameterization. The recited identification of “a fault element” from the combined ranking is the mental judgment of which element is the most likely fault from the sorted list. That is, nothing in the claim elements precludes the step from practically being performed in the mind or with a pen and paper, (i.e., “determining”) can be performed in the human mind though observation, evaluation, judgment, opinion with the aid of pen and paper, which many of these steps are also directed to mathematical concepts. Thus, these limitations fall within the “Mental Processes” and “Mathematical Concepts” grouping of abstract ideas.
Therefore, Yes, claims 1 and 14 recite judicial exceptions.
The claims have been identified to recite judicial exceptions, Step 2A Prong 2 will evaluate whether the claims are directed to the judicial exception.
Step 2A Prong 2:
This judicial exception is not integrated into a practical application. The claims recite the following additional elements: “an electronic device”, “a non-transitory computer-readable storage medium”, “one or more processors”. The additional elements are merely instructions to implement an abstract idea on a computer, or merely using a generic computer or computer components as a tool to perform the abstract idea (see MPEP 2106.05(f)).
Additional element 1 — “executing a program corresponding to source code, ..., the executing based on a test case”. This recitation is mere data gathering — running the program with a test case produces the test pass/fail outcomes and the per-element execution coverage that feed the downstream analytical limitations. As such, it is insignificant extra-solution activity. MPEP § 2106.05(g); see In re Bilski, 545 F.3d 943, 963 (Fed. Cir. 2008) (data gathering); CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1371 (Fed. Cir. 2011).
Additional element 2 — “a machine learning model”. This is recited at the highest level of generality, with no specific architecture, no specific training methodology, no specific input-output mapping, and no specific structural improvement to the model itself. The recitation amounts to applying the abstract idea on a generic machine-learning platform. MPEP § 2106.05(f) (mere instructions to apply); MPEP § 2106.05(h) (field of use). See PurePredictive, Inc. v. H2O.ai, Inc., No. 17-CV-03049, 2017 WL 3721480 (N.D. Cal. Aug. 29, 2017), aff'd, 741 F. App'x 802 (Fed. Cir. 2018) (generic application of machine learning to data is abstract). Considered as an ordered combination, the additional elements add nothing of substance to the abstract idea beyond instructing one of ordinary skill to apply the mathematical and mental concepts on a generic computer using generic data gathering and a generic ML platform. This is the same pattern the Federal Circuit held to be unintegrated in Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1353–55 (Fed. Cir. 2016) (collecting data, analyzing it with mathematical techniques, and presenting the results) and in SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1167–68 (Fed. Cir. 2018) (“an advance of mathematical techniques” on a generic computer does not transform an abstract idea into patent-eligible subject matter); see also In re Bd. of Trustees of Leland Stanford Jr. Univ., 989 F.3d 1367, 1373 (Fed. Cir. 2021) (statistical method run on a computer remains abstract).
The claims as a whole do not improve the functioning of a computer (cf. Enfish, LLC v. Microsoft Corp., 822 F.3d 1327 (Fed. Cir. 2016)) nor effect a technical solution to a technical problem in the sense of McRO, Inc. v. Bandai Namco Games America Inc., 837 F.3d 1299 (Fed. Cir. 2016). The asserted improvement is to the accuracy of an analytical output (a ranked list of suspicious programming elements) — i.e., the improvement is to the abstract analysis itself, not to how the computer functions. Such an improvement is not a practical application. SAP America, 898 F.3d at 1167.
Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea, thus failing to integrate the abstract idea into a practical application.
Therefore, “Do the claims recite additional elements that integrate the judicial exception into a practical application? No, these additional elements do not integrate the abstract idea into a practical application and they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
After having evaluating the inquires set forth in Steps 2A Prong 1 and 2, it has been concluded that claims 1 and 14 not only recites a judicial exception but that the claim is directed to the judicial exception as the judicial exception has not been integrated into practical application.
Step 2B:
As discussed above with respect to integration of the abstract idea into a practical application, the additional elements “an electronic device”, “a non-transitory computer-readable storage medium (dependent claim 13)”, “one or more processors” are generic computer components used as tools to perform the abstract idea.
The recited electronic device and processor configured to execute instructions are well-understood, routine, and conventional generic computer components. MPEP § 2106.05(d)(II) (examples of WURC computer functions include performing repetitive calculations, receiving / transmitting data over a network, and storing / retrieving information). The recited “executing a program ... based on a test case” is the well-understood, routine, and conventional practice of automated software testing; it is treated as routine in the very prior art applied in the present Office Action against claims 1–20 under 35 U.S.C. § 103 (see, e.g., Li, section 4.1 “Implementation Details”; Zou, section 3.3; Bier, paragraph [0028]), confirming that automated test execution to gather pass/fail outcomes and per-element execution coverage is conventional. The recited “machine learning model”, recited at high level of generality with no specific architecture, captures well-understood, routine, and conventional uses of machine learning for ranking / classification tasks in the field; this too is treated as routine in the prior art (see Li, generally, applying generic MLP and related deep-learning architectures to fault-localization data).
Berkheimer v. HP Inc., 881 F.3d 1360, 1368 (Fed. Cir. 2018), requires that a finding of WURC be supportable by evidence. The references applied against the claims under 35 U.S.C. § 103 — Li, Zou, and Bier — collectively confirm that automated test execution, ML application to fault-localization data, and combination of ranking outputs were all well-understood, routine, and conventional in the field before the effective filing date. The additional elements therefore do not supply an inventive concept. Accordingly, the additional elements recited in the claims cannot provide an inventive concept. In addition, after further evaluation the claim as a whole doesn’t improve any function of a computer or to any other technology or technical field. Thus, the claims are not patent eligible.
Therefore, “Do the claims recite additional elements that amount to significantly more than the judicial exception? No, these additional elements, alone or in combination, do not amount to significantly more than the judicial exception.
Having concluded analysis within the provided framework, Claims 1 and 14 do not recite patent eligible subject matter under 35 U.S.C. § 101. Dependent claims 2-5, 8-10, 15 and 17-18 adds further mathematical and / or mental-process detail to the analytical operations of the independent claim — for example, the sub-steps of computing rule-based probability scores, adjusting those scores by various rules, sorting by score, computing weighted combinations, and parameterizing weights based on fault characteristics. These additional limitations further recite the same abstract idea (mathematical concepts and mental processes) without introducing any new additional element. MPEP § 2106.04(a)(2)(I), (III). They therefore do not integrate the abstract idea into a practical application and do not supply an inventive concept.
Dependent claims 6 and 16 further recite the sub-operations of the machine-learning-based second ranking — obtaining element features, inferring probability scores by applying features to a “machine learning model,” and sorting by score. As to the abstract-idea content, this further recites mathematical concepts and mental processes. As to the additional element of the “machine learning model,” the model is recited at the highest level of generality, with no specific architecture, no training methodology, and no structural improvement specified. This recitation is mere instructions to apply the abstract idea via generic ML (MPEP § 2106.05(f)), generally links to a particular technological environment (MPEP § 2106.05(h)), and at Step 2B is well-understood, routine, and conventional (MPEP § 2106.05(d)). See PurePredictive, 741 F. App'x at 803. Dependent claim 7 recites a Markush list of alternative types of element features used as inputs to the machine-learning model. Selecting a particular type of data on which the abstract analysis operates is insignificant extra-solution activity under MPEP § 2106.05(g). See In re Bilski, 545 F.3d at 963. The claim therefore does not integrate or supply an inventive concept.
Dependent claims 11, 12, 19, and 20 recite operations performed before and after the analytical limitations — receiving a change request or error report, executing the program on the updated version, outputting information on the determined fault element, changing the source code, verifying the updated version, and distributing information on the updated version to an external device. These operations are pre-solution and post-solution activity: gathering inputs to the abstract analysis (a change request or an error report), and acting on the output of the abstract analysis (modifying code, verifying, and distributing). They are insignificant extra-solution activity under MPEP § 2106.05(g) (data gathering and use-of-results) and, taken as an ordered workflow, are well-understood, routine, and conventional in the field of software development under MPEP § 2106.05(d). The prior art applied against these same claims under 35 U.S.C. § 103 — Bier, particularly paragraphs [0006], [0007], [0028], and [0029] — itself describes the iterative change / test / fault-localize / verify / recommend workflow as a conventional practice. See Berkheimer, 881 F.3d at 1368. The additional workflow limitations therefore do not integrate the abstract idea into a practical application and do not supply an inventive concept.
Therefore, Claims 1-20 do not recite patent eligible subject matter under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-7 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 2 recites limitation “determining the rule-based first ranking by sorting the adjusted rule-based possibility scores.” There is insufficient antecedent basis for this limitation in the claim. Claim 3 recites “wherein the adjusting the rule-based possibility scores comprises: when the test case fails to pass for a programming element, increasing the rule-based probability score of the programming element corresponding to a function that is indicated by a call stack of the program.”. There is insufficient antecedent basis for this limitation in the claim. Claim 6 recites “wherein the determining the machine learning-based second ranking comprises: obtaining element features of the programming elements, respectively; inferring machine learning-based probability scores of the respective programming elements, wherein each machine learning-based probability score represents the probability that a corresponding programming element has a fault and is generated by applying the corresponding element feature to the machine learning model; and determining the machine learning-based second ranking by sorting machine learning-based possibility scores.”. It is unclear whether “probability score” and “possibility score” are the same. For purpose of examination, it will be interpreted as the same.
Dependent claim 4-5 and 7 do not overcome the deficiency of the base claim and, therefore, are rejected for the same reasons as the base claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-8 and 13-17 are rejected under 35 U.S.C. 103 as being unpatentable over Xia Li et al. ("DeepFL: Integrating Multiple Fault Diagnosis Dimensions for Deep Fault Localization", hereinafter Li – IDS 07/24/2024) in view of Daming Zou et al. (“An Empirical Study of Fault Localization Families and Their Combinations”, hereinafter Zou).
With respect to claim 1, Li teaches a method performed by an electronic device (To the extent the preamble is given patentable weight, Li discloses a method performed by an electronic device. Li implements DeepFL using the TensorFlow framework executing on a Dell machine (Li, section 4.1 Implementation Details), which is an "electronic device" within the broadest reasonable interpretation ("BRI") of the term), the method comprising: executing a program corresponding to source code, the program comprising instructions comprising programming elements, the executing based on a test case (The "programming elements" of the claim are defined in the present specification at paragraph [0058] as any of "a file, function, line of code, module, or object of the source code." Under BRI, "programming elements" encompass at least source-code methods and source-code statements. Li discloses executing programs from the Defects4J V1.2.0 benchmark of real-world Java programs. Li, section 4.1 ("Implementation Details"), expressly states that DeepFL was evaluated on "all the 6 Defects4J V1.2.0 subjects, totalling 395" real bugs. Each Defects4J program corresponds to Java source code and comprises instructions and programming elements (Java methods and Java statements). Li further discloses executing each program "based on a test case" by running the JUnit test cases that ship with each Defects4J fault to collect spectrum-based features. Li, section 2.1 under the heading "Spectrum-based Fault Localization", explains that SBFL operates by analyzing the coverage data of failed/passed tests — i.e., which programming elements are executed under each test — and Li, section 3.2 under the heading "Spectrum-based Suspiciousness", confirms that DeepFL collects these execution-derived counts as the basis for its 34 SBFL features per element. determining a rule-based first ranking of fault probability of the respective programming elements, based on whether the test case passes and based on whether each programming element is executed (Li discloses determining a rule-based first ranking of fault probability of programming elements by computing SBFL (spectrum-based fault-localization) suspiciousness scores. Li, section 3.2 under the heading "Spectrum-based Suspiciousness", states that DeepFL uses 34 SBFL features per programming element, with each feature being a rule-based formula such as Tarantula, Ochiai, DStar, Jaccard, Ochiai2, and Kulczynski2 (Li, section 2.1 "Spectrum-based Fault Localization", and Li, section 3.2 "Spectrum-based Suspiciousness"), each of which computes a suspiciousness score per programming element from the four counts of failed/passed tests that execute or do not execute the element (i.e., ef, ep, nf, np). Sorting programming elements by SBFL score yields a ranked list — i.e., a "rule-based first ranking of fault probability" within the BRI of claim 1. Several of Li's SBFL formulas (Ochiai, DStar, Jaccard, Tarantula) correspond directly to the rule-based ranking formulas set out in the present specification at Equations 1, 2, 6, and 8 (paragraphs [0095]–[0102]).). determining a machine learning-based second ranking of the programming elements based on a result of applying a machine learning model to element features of the respective programming elements (Li discloses determining a machine learning-based second ranking by applying an MLP-based deep-learning model (denoted "MLP_DFL") to element features of the respective programming elements. Li, section 3.1.3 ("Tailored MLP Based Neural Network"), describes the MLP_DFL model variant that receives an element-feature vector x_i for each programming element i and, via Li Equation (8), produces a fully-connected output from which a per-element probability of being faulty is computed (Li, section 3.1.1, Equation (2), defining the per-element output of an MLP as a probability vector ŷ_i = [ŷ_i^(1), ŷ_i^(2)] in which ŷ_i^(1) represents the probability that the corresponding programming element belongs to the buggy class). Li, section 3.2 ("DeepFL Features"), identifies the element features fed to the model as comprising the SBFL features (Li, section 3.2 under the heading "Spectrum-based Suspiciousness"), MBFL features (Li, section 3.2 under the heading "Mutation-based Suspiciousness"), complexity-based fault-proneness features comprising 21 code-complexity metrics plus 16 bytecode metrics per element (Li, section 3.2 under the heading "Complexity-based Fault Proneness"; Li, Table 1 "Studied code metrics"), and textual-similarity features comprising 15 similarity scores between source-code methods and failed tests (Li, section 3.2 under the heading "Textual Similarity Information"). Sorting the programming elements by ŷ_i^(1) values yields a second ranked list — i.e., a "machine learning-based second ranking of the programming elements" within the BRI of claim 1.). Li is silent to disclose, however in an analogous art, Zou teaches: determining, from among the programming elements, a fault element, the determining based on a result of combining the rule-based first ranking with the machine learning-based second ranking, wherein the combining is based on a fault feature of a fault occurring in the execution of the program (The concept of this limitation is the determination of a fault element from a combination of two separately-produced rankings — one rule-based, one ML-based — where the combining depends on a feature of the observed fault. Li conceptually teaches the overall fault-localization objective of "determining, from among the programming elements, a fault element" (Li, section 5, presenting Top-1, Top-3, Top-5 fault-localization results based on the ranked output of DeepFL). Li further teaches that combining multiple sources of fault-diagnosis information improves fault-localization performance (Li, Abstract: "DeepFL ... can learn from existing fault diagnosis information ... to identify the most suspicious code for a given fault"). However, in Li the SBFL scores are fed as input features into the MLP_DFL model, which produces a single final ranking. Li does not, in so many words, take its standalone SBFL ranking and its MLP_DFL ranking as two separate ranked lists and combine those two ranked lists into a final ranking with weights derived from a feature of the observed fault. That conceptual gap is filled by Zou. Zou, section 3.3.7 ("Learning to Rank"), discloses applying a learning-to-rank model (RankSVM) that combines suspiciousness scores reported by multiple FL techniques drawn from different FL families — including the SBFL family (Zou, section 2.1 and section 3.3.1, using Ochiai and DStar) and learning-based techniques represented by MULTRIC, Savant, FLUCCS, and TraPT (Zou, section 4.6 and section 5.1, comparing CombineFL against these as state-of-the-art learning-based FL approaches). For each programming element e, Zou associates a suspiciousness vector Suspiciousness(e) = ⟨st₁(e), st₂(e), …⟩, where stᵢ(e) is the suspiciousness score that technique tᵢ reports for e (Zou, section 3.3.7). The trained RankSVM model assigns a learned weight to each technique's contribution to the final ranking. Zou, section 4.3 Finding 3.2 reports the combined technique improves 200%/63%/51%/31% at Top-1/3/5/10 over the best standalone, and Zou, section 4.6 Table 13 confirms the combined technique outperforms the four state-of-the-art learning-based FL approaches (MULTRIC by 133%, Savant by 167%, FLUCCS by 11%, TraPT by 18%) at Top-1 method granularity.
Under BRI, "combining the rule-based first ranking with the machine learning-based second ranking" reads on Zou's learning-to-rank combination of an SBFL technique's suspiciousness scores with a learning-based FL technique's suspiciousness scores — both of which are ranked lists (Zou, section 3.3) — to produce a final combined ranking. Equivalently, the trained learning-to-rank model of Zou applies a learned weight to each contributing technique's ranking, so that the final combined score for each programming element is a weighted combination of the SBFL-derived score (rule-based first ranking score) and the ML-derived score (machine-learning-based second ranking score). The fault element of claim 1 is then the top-ranked element of the final combined ranking.
Regarding the wherein clause — "wherein the combining is based on a fault feature of a fault occurring in the execution of the program" — the present specification at paragraph [0071] gives non-limiting examples of "fault feature" including the lifespan of a fault, the stability of the fault, and the fault type of the fault. The specification does not narrowly limit "fault feature" to these specific examples, and the wherein clause under BRI is met by any combining whose weights depend on features of the fault observed in the execution. In Zou, the learning-to-rank weights are trained on the empirical fault data of the Defects4J benchmark, i.e., on the observed pass/fail counts, the observed element execution patterns, and the observed crash/non-crash characteristics of each fault (Zou, section 3.3 and section 4.2). Under BRI, the weights learned from such empirical fault data are "based on a fault feature of a fault occurring in the execution of the program," because the weights reflect — at a population level — which technique's ranking is more reliable for faults exhibiting given execution-derived features).
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to combine Li with Zou by applying Zou's learning-to-rank combining methodology (Zou, section 3.3.7) to Li's separately-produced SBFL rule-based ranking (Li, section 3.2 "Spectrum-based Suspiciousness") and Li's MLP_DFL machine-learning ranking (Li, section 3.1.3 "Tailored MLP Based Neural Network"), so as to obtain a combined ranking whose top element is identified as the fault element. The motivation is supplied by Zou's empirical demonstration that combining fault-localization rankings from different families yields a 200% Top-1 improvement over the best standalone technique (Zou, section 4.3.2 Finding 3.2), together with Zou's express recommendation against using any technique standalone (Zou, section 6.1).
With respect to claim 2, Li teaches wherein the determining the rule-based first ranking comprises: determining rule-based probability scores representing the probabilities that respective programming elements have a fault, based on whether the test case passes and whether the programming element is executed (Li discloses determining rule-based probability scores representing per-element fault probabilities, computed from the test pass/fail outcomes and the element execution information. Li, section 3.2 under the heading "Spectrum-based Suspiciousness", describes the 34 SBFL features as suspiciousness scores per element, each computed from the four counts ef, ep, nf, np (Li, section 2.1 "Spectrum-based Fault Localization"), where ef is the number of failed tests that execute the element, ep is the number of passed tests that execute the element, and nf, np are the corresponding "do-not-execute" counts. Under BRI, each such suspiciousness score is a "rule-based probability score representing the probability that the corresponding programming element has a fault," computed "based on whether the test case passes and whether the programming element is executed.".) adjusting the rule-based probability scores of the respective programming elements based on a result of the execution of the program (Li discloses adjusting the rule-based probability scores based on the result of execution. Li, section 3.2 "DeepFL Features", describes that the SBFL scores are themselves derived from the result of program execution (ef, ep, nf, np are execution-derived counts), and Li further describes normalizing and aggregating the SBFL feature values across the 34 SBFL formulas as inputs to the MLP_DFL model (Li, section 3.1.3 "Tailored MLP Based Neural Network"; Li, section 3.2 under the heading "Spectrum-based Suspiciousness"). Under BRI, the computation, normalization, and aggregation of SBFL scores from the four execution-derived counts is "adjusting the rule-based probability scores … based on a result of the execution of the program.") and determining the rule-based first ranking by sorting the adjusted rule-based possibility scores (Li discloses determining the rule-based first ranking by sorting elements according to the SBFL suspiciousness scores. Li, section 5, presents fault-localization results in the form of Top-N rankings derived from the suspiciousness scores. Under BRI, presenting Top-N results by SBFL score is "determining the rule-based first ranking by sorting the adjusted rule-based possibility scores."). With respect to claim 3, Li is silent to disclose, however in an analogous art, Zou teaches wherein the adjusting the rule-based possibility scores comprises: when the test case fails to pass for a programming element, increasing the rule-based probability score of the programming element corresponding to a function that is indicated by a call stack of the program (The concept of this limitation is, on a failed test, increasing the SBFL score of a programming element that corresponds to a function appearing in the call stack of the program at the time of the failure. Li does not expressly disclose adjusting SBFL scores upward for elements indicated by the call stack of the program. Zou fills this gap. Zou, section 2.4 and section 3.3.3 ("Stack Trace Analysis"), discloses that, on a failed test that throws an exception (a "crash fault"), the stack trace consists of the active frames of the program at the time of the failure, each frame corresponding to a function call that has not yet returned. Zou, section 3.3.3, expressly assigns to each function corresponding to a frame at depth d a suspiciousness score of 1/d, with the maximum of such scores over all failed tests being attributed to the corresponding programming element. Under BRI, assigning a 1/d score to a function indicated by the call stack — and aggregating that score with other SBFL counts as part of the combined fault-localization output — is "increasing the rule-based probability score of the programming element corresponding to a function that is indicated by a call stack of the program," on a failed test. Zou, section 4.1.2 Finding 1.3, further reports that stack trace analysis is the most effective standalone technique on crash faults, locating 22% of crash faults at Top-1 versus 11% for the second-best, MBFL Metallaxis (Zou, Table 4).)
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to incorporate Zou's stack-trace teaching (Zou, section 3.3.3) into Li's SBFL framework on failed tests, because Zou identifies stack-trace analysis as the most effective standalone technique on crash faults (Zou, section 4.1.2 Finding 1.3) and teaches that all families contribute to overall results when combined (Zou, section 4.3.2 Finding 3.4). With respect to claim 4, Li is silent to disclose, however in an analogous art, Zou teaches wherein the adjusting the rule-based probability scores comprises: when the test case fails to pass, increasing a rule-based probability score of a programming element that corresponds to a termination point of the test case (The concept of this limitation is, on a failed test, increasing the SBFL score of a programming element at the termination point of the test — i.e., where execution terminated. The specification at paragraph [0109] defines a "termination point" as "the last execution point (e.g., a function or a line) in the source code when execution of the program for the test case terminates.". Zou fills this gap. Zou, section 3.3.2 ("Dynamic Slicing"), in addressing crash faults, states: "we use the execution of the statement that throws the exception as the slicing criterion." The statement that throws the exception on a crash fault is the termination point of the failed test case — i.e., the last execution point at which execution terminates. Further, Zou, section 3.3.3 ("Stack Trace Analysis"), assigns suspiciousness score 1/d to the function at stack-frame depth d; the top stack frame (d = 1) receives the maximum score (1/1 = 1), and that top frame corresponds to the termination point of the failed test case. Under BRI, both teachings independently disclose "increasing a rule-based probability score of a programming element that corresponds to a termination point of the test case" on a failed test.)
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to incorporate Zou's stack-trace teaching (Zou, section 3.3.3) into Li's SBFL framework on failed tests, because Zou identifies stack-trace analysis as the most effective standalone technique on crash faults (Zou, section 4.1.2 Finding 1.3) and teaches that all families contribute to overall results when combined (Zou, section 4.3.2 Finding 3.4). With respect to claim 5, Li is silent to disclose, however in an analogous art, Zou teaches wherein the adjusting the rule-based probability scores comprises: when the test case fails to pass, increasing a rule-based possibility score of a programming element that is changed compared to a previous version of the program (The concept of this limitation is, on a failed test, increasing the SBFL score of a programming element that has been changed in the current version of the program relative to a previous version.
Zou fills this gap. Zou, section 2.7 ("History-Based Fault Localization"), states the foundational premise: "Program files that contained more bugs in the past are likely to have more bugs in the future." Zou, section 3.3.6 ("History-Based Fault Localization"), specifically describes Bugspots, which "collects revision control changes with descriptions related to 'fix' or 'close'" and "ranks more recent bug-fixing changes higher than older ones." A programming element that has been changed compared to the previous version is, by definition, more recently changed than one that has not. Bugspots maps a file's score to all executable statements in that file (Zou, section 3.3.6). Under BRI, "increasing a rule-based possibility score of a programming element that is changed compared to a previous version of the program" reads on the application of Bugspots' recency-based score increase to elements modified in the current version.
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to incorporate Zou's history-based teaching (Zou, section 3.3.6) into Li's SBFL framework on failed tests, because Zou's section 4.3.2 Finding 3.4 establishes that history-based information improves combined fault localization. With respect to claim 6, Li teaches wherein the determining the machine learning-based second ranking comprises: obtaining element features of the programming elements, respectively (Li discloses obtaining element features of the programming elements. Li, section 3.2 "DeepFL Features", sets out four types of element features for each programming element: spectrum-based features (the 34 SBFL formulas outputs per element, Li, section 3.2 under the heading "Spectrum-based Suspiciousness"), mutation-based features (the MBFL formulas outputs per element, Li, section 3.2 under the heading "Mutation-based Suspiciousness", yielding (34+1)×4 = 140 mutation-based suspiciousness values per element), complexity-based fault-proneness features (37 per-element code-complexity and bytecode metrics, Li, section 3.2 under the heading "Complexity-based Fault Proneness"; Li, Table 1 "Studied code metrics"), and textual-similarity features (15 per-element TF-IDF similarity scores between source-code methods and failed tests, Li, section 3.2 under the heading "Textual Similarity Information"). Under BRI, this is "obtaining element features of the programming elements, respectively."). inferring machine learning-based probability scores of the respective programming elements, wherein each machine learning-based probability score represents the probability that a corresponding programming element has a fault and is generated by applying the corresponding element feature to the machine learning model (Li discloses inferring machine-learning-based probability scores by applying the MLP_DFL model to the element features. Li, section 3.1.3 "Tailored MLP Based Neural Network", and Li, Equation (8), disclose the MLP_DFL output formula; Li, section 3.1.1 Equation (2), defines the per-element output ŷ_i = (Pr[y_i = (1,0)], Pr[y_i = (0,1)]) of an MLP using softmax activation, in which the first component represents the probability that the element belongs to the buggy class. Under BRI, the first component of ŷ_i is a "machine learning-based probability score" that "represents the probability that a corresponding programming element has a fault and is generated by applying the corresponding element feature to the machine learning model.") and determining the machine learning-based second ranking by sorting machine learning-based possibility scores (Li discloses determining the machine-learning-based second ranking by sorting the elements according to the ŷ_i^(1) values output by the MLP_DFL model. Li, section 5, presents the DeepFL results in the form of a Top-N ranking sorted by the MLP_DFL probability score. Under BRI, this is "determining the machine learning-based second ranking by sorting machine learning-based possibility scores."). With respect to claim 7, Li teaches wherein each of the element features comprises: [[an indication of a change history of the corresponding programming element, a probability that the corresponding programming element has a fault in a previous version of the program, a call distance between the corresponding programming element and another programming element, an indication of whether the corresponding programming element is executed in the execution of versions of the program]], or a rule-based probability score of the corresponding programming element (The Markush list uses "or," so under BRI the limitation is satisfied if any one of the listed alternative features is taught. Li teaches at least the last alternative — "a rule-based probability score of the corresponding programming element" — because Li expressly uses the 34 SBFL suspiciousness scores per element as element features fed to the MLP_DFL model (Li, section 3.2 under the heading "Spectrum-based Suspiciousness"; Li, section 3.1.3 "Tailored MLP Based Neural Network"). Each SBFL score is a rule-based probability score of the corresponding programming element). With respect to claim 8, Li is silent to disclose, however in an analogous art, Zou teaches wherein the combining comprises: determining a first weight for the rule-based first ranking and a second weight for the machine learning-based second ranking (The concept of these limitations together is the assignment of a respective weight to each of the two rankings, followed by the formation of a third (ensemble) ranking from the two rankings weighted accordingly. Li does not expressly disclose assigning respective weights to the SBFL ranking and the MLP_DFL ranking and combining them into a third ranking; rather, Li feeds the SBFL features into the MLP model, producing a single output ranking. Zou fills this gap. Zou, section 3.3.7 ("Learning to Rank"), associates each programming element with a vector Suspiciousness(e) = ⟨st₁(e), st₂(e), …⟩ where stᵢ(e) is the suspiciousness score reported by technique tᵢ — including SBFL techniques (Zou, secton 3.3.1, Ochiai and DStar) and other techniques (Zou, section 4.6, comparing against state-of-the-art learning-based FL techniques such as MULTRIC, Savant, FLUCCS, and TraPT). The RankSVM model trained over this vector implements a pairwise learning-to-rank algorithm (Zou, section 3.3.7) which, in operation, produces a combined ranking whose effective form is a weighted combination of the contributing techniques' suspiciousness scores. Each technique's contribution to the trained model is therefore characterized by a learned weight: under BRI, a "first weight" for the SBFL technique's ranking and a "second weight" for the learning-based technique's ranking. This limitation is met by Zou's training of the RankSVM model on the suspiciousness vector ⟨st_SBFL(e), st_ML(e), …⟩, which assigns a respective learned weight to each technique's score (Zou, section 3.3.7; see also Zou,section 4.3.2 Finding 3.3, expressly discussing the contribution and effective weight of each family in the trained combined model) and determining an ensemble-based third ranking of the programming elements by combining the rule-based first ranking and the machine learning-based second ranking according to the first and second weights (This limitation is met by Zou's combined output ranking, which is the third ranking produced by the trained RankSVM model over the suspiciousness vectors of all elements (Zou, section 4.3.2, Table 8, "All Families" row). The combined ranking improves Einspect@1 by 200% over the best standalone (Zou, Finding 3.2), confirming it is a distinct, more-effective output than either of the two contributing rankings considered alone).
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to combine Li with Zou by applying Zou's learning-to-rank combining methodology (Zou, section 3.3.7) to Li's separately-produced SBFL rule-based ranking (Li, section 3.2 "Spectrum-based Suspiciousness") and Li's MLP_DFL machine-learning ranking (Li, section 3.1.3 "Tailored MLP Based Neural Network"), so as to obtain a combined ranking whose top element is identified as the fault element. The motivation is supplied by Zou's empirical demonstration that combining fault-localization rankings from different families yields a 200% Top-1 improvement over the best standalone technique (Zou, section 4.3.2 Finding 3.2), together with Zou's express recommendation against using any technique standalone (Zou, section 6.1). With respect to claim 13, Li teaches a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 (Li discloses a computer-implemented embodiment of the disclosed fault-localization method. Li, section 4.1, describes implementing DeepFL using TensorFlow on the experimental hardware platform; such a computer-implemented embodiment necessarily comprises a non-transitory computer-readable storage medium (e.g., the hard disk or other non-volatile storage of the computer) storing the executable instructions, which when executed by the processor of the computer cause the processor to perform the method described in Li). With respect to claim 14, the claim is directed to an electronic device that corresponds to the method recited in claim 1, respectively (see the rejection of claim 1 above; wherein Li also teaches such device. Li implements DeepFL using the TensorFlow framework executing on a Dell machine (Li, § 4.1 Implementation Details)).
With respect to claim 15, the claim is directed to an electronic device that corresponds to the method recited in claim 2, respectively (see the rejection of claim 2 above).
With respect to claim 16, the claim is directed to an electronic device that corresponds to the method recited in claim 6, respectively (see the rejection of claim 6 above).
With respect to claim 17, the claim is directed to an electronic device that corresponds to the method recited in claim 8, respectively (see the rejection of claim 8 above).
Claims 9-10 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Xia Li et al. ("DeepFL: Integrating Multiple Fault Diagnosis Dimensions for Deep Fault Localization", hereinafter Li – IDS 07/24/2024) in view of Daming Zou et al. (“An Empirical Study of Fault Localization Families and Their Combinations”, hereinafter Zou) and further in view of Thomas Hirsch (“A Fault Localization and Debugging Support Framework driven by Bug Tracking Data”, hereinafter Hirsch). With respect to claim 9, Li in view of Zou is silent to disclose, however in an analogous art, Hirsch teaches wherein the second weight depends on a number of versions from a version where the fault newly occurs to a current version among versions of the program (The concept of this limitation is that the ML-based second weight applied to the machine-learning-based second ranking varies with the bug's lifespan — i.e., the number of program versions the bug has persisted from its first appearance to the current version.
Li and Zou together teach the assignment of respective weights to the rule-based and ML-based rankings, but neither teaches that the ML weight is parametrically tied to the bug's lifespan across program versions.
Hirsch fills this gap. Hirsch's broad teaching is that bug-side characteristics (Hirsch, section III RQ2: bug type; Hirsch, section V findings on "time to fix" and "time to reproduce and locate" for different bug categories; Hirsch, section IV: "TTF (Time To Fix) and magnitude of code changes are used to estimate difficulty and cost") drive the weighting of FL techniques when combining them. Lifespan — the number of versions over which a fault has persisted — is itself a temporally-extended bug-side characteristic of the same family as Hirsch's "time to fix" and "time to locate" characteristics. Under Hirsch's framework, lifespan is therefore a permissible "weighting criteria when combining multiple localization methods" (Hirsch, section III RQ2). The specific direction recited in claim 9 — that as lifespan increases, the ML weight increases — is a routine engineering optimization that follows from the well-known principle that machine-learning models are more reliable for fault categories represented by more training data, and longer-lived bugs accumulate more cross-version training-data points than newly occurring bugs.
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to further modify the Li-Zou combined ranking system by parameterizing the ML-based second weight on the bug's lifespan, in light of Hirsch's framework in which bug-side characteristics serve as weighting criteria when combining multiple fault-localization methods (Hirsch, section III RQ2; Hirsch, section V). The specific direction (longer lifespan increases the ML weight) is a routine engineering optimization consistent with the principle that machine-learning models become more reliable as cross-version training data accumulates. With respect to claim 10, Li in view of Zou is silent to disclose, however in an analogous art, Hirsch teaches wherein the determining the first weight and the second weight comprises: determining a fault type of the fault among candidate fault types (Li and Zou together teach the assignment of respective weights to the rule-based and ML-based rankings, but neither expressly maps those weights from a determined fault type drawn from a set of candidate fault types. Hirsch fills this gap. Hirsch, section I and section III, expressly teaches the use of bug type as the basis for weighting multiple FL methods: "Although manual debugging approaches can vary significantly depending on bug type (e.g. memory bugs or semantic bugs), these differences are not reflected in most existing fault localization tools" (Hirsch, Abstract). Hirsch, section III RQ2, states: "Results from RQ2 will provide an evaluation of existing FL techniques, and serve as a weighting criteria when combining multiple localization methods." Hirsch, section IV ("Classification schema") and section V ("Progress"), further disclose that Hirsch introduces and validates a bug classification schema covering at least memory, concurrency, and semantic bug types (Hirsch, section V), and that Hirsch automates the bug-type classification via NLP and machine learning, achieving mean F1 ≈ 74% on bug-type prediction (Hirsch, section V). The limitation is met by Hirsch's automated bug-type classifier, which determines a fault's bug type as one of several candidate types (e.g., memory, concurrency, semantic) (Hirsch, sections IV, V)) and determining a weight pair mapped to the determined fault type to the first weight and the second weight (This limitation is met by Hirsch's framework of using the determined bug type as "weighting criteria when combining multiple localization methods" (Hirsch, section III RQ2). When the FL combination at issue is the two-ranking ensemble of Li and Zou (i.e., the rule-based first ranking and the ML-based second ranking), Hirsch's framework teaches assigning a pair of weights for those two rankings as a function of the determined bug type — that is, a weight pair mapped from the fault type to the first weight and the second weight.
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to further modify the Li-Zou combined ranking system by looking up a fault-type-mapped weight pair, in light of Hirsch's express teaching that bug type serves as weighting criteria when combining multiple fault-localization methods (Hirsch, section III RQ2).
With respect to claim 18, the claim is directed to an electronic device that corresponds to the method recited in claim 9, respectively (see the rejection of claim 9 above).
Claims 11-12 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Xia Li et al. ("DeepFL: Integrating Multiple Fault Diagnosis Dimensions for Deep Fault Localization", hereinafter Li – IDS 07/24/2024) in view of Daming Zou et al. (“An Empirical Study of Fault Localization Families and Their Combinations”, hereinafter Zou) and further in view of Bier et al. (US Pub. No. 2022/0206930, hereinafter Bier). With respect to claim 11, the combination of Li and Bier teaches the determining the rule-based first ranking comprises: determining the rule-based first ranking for the updated version of the program in response to the test case failing to pass (Specifically, Li, section 2.1 under the heading "Spectrum-based Fault Localization", and Li, section 3.2 under the heading "Spectrum-based Suspiciousness", teach the rule-based SBFL ranking computation based on the four counts (ef, ep, nf, np) of failed/passed tests that execute or do not execute each programming element, yielding a ranked list of programming elements by SBFL suspiciousness score. The limitation refines that substantive operation with two qualifying conditions: (i) the rule-based ranking is determined "for the updated version of the program," and (ii) the determination is performed "in response to the test case failing to pass." Both qualifying conditions are taught by Bier. Bier, paragraph [0028], expressly teaches that fault localization is triggered by failed test cases: "the process continues with fault localization, in which the failed test cases are used to … identify portions of the software that are likely to cause the failure." Under BRI, this teaches the qualifier "in response to the test case failing to pass."
Bier, paragraph [0029], expressly teaches that the post-change (updated) version of the program is the subject of the test-execution and subsequent fault-localization steps: "After each mutation, the test suite is run again to see if the mutation has removed the error … The entire process is repeated … until the mutated program can pass all tests." The "updated version of the program" of this limitation reads on the post-change version of the program produced by the other limitations below, which Bier teaches as the subject of the subsequent test execution and FL pass.
the determining the machine learning-based second ranking comprises: determining the machine learning-based second ranking for the updated version of the program in response to at least one test case failing to pass (The substantive operation recited in this limitation — "determining the machine learning-based second ranking" — is taught by Li. Specifically, Li, section 3.1.3 ("Tailored MLP Based Neural Network"), and Li, Equation (8), teach the MLP_DFL machine-learning model that, when applied to the element features set out at Li, section 3.2 ("DeepFL Features"), produces a per-element probability of being faulty (Li, section 3.1.1, Equation (2), defining the per-element output of an MLP as a probability vector for the buggy and correct classes). This limitation refines that substantive operation with the same two qualifying conditions recited above: (i) the ML-based ranking is determined "for the updated version of the program," and (ii) the determination is performed "in response to at least one test case failing to pass." Both qualifying conditions are taught by Bier for the same reasons articulated for limitation above. Under BRI, "at least one test case failing to pass" reads on Bier's paragraph [0028] teaching that "one or more failed test cases" trigger the fault-localization step.) changing the source code and updating a version of the program in response to receiving a change request for the source code (Bier expressly teaches changing the source code and updating a version of the program in response to a change request. Bier, paragraph [0006], recites: "mutating the first set of components under repair for each execution of the fast-result test." Bier, paragraph [0007], defines "mutating" to include "adding a line of code, deleting a line of code, and modifying a line of code." Under BRI, each such mutation operation is a "change[] [to] the source code" that "update[s] a version of the program." The "change request for the source code" in this limitation reads on the automated mutation request generated by Bier's APR system at each iteration of the repair loop (Bier, paragraph [0029]: "The entire process is repeated … until the mutated program can pass all tests") and executing the program based on the test case after changing the source code (Bier expressly teaches re-executing the test on the program after each source-code change. Bier, paragraph [0029]: "After each mutation, the test suite is run again to see if the mutation has removed the error." This is "executing the program based on the test case after changing the source code").
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to operate the Li-Zou combined fault-localization framework within Bier's iterative pipeline of code mutation (Bier, paragraphs [0006], [0007]), test execution after each change (Bier, paragraph [0029]), and fault localization triggered by failed test cases (Bier, paragraph [0028]). Bier itself relies on SBFL as its fault-localization step (Bier, paragraph [0051]), so substituting Li-Zou's more accurate combined SBFL-and-ML-ranking output for Bier's SBFL-only fault-localization step is the routine application of an improved tool to a known pipeline ready for such improvement. With respect to claim 12, the combination of Li in view of Zou and Bier teaches determining the rule-based first ranking, the machine learning-based second ranking, and the fault element in response to receiving an error report for an error occurring in the program from an external device using the program (The substantive operations of "determining the rule-based first ranking," "determining the machine learning-based second ranking," and "determining the fault element" are taught by Li. Specifically, Li, section 2.1 under the heading "Spectrum-based Fault Localization", and Li, section 3.2 under the heading "Spectrum-based Suspiciousness", teach determining the rule-based first ranking by SBFL computation; Li, section 3.1.3 ("Tailored MLP Based Neural Network") and Li, Equation (8), teach determining the machine-learning-based second ranking by MLP_DFL inference over the element features set out at Li, section 3.2; and the combined ranking from Zou, section 3.3.7 ("Learning to Rank"), per the analysis of claim 1(d), yields the fault element as the top-ranked output. The qualifying triggering condition — that the three substantive determinations are performed "in response to receiving an error report for an error occurring in the program from an external device using the program" — is taught by Bier. Bier, paragraph [0028], expressly teaches that the fault-localization step is triggered by the report of one or more failed test cases: "The process typically starts with the execution of a test suite … One or more failed test cases can indicate errors … The process continues with fault localization, in which the failed test cases are used to … identify portions of the software that are likely to cause the failure." Under BRI, the device executing the test suite is an "external device using the program," and its report of failed test cases is an "error report for an error occurring in the program from [the] external device." Bier therefore teaches the qualifying triggering condition. outputting information on the determined fault element (Li teaches outputting information on the determined fault element. Li, section 5 ("RESULT ANALYSIS"), presents the DeepFL output as a ranked list of suspicious programming elements, reporting Top-1, Top-3, Top-5, and Top-10 fault-localization results — i.e., information identifying the determined fault element (the top-ranked element or top-N elements) is output for downstream evaluation). changing the source code and updating a version of the program based thereon (Bier expressly teaches changing the source code and updating a version of the program. Bier, paragraph [0006], recites: "mutating the first set of components under repair for each execution of the fast-result test." Bier, paragraph [0007], defines "mutating" to include "adding a line of code, deleting a line of code, and modifying a line of code." Under BRI, each such mutation operation is a "change[] [to] the source code" that "update[s] a version of the program."). verifying the updated version of the program (Bier expressly teaches verifying the updated version of the program. Bier, paragraph [0006], recites a two-stage verification: "in response to a mutation resulting in a successful execution of the fast-result test, re-running the original test." Re-running the original (slower, more comprehensive) test on the updated (mutated) program confirms that the change actually fixes the fault and does not introduce regressions) and distributing information on the updated version of the program to the external device, based on success in verifying the updated version of the program (Bier expressly teaches distributing information on the verified updated version of the program upon successful verification. Bier, paragraph [0006]: "in response to the mutation resulting in a successful execution of the original test, recommending the mutation as a correct repair." The recommended mutation is information on the updated version of the program, and the recommendation is conditioned on success in verifying the updated version (i.e., on the original test passing on the mutated version). Under BRI, the downstream recipient of the recommendation — which, per the present rejection, is the same "external device using the program").
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to operate the Li-Zou combined fault-localization framework within Bier's iterative pipeline of error detection by failed-test reports (Bier, paragraph [0028]), code mutation (Bier, paragraphs [0006], [0007]), verification by re-running original tests (Bier, paragraph [0006]), and distribution of the verified repair recommendation (Bier, paragraph [0006]). Bier itself relies on SBFL for its fault-localization step (Bier, paragraph [0051]), so substituting Li-Zou's more accurate combined SBFL-and-ML-ranking output for Bier's SBFL-only fault-localization step is the routine application of an improved tool to a known pipeline ready for such improvement.
With respect to claim 19, the claim is directed to an electronic device that corresponds to the method recited in claim 11, respectively (see the rejection of claim 11 above).
With respect to claim 20, the claim is directed to an electronic device that corresponds to the method recited in claim 12, respectively (see the rejection of claim 12 above).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Papadopoulos et al. (US Pub. No. 2025/0013555) discloses a method for prediction of test failures. The method may include retrieving, by a test failure prediction computer program, test results for a test; training, by the test failure prediction computer program, a machine learning engine based on one or more hypotheses; receiving, by the test failure prediction computer program, metrics for a code file; providing, by the test failure prediction computer program, the metrics for the code file to the trained machine learning engine; receiving, by the test failure prediction computer program, an output from the trained machine learning engine that provides a probability of failure; and outputting, by the test failure prediction computer program, the output. (see abstract).
Boue et al. (US Pub. No. 2024/0303185) discloses a computing system encodes a next graph based on modified source code files recorded by the next code commit event. The computing system inputs the next graph to a graph machine learning model, the graph machine learning model being trained by graphs representing modified source code files and software test results corresponding to multiple code commit events occurring prior to the next code commit event in the sequence of code commit events. The computing system determines an order of test cases of the next code commit event using the graph machine learning model in an inference mode. The computing system executes the test cases according to the order during the software development build process corresponding to the next code commit event. (see abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANIBAL RIVERACRUZ whose telephone number is (571)270-1200. The examiner can normally be reached Monday-Friday 9:30 AM-6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hyung S Sough can be reached at 5712726799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANIBAL RIVERACRUZ/Primary Examiner, Art Unit 2192
Prosecution Insights