Prosecution Insights
Last updated: November 04, 2026
Application No. 18/738,409

SYSTEM AND METHOD FOR SOFTWARE TEST ANALYSIS

Final Rejection §101§103
Filed
Jun 10, 2024
Priority
Jun 15, 2023 — provisional 63/508,403
Examiner
MALIK, ZEERICK ASIM
Art Unit
2193
Tech Center
2100 — Computer Architecture & Software
Assignee
Sealights Technologies Ltd.
OA Round
2 (Final)
0%
Grant Probability
At Risk
3-4
OA Rounds
8m
Est. Remaining
0%
With Interview

Examiner Intelligence

Grants only 0% of cases
0%
Career Allowance Rate
0 granted / 2 resolved
-55.0% vs TC avg
Minimal +0% lift
Without
With
+0.0%
Interview Lift
resolved cases with interview
Typical timeline
3y 1m
Avg Prosecution
15 currently pending
Career history
28
Total Applications
across all art units

Statute-Specific Performance

§101
30.2%
-9.8% vs TC avg
§103
43.0%
+3.0% vs TC avg
§102
5.8%
-34.2% vs TC avg
§112
19.8%
-20.2% vs TC avg
Black line = Tech Center average estimate • Based on career data from 2 resolved cases

Office Action

§101 §103
DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claim Rejections - 35 USC § 101 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claim 1-20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Claim(s) 1 and 14 recite(s): a test listener computational device for receiving a first test result of the first test; a backend computational device for analyzing a behavior of the code during the test according to an analysis of a plurality of functions, and a computer network for connecting said test listener computational device to said backend computational device; selecting a second test result of a second test; comparing said first test result of said first test to said second test result of said second test; analyzing a plurality of functions of the code during each of said first and second tests, wherein said analyzing the plurality of functions of the code during each of said first and second tests comprises: obtaining, via a data collector, test trace results tracing execution of the code during each of said first and second tests, wherein the test trace results comprise trace structures for said first and second tests that record sequences of operations performed for said plurality of functions, grouping said plurality of functions into a plurality of spans in said test trace results, determining a structural difference between said trace structures of said first and second tests based on said plurality of spans, and applying a data differential to the plurality of spans with the structural difference to identify one or more suspected causes exhibiting differential behavior; and determining a root cause for a difference in said test results according to said plurality of functions based on the one or more suspected causes. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 1 is a system Yes. Claim 14 is a method Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation of "analyzing", as drafted in #2 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "backend computational device", nothing in the claim element precludes the step from being performed by a person. The limitation of "selecting", as drafted in #4 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting “a processor", nothing in the claim element precludes the step from being performed by a person. The limitation of "comparing", as drafted in #5 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting “a processor", nothing in the claim element precludes the step from being performed by a person. The limitation of "analyzing", as drafted in #6 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting “a processor", nothing in the claim element precludes the step from being performed by a person. The limitation of "grouping", as drafted in #8 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting “a processor", nothing in the claim element precludes the step from being performed by a person. The limitation of "determining", as drafted in #9 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting “a processor", nothing in the claim element precludes the step from being performed by a person. The limitation of "identifying", as drafted in #10 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting “a processor", nothing in the claim element precludes the step from being performed by a person. The limitation of "determining", as drafted in #11 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting “a processor", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. The "receiving" limitations in #1 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "receiving a first test result" in the context of this claim encompasses merely data transmission. See in the MPEP §§ 2106.05(g). The "connecting" limitations in #3 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "connecting" in the context of this claim encompasses mere data transmission. See in the MPEP §§ 2106.05(g). The "obtaining" limitations in #7 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "obtaining, via a data collector" in the context of this claim encompasses mere data transmission. See in the MPEP §§ 2106.05(g). Additionally, the claims recite the following additional element: A system, a test listener computational device, a backend computational device, a computer network, a processor, a memory The element that is recited in the claims are stated at a high level of generality (i.e. as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using generic computer component. See the MPEP §§ 2106.05(f). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limitation on practicing the abstract idea(s). Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Additionally, with regards to #1, 3, and 7 above, per MPEP 2106.05(d)(ll), the courts have recognized the following computer function(s) as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity: Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); Claim(s) 2 and 15 recite(s): wherein said first and second tests are different runs of the same test; wherein behavior of the same test varies over multiple runs, such that said root cause is determined to be flakiness of the same test Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 2 is a system Yes. Claim 15 is a method Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation of "root cause is determined”, as drafted in #13 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "the system", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. The "first and second tests" limitation in #12 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "are different runs of the same test" in the context of this claim encompasses merely running a program twice. See in the MPEP §§2106.05(f). Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 3 and 16 recite(s): wherein said first and second tests are different tests, wherein at least one of said first and second tests exhibits different behavior, such that said backend computational device analyzes said plurality of functions to determine a reason for said different behavior. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 3 is a system Yes. Claim 16 is a method Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation of "analyzes", as drafted in #14 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "backend computational device", nothing in the claim element precludes the step from performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. Additionally, the claims recite the following additional element: backend computational device, the system The element that is recited in the claims are stated at a high level of generality (i.e. as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using generic computer component. See the MPEP §§ 2106.05(f). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limitation on practicing the abstract idea(s). Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 4 and 17 recite(s): wherein the data collector comprises an open telemetry data collector, and wherein said backend computational device analyzes said plurality of functions according to an open telemetry using said open telemetry data collector. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 4 is a system Yes. Claim 17 is a method Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation of "analyzes", as drafted in #16 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "backend computational device", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. The "comprises" limitation in #15 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "comprises an open telemetry collector" in the context of this claim encompasses merely having extra functionality within a program. See in the MPEP §§2106.05(f). Additionally, the claims recite the following additional element: backend computational device The element that is recited in the claims are stated at a high level of generality (i.e. as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using generic computer component. See the MPEP §§ 2106.05(f). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limitation on practicing the abstract idea(s). Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 5 and 18 recite(s): wherein said backend computational device analyzes said plurality of functions in the plurality of spans according to said test results. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 5 is a system Yes. Claim 18 is a method Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation of "analyzes", as drafted in #17 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting backend computational device, nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. Additionally, the claims recite the following additional element: backend computational device The element that is recited in the claims are stated at a high level of generality (i.e. as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using generic computer component. See the MPEP §§ 2106.05(f). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limitation on practicing the abstract idea(s). Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 6 recite(s): wherein said different behavior comprises failure of at least one test. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 6 is a Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation, as drafted in #18 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "the system", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 7 and 19 recite(s): wherein the backend computational device is further configured to utilize a build mapper to correlate test results with code changes. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 7 is a system Yes. Claim 19 is a method Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation, as drafted in #19 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "backend computational device", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. Additionally, the claims recite the following additional element: backend computational device The element that is recited in the claims are stated at a high level of generality (i.e. as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using generic computer component. See the MPEP §§ 2106.05(f). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limitation on practicing the abstract idea(s). Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 8 recite(s): wherein the build mapper is configured to determine relevance of tests based on code coverage and changes in the code. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 8 is a system Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation, as drafted in #20 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a build mapper", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 9 recite(s): wherein the build mapper includes a machine learning analyzer configured to apply a machine learning model to determine the relevance of tests to specific code sections; wherein the machine learning analyzer is further configured to receive information from a history analyzer and a statistical analyzer to inform the machine learning model. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 9 is a system Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation of "determine", as drafted in #21 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a build mapper", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. The "receive" limitations in #22 above, as claimed and under BRI, is an additional element that is insignificant extra-solution activity. For example, "receive" in the context of this claim encompasses mere data transmission. See in the MPEP §§ 2106.05(g). Additionally, the claims recite the following additional element: statistical analyzer, history analyzer The element that is recited in the claims are stated at a high level of generality (i.e. as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using generic computer component. See the MPEP §§ 2106.05(f). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limitation on practicing the abstract idea(s). Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Additionally, with regards to #22 above, per MPEP 2106.05(d)(ll), the courts have recognized the following computer function(s) as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity: Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); Claim(s) 10 recite(s): wherein the backend computational device is further configured to perform test impact analysis to recommend tests for execution based on a policy set by a user. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 10 is a system Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation, as drafted in #23 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "a backend computational device", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 11 recite(s): wherein the policy includes criteria based on test execution frequency, time frames, and code usage in production. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 11 is a system Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. The limitation in #24 above. As claimed and under BRI, is an additional element that is mere instructions to apply an exception. For example, "policy includes criteria" in the context of this claim encompasses merely stating what information is stored in the data. See in the MPEP §§2106.05(f). Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 12 recite(s): wherein the backend computational device is further configured to utilize an analysis engine to analyze test results and determine underlying behavior of the code under test. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 12 is a system Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation, as drafted in #25 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "backend computational device", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. Additionally, the claims recite the following additional element: backend computational device The element that is recited in the claims are stated at a high level of generality (i.e. as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using generic computer component. See the MPEP §§ 2106.05(f). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limitation on practicing the abstract idea(s). Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 13 recite(s): wherein the backend computational device is further configured to compare test results to a threshold and apply span rules to determine a suspect score for test behavior. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 13 is a system Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation, as drafted in #26 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "backend computational device", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim(s) 20 recite(s): wherein the build mapper determines relevance of tests based on code coverage and changes in the code, and the method further comprises applying a machine learning model to determine the relevance of tests to specific code sections. Step 1: are the claims to a process, machine, manufacture, or a composition of matter? Yes. Claim 20 is a method Step 2A, Prong I; Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes: (an) abstract idea(s). The limitation of "determines relevance", as drafted in #27 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "build mapper", nothing in the claim element precludes the step from being performed by a person. The limitation of "determines the relevance", as drafted in #28 above, under its broadest reasonable interpretation, covers performance of the mind, but for generic computer parts. That is, other than reciting "machine learning model", nothing in the claim element precludes the step from being performed by a person. Step 2A Prong II: Does the claim recite additional elements that integrate the judicial exception into a practical application? No. Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because mere instructions to apply an exception using generic computer components cannot provide the inventive step. Claim Rejections - 35 USC § 103 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claim(s) 1-3, 12, and 14-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20180300229 A1 (Hereinafter referred to as Kornfeld) and US 9311220 B1 (Hereinafter referred to as Ha). Regarding claim 1, Kornfeld teaches: A system for determining a root cause of a behavior of code during a first test, the system comprising a test listener computational device for receiving a first test result of the first test (Para. [23], Kornfeld shows " during these original executions, a testing application (e.g., non-deterministic test instructions 125) may function to record information related to the original executions, such as whether the test has passed or failed, a time of the last modification of the code of the AUT, a time of the last modification of the code of the non-deterministic test, and/or any other suitable information. The information may be stored in a storage device (e.g., data repository 140) and/or otherwise transmitted to another component (e.g., root cause analysis device 210) for analysis."); a backend computational device for analyzing a behavior of the code during the test according to an analysis of a plurality of functions (Para. [27], Kornfeld shows "root cause analysis device 244 may determine code of the AUT that is being tested by the non-deterministic test and based on the determination identify instrumentation points within the code where instrumentation code may be inserted. The code may be inserted into the instrumentation points and may be used to analyze the systems associated with the AUT and the test as the AUT and the test execute. In some examples, the instrumentation of the code may be used to determine a flow of the AUT (e.g., by tracking the method call chain and saving an image of the execution). For example, data capture instructions 244 may cause processor 220 and/or another processor to determine when an object is created on a thread by instrumenting a constructor of an object. For example, the AUT may include constructors (e.g., blocks of code that are executed to create an object) that may determine the actions taken when an object is created (e.g., the functions performed by the object)"), and a computer network for connecting said test listener computational device to said backend computational device (Fig. 1, Para. [12], Kornfeld shows "system 100 may include a root cause analysis device 110, a client device 120, an AUT device 130, a data repository 140, and a network 150 for connecting root cause analysis device 110 with client device 120, AUT device 130, and/or data repository 140."); wherein said backend computational device comprises a memory storing a plurality of instructions and a processor for executing said instructions for: selecting a second test result of a second test (Para. [13], Kornfeld shows "root cause analysis device 110 may capture data during rerun executions of a non-deterministic test and the analyze captured data to determine a root cause of a failure of the non-deterministic test"); comparing said first test result of said first test to said second test result of said second test (Para. [28], Kornfeld shows " root cause analysis instructions 246 may function to analyze the captured data to determine a root cause of a failure of the non-deterministic test based on a comparison between failed rerun executions (i.e., those of the rerun executions that failed) and passed rerun executions (i.e., those of the rerun executions that passed)"); analyzing a plurality of functions of the code during each of said first and second tests (Para. [13], Kornfeld shows "root cause analysis device 110 may capture data during rerun executions of a non-deterministic test and the analyze captured data to determine a root cause of a failure of the non-deterministic test" Para. [23], Kornfeld shows "During these original executions, a testing application (e.g., non-deterministic test instructions 125) may function to record information related to the original executions, such as whether the test has passed or failed, a time of the last modification of the code of the AUT, a time of the last modification of the code of the non-deterministic test, and/or any other suitable information. The information may be stored in a storage device (e.g., data repository 140) and/or otherwise transmitted to another component (e.g., root cause analysis device 210) for analysis." Para. [25], "the captured data may include application data, test data, and environment data. The application data may include data related to the AUT (e.g., execution flows of the AUT, such as execution flows performed during the rerun executions, AUT version, AUT code creator, etc.), the test data may include data related to the rerun executions of the test (e.g., tester name, test date/time, test status data (e,g., whether the test passed or failed), testing actions and/or steps performed during a particular test execution state, etc.)"), wherein said analyzing the plurality of functions of the code during each of said first and second tests comprises: obtaining, via a data collector, test trace results tracing execution of the code during each of said first and second tests, wherein the test trace results comprise trace structures for said first and second tests that record sequences of operations performed for said plurality of functions (Para. [27], Kornfeld shows “For example, root cause analysis device 244 may determine code of the AUT that is being tested by the non-deterministic test and based on the determination identify instrumentation points within the code where instrumentation code may be inserted. The code may be inserted into the instrumentation points and may be used to analyze the systems associated with the AUT and the test as the AUT and the test execute. In some examples, the instrumentation of the code may be used to determine a flow of the AUT (e.g., by tracking the method call chain and saving an image of the execution).” Examiner notes the above citation shows taking an image and method call chain for the tested application. This can be considered a type of test trace.), and determining a root cause for a difference in said test results according to said plurality of functions based on the one or more suspected causes (Para. [28], Kornfeld shows "Root cause analysis instructions 246 may function to analyze the captured data to determine a root cause of a failure of the non-deterministic test based on a comparison between failed rerun executions (i.e., those of the rerun executions that failed) and passed rerun executions (i.e., those of the rerun executions that passed). For example, when root cause analysis instructions 246 are executed by processor 220, root cause analysis instructions 246 may cause processor 220 to analyze the captured data by identifying patterns in the captured data such, as for example, by identifying patterns in the captured data that are common among failed rerun executions of the non-deterministic test and not common among passed rerun executions of the non-deterministic test. In some implementations, the patterns in the captured data may be identified based on an examination of attributes (characteristics) of the rerun executions of the non-deterministic test. For example, the attributes of the failed rerun executions may be compared to the attributes of the passed rerun executions and, based on the comparison, a determination may be made as to whether a difference between the attributes (e.g., a difference between the attributes of the failed rerun executions and the attributes of the passed rerun executions) is statistically significant. Any suitable attribute may be compared, including attributes based on execution flow, thread metrics, the environment of the AUT, the environment of the rerun executions of the non-deterministic test, and the like. If the difference between the attributes is statistically significant (e.g., an analysis of the data results in a particular p-value, the analysis of the data indicates that the results pass a particular threshold, etc.), then the difference may be identified as a pattern. Conversely, if the difference is not statistically significant, the difference may not be identified as a pattern. In some examples, the identified patterns may be considered to be potential root causes of the failure of the non-deterministic test"). Kornfeld does not disclose: grouping said plurality of functions into a plurality of spans in said test trace results, determining a structural difference between said trace structures of said first and second tests based on said plurality of spans, and applying a data differential to the plurality of spans with the structural difference to identify one or more suspected causes exhibiting differential behavior; However, in the analogous art of root-cause of non-deterministic tests, Ha teaches: grouping said plurality of functions into a plurality of spans in said test trace results (Fig. 4a, Fig. 5a, Col. [3] lines [32-47], Ha shows "Each subtree partially represents some call relations demonstrated in the overall trace. By comparing the subtree sets derived from PASS traces and those from FAIL traces, an exemplary system determines the subtree patterns unique to each set (113). FIGS. 4a and 4b illustrate subtrees of the successful execution shown in FIG. 3a. FIGS. 5a-5c illustrate subtrees of the failing execution shown in FIG. 3b. By comparing the two sets, we find differences in the methods called by method B. The exemplary system may then report patterns that only happen in PASS traces and subpatterns that only happen in FAIL traces. In some embodiments, the exemplary system may rank the diverging subtree patterns between the two sets based on the frequency of each pattern. The higher a pattern is ranked, the more likely it is to be a root cause of the non-deterministic test." Col. [1] lines [49-59], Ha shows "running the test, enabling call trace collection to record call traces from the test until at least one execution has succeeded and one execution has failed; for each trace in the call trace collection, determining whether the trace was part of a passing test or a failing test and labeling the trace according to the determination; constructing a dynamic call tree for the passing traces; constructing a dynamic call tree for the failing traces; comparing the dynamic call tree for the passing traces with the dynamic call tree for the failing traces to determine the subtree patterns that only occur in one tree or the other" The citation above shows grouping function calls into subtrees and each individual leaf node is a call. The tree is split into smaller subtrees that are analyzed and can be considered grouping or clustering the calls for analysis. Spans as defined in the specification are functional groups in relation to the code), determining a structural difference between said trace structures of said first and second tests based on said plurality of spans (Fig. 4a, Fig. 5a, Col. [3] lines [32-47], Ha shows "Each subtree partially represents some call relations demonstrated in the overall trace. By comparing the subtree sets derived from PASS traces and those from FAIL traces, an exemplary system determines the subtree patterns unique to each set (113). FIGS. 4a and 4b illustrate subtrees of the successful execution shown in FIG. 3a. FIGS. 5a-5c illustrate subtrees of the failing execution shown in FIG. 3b. By comparing the two sets, we find differences in the methods called by method B. The exemplary system may then report patterns that only happen in PASS traces and subpatterns that only happen in FAIL traces. In some embodiments, the exemplary system may rank the diverging subtree patterns between the two sets based on the frequency of each pattern. The higher a pattern is ranked, the more likely it is to be a root cause of the non-deterministic test." Col. [1] lines [49-59], Ha shows "running the test, enabling call trace collection to record call traces from the test until at least one execution has succeeded and one execution has failed; for each trace in the call trace collection, determining whether the trace was part of a passing test or a failing test and labeling the trace according to the determination; constructing a dynamic call tree for the passing traces; constructing a dynamic call tree for the failing traces; comparing the dynamic call tree for the passing traces with the dynamic call tree for the failing traces to determine the subtree patterns that only occur in one tree or the other" The citation above shows grouping function calls into subtrees and each individual leaf node is a call. The tree is split into smaller subtrees that are analyzed and can be considered grouping or clustering the calls for analysis. Spans as defined in the specification are functional groups in relation to the code. Examiner notes the above citation also shows examining patterns defining differences (structural differences) between failed and passed tests), and applying a data differential to the plurality of spans with the structural difference to identify one or more suspected causes exhibiting differential behavior (Fig. 4a, Fig. 5a, Col. [3] lines [32-47], Ha shows "Each subtree partially represents some call relations demonstrated in the overall trace. By comparing the subtree sets derived from PASS traces and those from FAIL traces, an exemplary system determines the subtree patterns unique to each set (113). FIGS. 4a and 4b illustrate subtrees of the successful execution shown in FIG. 3a. FIGS. 5a-5c illustrate subtrees of the failing execution shown in FIG. 3b. By comparing the two sets, we find differences in the methods called by method B. The exemplary system may then report patterns that only happen in PASS traces and subpatterns that only happen in FAIL traces. In some embodiments, the exemplary system may rank the diverging subtree patterns between the two sets based on the frequency of each pattern. The higher a pattern is ranked, the more likely it is to be a root cause of the non-deterministic test." Col. [1] lines [49-59], Ha shows "running the test, enabling call trace collection to record call traces from the test until at least one execution has succeeded and one execution has failed; for each trace in the call trace collection, determining whether the trace was part of a passing test or a failing test and labeling the trace according to the determination; constructing a dynamic call tree for the passing traces; constructing a dynamic call tree for the failing traces; comparing the dynamic call tree for the passing traces with the dynamic call tree for the failing traces to determine the subtree patterns that only occur in one tree or the other" The citation above shows grouping function calls into subtrees and each individual leaf node is a call. The tree is split into smaller subtrees that are analyzed and can be considered grouping or clustering the calls for analysis. Spans as defined in the specification are functional groups in relation to the code. Examiner notes the above citation shows analyzing patterns (structural differences) and analyzing the patterns and ranking them based on frequency (data differential) to identify root causes for failure); Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Ha into the teachings of Kornfeld to implement “grouping said plurality of functions into a plurality of spans in said test trace results, determining a structural difference between said trace structures of said first and second tests based on said plurality of spans, and applying a data differential to the plurality of spans with the structural difference to identify one or more suspected causes exhibiting differential behavior”. The modification would have been obvious as one of ordinary skill in the art would be motivated to determine the root cause of the software test failures (Ha, [Abstract]).  Regarding claim 2, Kornfeld as modified in claim 1 teaches: wherein said first and second tests are different runs of the same test; wherein behavior of the same test varies over multiple runs, such that said root cause is determined to be flakiness of the same test (Para. [28], Kornfeld shows "Root cause analysis instructions 246 may function to analyze the captured data to determine a root cause of a failure of the non-deterministic test based on a comparison between failed rerun executions (i.e., those of the rerun executions that failed) and passed rerun executions (i.e., those of the rerun executions that passed). For example, when root cause analysis instructions 246 are executed by processor 220, root cause analysis instructions 246 may cause processor 220 to analyze the captured data by identifying patterns in the captured data such, as for example, by identifying patterns in the captured data that are common among failed rerun executions of the non-deterministic test and not common among passed rerun executions of the non-deterministic test. In some implementations, the patterns in the captured data may be identified based on an examination of attributes (characteristics) of the rerun executions of the non-deterministic test. For example, the attributes of the failed rerun executions may be compared to the attributes of the passed rerun executions and, based on the comparison, a determination may be made as to whether a difference between the attributes (e.g., a difference between the attributes of the failed rerun executions and the attributes of the passed rerun executions) is statistically significant. Any suitable attribute may be compared, including attributes based on execution flow, thread metrics, the environment of the AUT, the environment of the rerun executions of the non-deterministic test, and the like. If the difference between the attributes is statistically significant (e.g., an analysis of the data results in a particular p-value, the analysis of the data indicates that the results pass a particular threshold, etc.), then the difference may be identified as a pattern. Conversely, if the difference is not statistically significant, the difference may not be identified as a pattern. In some examples, the identified patterns may be considered to be potential root causes of the failure of the non-deterministic test"). Regarding claim 3, Kornfeld as modified in claim 1 teaches: wherein said first and second tests are different tests, wherein at least one of said first and second tests exhibits different behavior, such that said backend computational device analyzes said plurality of functions to determine a reason for said different behavior (Para. [14], Kornfeld shows " data related to action steps performed during application testing, environment data related to the environment of the testing application, and/or any other suitable data. For example, client device 120 may be a computing system operated by a tester of an AUT and thus client device 120 may be adapted to transmit test data to root cause analysis device 110" Para. [28], Kornfeld shows "Root cause analysis instructions 246 may function to analyze the captured data to determine a root cause of a failure of the non-deterministic test based on a comparison between failed rerun executions (i.e., those of the rerun executions that failed) and passed rerun executions (i.e., those of the rerun executions that passed). For example, when root cause analysis instructions 246 are executed by processor 220, root cause analysis instructions 246 may cause processor 220 to analyze the captured data by identifying patterns in the captured data such, as for example, by identifying patterns in the captured data that are common among failed rerun executions of the non-deterministic test and not common among passed rerun executions of the non-deterministic test. In some implementations, the patterns in the captured data may be identified based on an examination of attributes (characteristics) of the rerun executions of the non-deterministic test. For example, the attributes of the failed rerun executions may be compared to the attributes of the passed rerun executions and, based on the comparison, a determination may be made as to whether a difference between the attributes (e.g., a difference between the attributes of the failed rerun executions and the attributes of the passed rerun executions) is statistically significant. Any suitable attribute may be compared, including attributes based on execution flow, thread metrics, the environment of the AUT, the environment of the rerun executions of the non-deterministic test, and the like. If the difference between the attributes is statistically significant (e.g., an analysis of the data results in a particular p-value, the analysis of the data indicates that the results pass a particular threshold, etc.), then the difference may be identified as a pattern. Conversely, if the difference is not statistically significant, the difference may not be identified as a pattern. In some examples, the identified patterns may be considered to be potential root causes of the failure of the non-deterministic test"). Regarding claim 12, Kornfeld as modified in claim 1 teaches: wherein the backend computational device is further configured to utilize an analysis engine to analyze test results and determine underlying behavior of the code under test (Fig. 3, Para. [38], Kornfeld shows "determining, in response to receiving the second request, a root cause of a failure of the non-deterministic test by analyzing the captured data to identify patterns in the captured data that are common among failed rerun executions of the non-deterministic test and not common among passed rerun executions of the non-deterministic test (step S440). For example, the determination may be performed via root cause analysis engine 340"). With regards to claim 14, it is a method claim having similar limitations as cited in claim 1 above. Thus, claim 14 is also rejected under the same rationale as cited in the rejection of claim 1 above. Regarding claim 15, Kornfeld as modified in claim 14 teaches: wherein the software test and the second software test are different executions of the same test, and the method further comprises determining flakiness of the software test based on variability in behavior over multiple test executions (Para. [28], Kornfeld shows "Root cause analysis instructions 246 may function to analyze the captured data to determine a root cause of a failure of the non-deterministic test based on a comparison between failed rerun executions (i.e., those of the rerun executions that failed) and passed rerun executions (i.e., those of the rerun executions that passed). For example, when root cause analysis instructions 246 are executed by processor 220, root cause analysis instructions 246 may cause processor 220 to analyze the captured data by identifying patterns in the captured data such, as for example, by identifying patterns in the captured data that are common among failed rerun executions of the non-deterministic test and not common among passed rerun executions of the non-deterministic test. In some implementations, the patterns in the captured data may be identified based on an examination of attributes (characteristics) of the rerun executions of the non-deterministic test. For example, the attributes of the failed rerun executions may be compared to the attributes of the passed rerun executions and, based on the comparison, a determination may be made as to whether a difference between the attributes (e.g., a difference between the attributes of the failed rerun executions and the attributes of the passed rerun executions) is statistically significant. Any suitable attribute may be compared, including attributes based on execution flow, thread metrics, the environment of the AUT, the environment of the rerun executions of the non-deterministic test, and the like. If the difference between the attributes is statistically significant (e.g., an analysis of the data results in a particular p-value, the analysis of the data indicates that the results pass a particular threshold, etc.), then the difference may be identified as a pattern. Conversely, if the difference is not statistically significant, the difference may not be identified as a pattern. In some examples, the identified patterns may be considered to be potential root causes of the failure of the non-deterministic test"). With regards to claim 16, Kornfeld teaches claim 14 as cited above and it is a method claim having similar limitations as cited in claim 3 above. Thus, claim 16 is also rejected under the same rationale as cited in the rejection of claim 6 above. Claim(s) 4-6 and 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20180300229 A1 (Hereinafter referred to as Kornfeld) and US 9311220 B1 (Hereinafter referred to as Ha) in further view of US 20220147542 A1 (Hereinafter referred to as Asgar). Regarding claim 4, Kornfeld teaches claim 3 as cited above and:  wherein said backend computational device analyzes said plurality of functions (Para. [27], Kornfeld shows "root cause analysis device 244 may determine code of the AUT that is being tested by the non-deterministic test and based on the determination identify instrumentation points within the code where instrumentation code may be inserted. The code may be inserted into the instrumentation points and may be used to analyze the systems associated with the AUT and the test as the AUT and the test execute. In some examples, the instrumentation of the code may be used to determine a flow of the AUT (e.g., by tracking the method call chain and saving an image of the execution). For example, data capture instructions 244 may cause processor 220 and/or another processor to determine when an object is created on a thread by instrumenting a constructor of an object. For example, the AUT may include constructors (e.g., blocks of code that are executed to create an object) that may determine the actions taken when an object is created (e.g., the functions performed by the object)") Kornfeld does not disclose: wherein the data collector comprises an open telemetry data collector,    according to an open telemetry using said open telemetry data collector. However, in the analogous art of event logging and debugging, Asgar teaches:  wherein the data collector comprises an open telemetry data collector (Para. [44], Asgar shows "it is beneficial for developers or other users to monitor the functionality or performance of distributed applications. For example, gaining visibility into HTTP traffic can be valuable when working with distributed applications. This data can be used for performance, functional and security monitoring. Many applications accomplish this by utilizing middleware to add tracing or logging to requests (e.g., HTTP requests) in the application. In some cases, open source frameworks (e.g., Open Telemetry)" Examiner notes the above citation shows collecting data with use of middleware. The middleware can be considered an open telemetry collector),    according to an open telemetry using said open telemetry data collector (Para. [44], Asgar shows "it is beneficial for developers or other users to monitor the functionality or performance of distributed applications. For example, gaining visibility into HTTP traffic can be valuable when working with distributed applications. This data can be used for performance, functional and security monitoring. Many applications accomplish this by utilizing middleware to add tracing or logging to requests (e.g., HTTP requests) in the application. In some cases, open source frameworks (e.g., Open Telemetry)" Examiner notes the above citation shows collecting data with use of middleware. The middleware can be considered an open telemetry collector) . Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Asgar into the teachings of Kornfeld as modified to implement “wherein said backend computational device analyzes said plurality of functions according to open telemetry”. The modification would have been obvious as one of ordinary skill in the art would be motivated as it benefits developers to monitor functionality and performance of applications (Asgar, Para. [44]).  Regarding claim 5, Kornfeld as modified teaches claim 4 as cited above, but does not disclose: wherein said backend computational device analyzes said plurality of functions in the plurality of spans according to said test results. However, in the analogous art of root cause of non-deterministic tests, ha teaches:  wherein said backend computational device analyzes said plurality of functions in the plurality of spans according to said test results (Fig. 4a, Fig. 5a, Col. [3] lines [32-47], Ha shows "Each subtree partially represents some call relations demonstrated in the overall trace. By comparing the subtree sets derived from PASS traces and those from FAIL traces, an exemplary system determines the subtree patterns unique to each set (113). FIGS. 4a and 4b illustrate subtrees of the successful execution shown in FIG. 3a. FIGS. 5a-5c illustrate subtrees of the failing execution shown in FIG. 3b. By comparing the two sets, we find differences in the methods called by method B. The exemplary system may then report patterns that only happen in PASS traces and subpatterns that only happen in FAIL traces. In some embodiments, the exemplary system may rank the diverging subtree patterns between the two sets based on the frequency of each pattern. The higher a pattern is ranked, the more likely it is to be a root cause of the non-deterministic test." Col. [1] lines [49-59], Ha shows "running the test, enabling call trace collection to record call traces from the test until at least one execution has succeeded and one execution has failed; for each trace in the call trace collection, determining whether the trace was part of a passing test or a failing test and labeling the trace according to the determination; constructing a dynamic call tree for the passing traces; constructing a dynamic call tree for the failing traces; comparing the dynamic call tree for the passing traces with the dynamic call tree for the failing traces to determine the subtree patterns that only occur in one tree or the other" The citation above shows grouping function calls into subtrees and each individual leaf node is a call. The tree is split into smaller subtrees that are analyzed and can be considered grouping or clustering the calls for analysis. Spans as defined in the specification are functional groups in relation to the code) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Ha into the teachings of Kornfeld as modified to implement "wherein said backend computational device first groups said plurality of functions into a plurality of spans, and then analyzes said plurality of functions according to said test results". The modification would have been obvious as one of ordinary skill in the art would be motivated to determine the root cause of the software test failures (Ha, [Abstract]).  Regarding claim 6, Kornfeld as modified in claim 5 teaches: wherein said different behavior comprises failure of at least one test (Para. [28], Kornfeld shows " root cause analysis instructions 246 may function to analyze the captured data to determine a root cause of a failure of the non-deterministic test based on a comparison between failed rerun executions (i.e., those of the rerun executions that failed) and passed rerun executions (i.e., those of the rerun executions that passed)"). With regards to claim 17, Kornfeld teaches claim 16 as cited above and it is a method claim having similar limitations as cited in claim 4 above. Thus, claim 17 is also rejected under the same rationale as cited in the rejection of claim 4 above. With regards to claim 18, Kornfeld teaches claim 17 as cited above and it is a method claim having similar limitations as cited in claim 5 above. Thus, claim 18 is also rejected under the same rationale as cited in the rejection of claim 5 above. Claim(s) 7-8, 10-11, 13, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20180300229 A1 (Hereinafter referred to as Kornfeld) and US 9311220 B1 (Hereinafter referred to as Ha) in further view of US 10678678 B1 (Hereinafter referred to as Tsoukalas). Regarding claim 7, Kornfeld as modified teaches claim 1 as cited above, but does not disclose:  wherein the backend computational device is further configured to utilize a build mapper to correlate test results with code changes However, in the analogous art of test execution, Tsoukalas teaches: wherein the backend computational device is further configured to utilize a build mapper to correlate test results with code changes (Fig. 6b(600-630), Col. [2] lines [22-39], Tsoukalas shows "tests may be automatically selected to run against program code based (at least in part) on the relevance of those tests to changes in the program code. In one embodiment, individual tests are run against the program code to determine which portions of the code (e.g., which source files and lines of code) are exercised by particular tests. The resulting code coverage data may be used to generate a mapping of individual tests to portions of program code. When a version of the program code with updated or new portions of code is sought to be tested, the portions that were changed or added (e.g., the source files and lines of code) may be determined, and the mapping may be used to determine which tests are relevant to (e.g., likely to be exercised by) these changed or new portions. A test selection system or service may produce an ordered sequence of tests that are likely to be exercised by the updated program code" Col [11] lines [29-32], Tsoukalas shows "the test execution results for a particular version of a software product may be provided to the test selection system 100 for aggregation with a global test execution history"). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tsoukalas into the teachings of Kornfeld to implement "wherein the backend computational device is further configured to utilize a build mapper to correlate test results with code change". The modification would have been obvious as one of ordinary skill in the art would be motivated to determine which tests are relevant to newly updated code (Tsoukalas, Col. [2] lines [27-29). Regarding claim 8, Kornfeld as modified teaches claim 7 as cited, but does not disclose:  wherein the build mapper is configured to determine relevance of tests based on code coverage and changes in the code. However, in the analogous art of test execution, Tsoukalas teaches: wherein the build mapper is configured to determine relevance of tests based on code coverage and changes in the code (Fig. 6b(600-630), Col. [2] lines [22-39], Tsoukalas shows "tests may be automatically selected to run against program code based (at least in part) on the relevance of those tests to changes in the program code. In one embodiment, individual tests are run against the program code to determine which portions of the code (e.g., which source files and lines of code) are exercised by particular tests. The resulting code coverage data may be used to generate a mapping of individual tests to portions of program code. When a version of the program code with updated or new portions of code is sought to be tested, the portions that were changed or added (e.g., the source files and lines of code) may be determined, and the mapping may be used to determine which tests are relevant to (e.g., likely to be exercised by) these changed or new portions. A test selection system or service may produce an ordered sequence of tests that are likely to be exercised by the updated program code"). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tsoukalas into the teachings of Kornfeld as modified to implement "wherein the build mapper is configured to determine relevance of tests based on code coverage and changes in the code". The modification would have been obvious as one of ordinary skill in the art would be motivated to reduce the number of tests to be run and also increase the effectiveness of the tests by automatically selecting the most relevant tests for a particular version of a software product (Tsoukalas, Col. [3] lines [20-25). Regarding claim 10, Kornfeld as modified teaches claim 1, but does not disclose:  wherein the backend computational device is further configured to perform test impact analysis to recommend tests for execution based on a policy set by a user. However, in the analogous art of test execution, Tsoukalas teaches:  wherein the backend computational device is further configured to perform test impact analysis to recommend tests for execution based on a policy set by a user (Col. [2] lines [37-63], Tsoukalas shows "a test selection system or service may produce an ordered sequence of tests that are likely to be exercised by the updated program code. In one embodiment, a confidence score may be determined for each test in a suite of tests, where the confidence score represents the likelihood that a test will exercise the updated or new portions of code, and tests whose confidences scores meet a predetermined confidence threshold may be included in the selection of tests while tests that fail to meet the confidence threshold may be excluded. In some embodiments, the order of the sequence may be determined based (at least in part) on factors such as the likelihood that a test will exercise the updated or new portions of code, the stability of a test (e.g., the likelihood that the test will detect a problem), the execution time of a test, and/or other factors that may contribute to a score for a test. A machine learning model may be used to score tests. In one embodiment, the number (N) of tests in the selection of tests may be determined based (at least in part) on user input, and the N tests having the highest scores may be ranked by their scores in the ordered sequence. The selected tests may be executed automatically, e.g., as part of a continuous integration system, continuous deployment system, or pipeline. By selecting and running only tests that are most relevant to changes or additions to program code, the techniques described herein may avoid taking the time and computational resources to run less relevant tests").  Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tsoukalas into the teachings of Kornfeld to implement “wherein the backend computational device is further configured to perform test impact analysis to recommend tests for execution based on a policy set by a user”. The modification would have been obvious as one of ordinary skill in the art would be motivated to reduce the number of tests to be run and also increase the effectiveness of the tests by automatically selecting the most relevant tests for a particular version of a software product (Tsoukalas, Col. [3] lines [20-25).  Regarding claim 11, Kornfeld as modified teaches claim 10, but does not disclose:  wherein the policy includes criteria based on test execution frequency, time frames, and code usage in production. However, in the analogous art of test execution, Tsoukalas teaches:  wherein the policy includes criteria based on test execution frequency, time frames, and code usage in production (Col. [3] line [49] - Col. [4] line [12], Tsoukalas shows "a suite of tests 180 may be determined based (at least in part) on user input. For example, a developer associated with program code 170 for a software product may supply or indicate tests that she or he deems to be relevant to the software product. However, some of the tests 180 may be relevant to portions of the software product that are not updated often, while others of the tests 180 may be relevant to portions of the software product that are more frequently updated, and yet others of the tests 180 may no longer be relevant to any portion of the software product due to maturation of the software product. Individual tests in the suite 180 may be configured with suitable parameters. In general, the test suite 180 may include performance tests such as sanity tests, latency tests, and/or load tests for scalability and throughput. Each test may be associated with an acceptable range of results, such that results outside the acceptable range may constitute a failure for that particular test. In one embodiment, various heuristics may be applied to determine whether the software product passes or fails a particular test. In one embodiment, the tests may include unit tests, e.g., tests in which the test host(s) do not access other systems over a network. In one embodiment, the tests may include integration tests, e.g., tests in which the test host(s) do access other systems over a network. The test selection system 100 may select relevant tests 181 to be run with respect to program code 171. The relevant tests 181 may be selected from the full test suite 180, such that the full test suite 180 typically includes more tests than the selected set 181. By reducing the number of tests to be executed, the test selection system 100 may reduce the time and computing hardware needed to test a software product" Col. [2] lines [37-63], Tsoukalas shows "a test selection system or service may produce an ordered sequence of tests that are likely to be exercised by the updated program code. In one embodiment, a confidence score may be determined for each test in a suite of tests, where the confidence score represents the likelihood that a test will exercise the updated or new portions of code, and tests whose confidences scores meet a predetermined confidence threshold may be included in the selection of tests while tests that fail to meet the confidence threshold may be excluded. In some embodiments, the order of the sequence may be determined based (at least in part) on factors such as the likelihood that a test will exercise the updated or new portions of code, the stability of a test (e.g., the likelihood that the test will detect a problem), the execution time of a test, and/or other factors that may contribute to a score for a test. A machine learning model may be used to score tests. In one embodiment, the number (N) of tests in the selection of tests may be determined based (at least in part) on user input, and the N tests having the highest scores may be ranked by their scores in the ordered sequence. The selected tests may be executed automatically, e.g., as part of a continuous integration system, continuous deployment system, or pipeline. By selecting and running only tests that are most relevant to changes or additions to program code, the techniques described herein may avoid taking the time and computational resources to run less relevant tests" The citation above shows reducing the amount of tests run to impact time needed to run test suites. The citation also shows determining relevancy of tests from disuse showing execution frequency. The citation also shows code usage in production as a determining factor as Tsoukalas looks at tests that were created to test not disused portions of code). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tsoukalas into the teachings of Kornfeld as modified to implement “wherein the policy includes criteria based on test execution frequency, time frames, and code usage in production”. The modification would have been obvious as one of ordinary skill in the art would be motivated to reduce the number of tests to be run and also increase the effectiveness of the tests by automatically selecting the most relevant tests for a particular version of a software product (Tsoukalas, Col. [3] lines [20-25).  Regarding claim 13, Kornfeld teaches claim 1 as cited above, but does not disclose :  wherein the backend computational device is further configured to compare test results to a threshold and apply span rules to determine a suspect score for test behavior However, in the analogous art of test execution, Tsoukalas teaches:  wherein the backend computational device is further configured to compare test results to a threshold and to determine a suspect score for test behavior (Col. [2] lines [37-46], Tsoukalas shows "A test selection system or service may produce an ordered sequence of tests that are likely to be exercised by the updated program code. In one embodiment, a confidence score may be determined for each test in a suite of tests, where the confidence score represents the likelihood that a test will exercise the updated or new portions of code, and tests whose confidences scores meet a predetermined confidence threshold may be included in the selection of tests while tests that fail to meet the confidence threshold may be excluded") In addition, in the analogous art of root cause of non deterministic tests, Ha teaches:  wherein the backend computational device is further configured to compare test results to a threshold and apply span rules (Col. [3] line [41-47], Ha shows "the exemplary system may then report patterns that only happen in PASS traces and subpatterns that only happen in FAIL traces. In some embodiments, the exemplary system may rank the diverging subtree patterns between the two sets based on the frequency of each pattern. The higher a pattern is ranked, the more likely it is to be a root cause of the non-deterministic test") Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tsoukalas into the teachings of Kornfeld to implement “wherein the backend computational device is further configured to compare test results to a threshold and apply span rules to determine a suspect score for test behavior”. The modification would have been obvious as one of ordinary skill in the art would be motivated to reduce the number of tests to be run and also increase the effectiveness of the tests by automatically selecting the most relevant tests for a particular version of a software product (Tsoukalas, Col. [3] lines [20-25). Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Ha into the teachings of Kornfeld to implement “wherein the backend computational device is further configured to compare test results to a threshold and apply span rules to determine a suspect score for test behavior”. The modification would have been obvious as one of ordinary skill in the art would be motivated to determine the root cause of the software test failures (Ha, [Abstract]). With regards to claim 19, Kornfeld teaches claim 14 as cited above and it is a method claim having similar limitations as cited in claim 7 above. Thus, claim 19 is also rejected under the same rationale as cited in the rejection of claim 7 above. Regarding claim 20, Kornfeld as modified teaches claim 19 as cited above, but does not disclose:  wherein the build mapper determines relevance of tests based on code coverage and changes in the code, and the method further comprises applying a machine learning model to determine the relevance of tests to specific code sections However, in the analogous art of test execution, Tsoukalas teaches:  wherein the build mapper determines relevance of tests based on code coverage and changes in the code, and the method further comprises applying a machine learning model to determine the relevance of tests to specific code sections (Fig. 6b(600-630), Col. [2] lines [22-39], Tsoukalas shows "tests may be automatically selected to run against program code based (at least in part) on the relevance of those tests to changes in the program code. In one embodiment, individual tests are run against the program code to determine which portions of the code (e.g., which source files and lines of code) are exercised by particular tests. The resulting code coverage data may be used to generate a mapping of individual tests to portions of program code. When a version of the program code with updated or new portions of code is sought to be tested, the portions that were changed or added (e.g., the source files and lines of code) may be determined, and the mapping may be used to determine which tests are relevant to (e.g., likely to be exercised by) these changed or new portions. A test selection system or service may produce an ordered sequence of tests that are likely to be exercised by the updated program code" Col [5] lines [59-65], Tsoukalas shows "In one embodiment, one or more machine learning models may be used to determine which tests are likely to be exercised by the updated program code. For example, if the updated program code includes new line numbers and/or new files, then the similarity of those new elements to previous elements may be assessed using machine learning so that relevant tests can be selected." Col. [12] lines [26-59], Tsoukalas shows "A client dataset 134 may be generated based (at least in part) on model data output by the machine learning model(s) 133, element data provided by the machine learning dataset 132, and test runtime data and test failure data derived from the raw dataset 131. The client dataset may be analogous to the mapping 130 shown in FIG. 1. The client dataset may include priority scores and/or confidence scores for various tests 180 as developed by the machine learning model(s) 133. The client dataset may include a model dataset for the data that defines the machine learning model(s) for each test, including coefficients, intercepts, and classes; a record dataset that maps data from an element of program code to a machine learning code coverage record; a test detail dataset that indicates the runtime and failure data for each test; and a test method mapping dataset that indicates the mapping data if a method is exercised by a test. The client dataset may again be reduced in size from the full machine learning dataset, thereby reducing storage requirements in storing the data and network latency in transmitting the data. The client dataset 134 may be stored in any suitable location, e.g., using a cloud-based storage service that is accessible to both the back end 400 and front end 410. The client dataset 134 may be downloaded to the test selection component 140 in the front end 410, where a set of relevant (and potentially ordered) tests 181 may be selected and ranked based (at least in part) on the change data for updated code 171 and on the client dataset 134. In one embodiment, the selected tests 181 may be executed in a test execution environment 420 using one or more computing devices such as the example computing device 3000 shown in FIG. 7. Feedback from the test execution environment 420 may be used to update the machine learning model(s) 133. For example, the machine learning model(s) 133 may be updated as test failure rates and test execution times change over time") Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tsoukalas into the teachings of Kornfeld as modified to implement “wherein the build mapper determines relevance of tests based on code coverage and changes in the code, and the method further comprises applying a machine learning model to determine the relevance of tests to specific code sections”. The modification would have been obvious as one of ordinary skill in the art would be motivated to reduce the number of tests to be run and also increase the effectiveness of the tests by automatically selecting the most relevant tests for a particular version of a software product (Tsoukalas, Col. [3] lines [20-25). Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20180300229 A1 (Hereinafter referred to as Kornfeld) and US 9311220 B1 (Hereinafter referred to as Ha) in further view of US 10678678 B1 (Hereinafter referred to as Tsoukalas) and US 20190095801 A1 (hereinafter referred to as Saillet). Regarding claim 9, Kornfeld as modified teaches claim 8 as cited above, but does not disclose:  wherein the build mapper includes a machine learning analyzer configured to apply a machine learning model to determine the relevance of tests to specific code sections; wherein the machine learning analyzer is further configured to receive information from a history analyzer to inform the machine learning model. machine learning analyzer is further configured to receive information from a statistical analyzer to inform the machine learning model. However, in the analogous art of test execution, Tsoukalas teaches:  wherein the build mapper includes a machine learning analyzer configured to apply a machine learning model to determine the relevance of tests to specific code sections (Col [5] lines [59-65], Tsoukalas shows "In one embodiment, one or more machine learning models may be used to determine which tests are likely to be exercised by the updated program code. For example, if the updated program code includes new line numbers and/or new files, then the similarity of those new elements to previous elements may be assessed using machine learning so that relevant tests can be selected." Col. [12] lines [26-59], Tsoukalas shows "A client dataset 134 may be generated based (at least in part) on model data output by the machine learning model(s) 133, element data provided by the machine learning dataset 132, and test runtime data and test failure data derived from the raw dataset 131. The client dataset may be analogous to the mapping 130 shown in FIG. 1. The client dataset may include priority scores and/or confidence scores for various tests 180 as developed by the machine learning model(s) 133. The client dataset may include a model dataset for the data that defines the machine learning model(s) for each test, including coefficients, intercepts, and classes; a record dataset that maps data from an element of program code to a machine learning code coverage record; a test detail dataset that indicates the runtime and failure data for each test; and a test method mapping dataset that indicates the mapping data if a method is exercised by a test. The client dataset may again be reduced in size from the full machine learning dataset, thereby reducing storage requirements in storing the data and network latency in transmitting the data. The client dataset 134 may be stored in any suitable location, e.g., using a cloud-based storage service that is accessible to both the back end 400 and front end 410. The client dataset 134 may be downloaded to the test selection component 140 in the front end 410, where a set of relevant (and potentially ordered) tests 181 may be selected and ranked based (at least in part) on the change data for updated code 171 and on the client dataset 134. In one embodiment, the selected tests 181 may be executed in a test execution environment 420 using one or more computing devices such as the example computing device 3000 shown in FIG. 7. Feedback from the test execution environment 420 may be used to update the machine learning model(s) 133. For example, the machine learning model(s) 133 may be updated as test failure rates and test execution times change over time"); wherein the machine learning analyzer is further configured to receive information from a history analyzer to inform the machine learning model (Col. [11] line [60] - Col. [12] line [15], Tsoukalas shows "the code coverage data resulting from the test execution 110 may be used to generate a raw dataset 131. The raw dataset 131 may be uncompressed data that includes element information, test failure data, test runtime data, and data about the portions of program code that were exercised by each test. For example, the raw dataset 131 may indicate which methods, classes, packages, and/or groups were exercised by each test. The raw dataset 131 may be restructured into a machine learning dataset 132. The machine learning dataset 132 may be more suitable for use in training one or more machine learning models 133. In one embodiment, the machine learning dataset 132 may be significantly smaller in file size than the raw dataset 131, e.g., using techniques to compress the data and eliminate redundancies. In one example, the restructuring of the raw dataset into the machine learning dataset reduced the file size from 1 GB to 115 MB. The machine learning dataset 132 may include a two-dimensional (2D) array for each test. Each vector in the 2D array may represent method coverage, group coverage, class coverage, and a test method mapping rate. The method coverage may represent the value that is predicted, and the test method mapping rate may represent a percentage of methods that are exercised by a test"). In addition, in the analogous art of data preparation, Saillet teaches:  wherein the machine learning analyzer is further configured to receive information from a statistical analyzer to inform the machine learning model (Para. [54], Saillet shows "dataset analyzer 313 may determine metrics and properties for each of the fields of dataset 311, for input to a classification model such as machine learning model 302" The above citation shows Saillet includes a dataset analyzer which analyzes metrics to be used as input to a machine learning model). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tsoukalas into the teachings of Kornfeld as modified to implement "wherein the build mapper determines relevance of tests based on code coverage and changes in the code, and the method further comprises applying a machine learning model to determine the relevance of tests to specific code sections". The modification would have been obvious as one of ordinary skill in the art would be motivated to reduce the number of tests to be run and also increase the effectiveness of the tests by automatically selecting the most relevant tests for a particular version of a software product (Tsoukalas, Col. [3] lines [20-25). Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Saillet into the teachings of Kornfeld as modified to implement “wherein the build mapper includes a machine learning analyzer configured to apply a machine learning model to determine the relevance of tests to specific code sections; wherein the machine learning analyzer is further configured to receive information from a history analyzer and a statistical analyzer to inform the machine learning model”. The modification would have been obvious as one of ordinary skill in the art would be motivated processing data is time consuming and doing so manually can be prone to error (Saillet, Para. [2]). Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US 20180293159 A1 – This prior art shows reducing flaky test possibility Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZEERICK A MALIK whose telephone number is (571)272-8110. The examiner can normally be reached Mon-Thurs, 7-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chat Do can be reached at (571) 272-3721. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /Z.A.M./Examiner, Art Unit 2193 /Chat C Do/Supervisory Patent Examiner, Art Unit 2193
Read full office action

Prosecution Timeline

Jun 10, 2024
Application Filed
May 12, 2026
Non-Final Rejection mailed — §101, §103
Jul 23, 2026
Interview Requested
Jul 29, 2026
Applicant Interview (Telephonic)
Aug 04, 2026
Examiner Interview Summary
Aug 11, 2026
Response Filed
Oct 21, 2026
Final Rejection mailed — §101, §103 (current)

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

3-4
Expected OA Rounds
0%
Grant Probability
0%
With Interview (+0.0%)
3y 1m (~8m remaining)
Median Time to Grant
Moderate
PTA Risk
Based on 2 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month