DETAILED ACTION
This action is responsive to the application filed on 04/06/2026. Claims 1-2, 3-18, and 22-24 are pending and have been examined. This action is final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C.
120, 121, 365(c), or 386(c) is acknowledged.
Response to Arguments
Argument 1: Applicant argues that amended claims 1, 8, and 15 are patent eligible because the amendments allegedly tie the claimed invention to a practical software-testing improvement, not merely an abstract idea. Specifically, applicant emphasizes the amended limitations requiring a trinary neural network, discrete positive/negative/zero correlation outputs, and masked test-result inputs where the value of the first test is set to zero so that the neural network receives no indication of whether the first test passed or failed. Applicant contends this improves how computer software is tested by making the process more efficient, improving debugging and execution efficiency, and using the neural network to determine test dependencies rather than merely performing a mental process. Applicant further argues that claim 1 integrates any alleged abstract idea into a practical application under Step 2A, Prong Two, and provides significantly more under Step 2B; applicant then applies the same reasoning to independent claims 8 and 15 and their dependent claims.
Response to Argument 1: Applicant’s arguments have been fully considered but are not persuasive. The amendments to claims 1 (8 and 15 by analogy) do not overcome the rejection under 35 U.S.C. 101 because the claims remain directed to abstract data analysis and mathematical/data-representation concepts implemented on generic computer components. Specifically, the amended limitations reciting a trinary neural network, positive/negative/zero correlation outputs, masking a first test value with zero to provide no indication of whether the first test passed or failed, determining a failure output, storing a dependency indication, and performing testing based on that indication merely further define how test-result data is numerically represented, analyzed, and used. The claims do not recite a specific improvement to the operation of the processor, memory, neural network architecture, neural network training, or execution of computer code itself. Rather, the claims use a generic processor, memory, and neural network as tools to process pass/fail test-result information and determine dependency information. The recited “performing testing on the computer code based on the indication” is also broadly stated and does not recite a particular technical manner of performing the testing or a specific technical improvement to computer functionality. Therefore, the claims do not integrate the abstract idea into a practical application under Step 2A, Prong Two, and do not add significantly more under Step 2B. Hence, the rejection is maintained
Argument 2: Applicant argues that the art rejections should be withdrawn because the cited combinations do not teach the newly emphasized claim limitations. For claim 1 and related claims, applicant focuses on the requirement that the input set masks the value of the first input/test with zero to provide no indication of whether the first test passed or failed. Applicant argues that Fong only teaches using neural networks to identify or focus on parameters related to test actions, which applicant says is not the same as masking a pass/fail value with zero. Applicant similarly argues that Tsoukalas teaches success/failure assessment as an output of test execution, not an input value masked to provide no pass/fail indication. Therefore, applicant asserts that Poornaki, Vanmali, Fong, and Tsoukalas fail to render obvious claim 1 and dependent claims 7 and 22. Applicant also separately argues that claim 22 is not taught because Vanmali’s synaptic weights range from -0.5 to 0.5, which applicant says is a continuous range of infinitely many values, not the claimed three discrete outputs of -1, 0, and 1. Applicant’s position is that merely including zero within a continuous range does not disclose a discrete zero state, and that classifying values as positive, negative, or zero does not change the fact that Vanmali allegedly uses continuous values rather than exactly three discrete outputs. Applicant then argues that the same deficiencies carry through to amended independent claims 8 and 15, and that the additional references used for dependent claims 2, 3, 5, 4, 6, 14, 16-18, and 20 do not cure the deficiencies of Poornaki and Vanmali as applied to the independent claims.
Response to Argument 2: Applicant’s arguments have been fully considered but are not persuasive because the rejection, as set forth, maps the amended limitations to the cited art combination as a whole, not to any single reference in isolation. Poornaki provides the base predictive-model system, Vanmali teaches neural-network-based software testing and evaluation of application errors, Sharif teaches software test execution-status inputs including pass, fail, and non-executed/no-pass-fail-status information, El-Khamy teaches masking neural-network input/feature values by puncturing and overwriting selected values with zero so that those values do not contribute to the neural-network computation, and Yalla teaches using a trained neural-network model to predict test cases to execute and causing those test cases to be executed in a software development platform. Thus, the amended limitation requiring the first input to identify whether the first test passes or fails and to indicate no pass/fail information is addressed by Sharif, the zero-masking limitation is addressed by El-Khamy, and the “performing testing on the computer code based on the indication” limitation is addressed by Yalla. Applicant’s arguments regarding Fong and Tsoukalas therefore do not rebut the rejection as currently presented because the amended claim 1 mapping no longer relies on Fong or Tsoukalas for the zero-masked first-input limitation. Applicant’s argument regarding claim 22 is also not persuasive because, under the broadest reasonable interpretation, Vanmali’s neural-network synaptic weights having signed numerical values within a range including positive values, negative values, and zero, together with the mapped neural-network output classifications, reasonably teaches or suggests using signed numerical states to represent positive, negative, and no-correlation states in a neural-network-based testing system. Accordingly, the cited combination renders claims 1, 8, 15, 22, and the related dependent claims obvious.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition
of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the
conditions and requirements of this title.
Claims 1-2, 3-18, and 22-24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1: The claim is directed to a system, which is considered to be in the category of machine. Therefore, claim 1 satisfies Step 1.
Step 2A Prong 1:
(a) “wherein respective weights of the neural network indicate one of a discrete set of three outputs, and wherein one output of the discrete set of three outputs is a positive number, which indicates a positive correlation between a respective output of a respective node of the neural network and the output of the neural network, another output of the discrete set of three outputs is a negative number, which indicates a negative correlation between the respective output and the output, and a third output of the discrete set of three outputs is zero, which indicates no correlation between the respective output and the output” -- This limitation is directed to representing correlation relationships using numerical values, including a positive number, a negative number, and zero. The limitation recites mathematical relationships and mathematical concepts because it uses numerical values to represent positive correlation, negative correlation, and no correlation between neural network outputs.
(b) “applying sets of inputs to the neural network, respective inputs of the sets of inputs identifying whether the respective tests pass or fail, wherein the sets of inputs mask a value of a first input to the neural network, the first input identifying whether the first test passes or fails, wherein the first input is configured to indicate one of the first test passing, the first test failing, or no indication of whether the first test passed or failed, and wherein the value of the first test is masked with a value of zero to provide no indication of whether the first test passed or failed” - This limitation is directed to manipulating and classifying test-result data using symbolic or numerical values, including setting a value to zero to represent no indication of a pass or fail result. The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, with/out aid of pen and paper, and thus it is directed to a mental process.
(c) “in response to determining that a first set of inputs of the sets of inputs to the neural network results in a failure output, storing an indication that the first test is dependent on a subset of the respective tests indicated as failing by the first set of inputs” -- This limitation is directed to evaluating test-result data and determining a dependency relationship based on a failure output and a subset of failed tests. The limitation is directed to a process that can be performed in the human mind using evaluation, observation, and judgement, with/out aid of pen and paper, and thus it is directed to a mental process.
Step 2A Prong 2 and Step 2B:
(a) “A system, comprising: at least one processor; and at least one memory that stores executable instructions that, when executed by the at least one processor, facilitate performance of operations, comprising: applying sets of inputs to the neural network” -- The limitation recites instructions executed by a generic processor to apply inputs, which amounts to no more than instructions to apply the abstract idea on a computer. The limitation does not integrate the judicial exception into a practical application, nor does it provide significantly more than the judicial exception (see MPEP 2106.05(f)).
(b) “generating a neural network, wherein an output of the neural network indicates whether a first test of a computer code will pass given an input of respective results of whether respective tests, of a group of tests of the computer code, pass” -- This limitation recites the gathering and manipulation of input data to produce output data. Such activity is considered insignificant extra-solution activity and does not integrate the abstract idea into a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, generating and transmitting data for further processing is a well-understood, routine, and conventional activity and does not provide significantly more (see MPEP 2106.05(d)(II)).
(c) “wherein the neural network comprises a trinary neural network” - This limitation merely limits the abstract data-analysis process to a particular type of neural network model. The limitation does not recite a specific improvement to neural network training, neural network architecture operation, processor operation, memory operation, or computer functionality. Rather, it further defines the mathematical/data-representation model used to perform the abstract analysis. Therefore, this limitation does not integrate the judicial exception into a practical application and does not provide significantly more than the judicial exception (see MPEP 2106.05(a) and 2106.05(h)).
(d) “applying sets of inputs to the neural network” - This limitation recites applying input data to the neural network for processing. The limitation amounts to using a generic computer tool to apply data to the abstract model and does not impose a meaningful technological limitation on the claimed system. Therefore, this limitation does not integrate the judicial exception into a practical application and does not provide significantly more than the judicial exception (see MPEP 2106.05(f)).
(e) “storing an indication that the first test is dependent on a subset of the respective tests indicated as failing by the first set of inputs” - This limitation recites storing the result of the abstract analysis. Storing and organizing data is an insignificant extra-solution activity and is also a well-understood, routine, and conventional (WURC) computer function. Therefore, this limitation does not integrate the judicial exception into a practical application and (see MPEP 2106.05(g)). Furthermore, under Step 2B, storing gathered data/results to be used is WURC, and does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II))
(f) “performing testing on the computer code based on the indication” - This limitation recites using the result of the abstract dependency determination to perform testing on computer code. However, the claim does not recite a particular technical manner of performing the testing, a specific change to the execution of the computer code, or an improvement to computer functionality itself. Instead, the limitation broadly applies the result of the abstract analysis in the field of software testing. Therefore, this limitation does not integrate the judicial exception into a practical application and does not provide significantly more than the judicial exception (see MPEP 2106.05(f), 2106.05(g), and/or 2106.05(h)).
Thus, claim 1 is non-patent eligible. Claims 8 and 15 are analogous to claim 1, aside from claim type, and therefore face the same rejection as recited above.
Regarding claim 2,
Step 1: The claim is directed to a system, which is considered to be in the category of machine. The claim satisfies
step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
(a) “The system of claim 1, wherein the output is a first output, and wherein the neural network
comprises a number of outputs that corresponds to a number of tests of the group of tests.” - The
limitation recites mere further limitations regarding the output, and further recites what the neural network
first introduced in claim 1 will include. The limitation cannot be integrated to a practical application, nor
can it provide significantly more than the judicial exception (see MPEР 2106.05(h)).
Thus, claim 2 is non-patent eligible.
Regarding claim 5,
Step 1: The claim is directed to a system, which is considered to be in the category of machine. The claim satisfies
step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
(a) “The system of claim 1, wherein the neural network is a first neural network, wherein the sets of
inputs are first sets of inputs, and wherein the operations further comprise: applying second sets of
inputs to a second neural network to determine whether a second test is dependent on a respective
second set of the second sets of inputs.” - The limitation recites mere instructions to apply sets of inputs
onto the neural network for the purpose of determining if a second test is dependent on a second set of
inputs. The limitation cannot amount to a practical application, nor can it provide significantly more than
the judicial exception (see MPEР 2106.05(f)).
Thus, claim 5 is non-patent eligible.
Regarding claim 6,
Step 1: The claim is directed to a system, which is considered to be in the category of machine. The claim satisfies
step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
(a) “The system of claim 5, wherein the indication is a first indication, wherein the first sets of inputs
comprises a second indication of whether the second test passes, and wherein the second sets of
inputs comprises a third indication of whether the first test passes.” - The limitation is directed to
merely further limiting the indications and tests that were introduced in earlier claims to more than one
indication or test run, and same for the second set of inputs. This cannot amount to a practical application,
nor can it provide significantly more than the judicial exception (see MPРЕР 2106.05(h)).
Thus, claim 6 is non-patent eligible.
Regarding claim 7,
Step 1: The claim is directed to a system, which is considered to be in the category of machine. The claim satisfies
step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
(a) “The system of claim 1, wherein the indication is a first indication, wherein the subset is a first
subset, and wherein the operations further comprise: in response to determining that a second set of
inputs of the sets of inputs to the neural network results in the failure output, storing a second
indication that the first test is dependent on a second subset of the respective tests indicated as failing
by the second set of inputs.” - The limitation recites the same limitation that was first recited in claim 1 to
further include a “first” indication, a “first” subset, determining a “second” set of inputs, and storing
another indication that the first test is depend on another subset of other tests that were indicated as failing.
The limitation does not amount to any more than further limiting elements of claim 1 to a particular field of
use/environment, and thus it cannot be integrated to a practical application, nor can it provide significantly
more than the judicial exception (see MPEP 2106.05 (h)).
Thus, claim 7 is non-patent eligible.
Regarding claim 9,
Step 1: The claim is directed to a system, which is considered to be in the category of machine. The claim satisfies
step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
(a) “The method of claim 8, wherein the respective weights of the neural network indicate one of the
positive correlation, the negative correlation, or no correlation between the respective outputs of the
respective nodes of the neural network and the output of the neural network.” -The limitation recites
similar limitations that were already introduced and recited in claim 1. The limitation is recited in a high
level of generality, merely limiting the respective weights of the neural network to further includes type of
correlation between outputs, thus it cannot be integrated to a practical application, nor can it provide
significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 9 is non-patent eligible.
Regarding claim 10,
Step 1: The claim is directed to a method, which is considered to be in the category of process. The claim satisfies
step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
(a) “The method of claim 8, further comprising: after determining that the first set of inputs to the
neural network results in the failure, and in response to determining that a second set of inputs of the
sets of inputs comprises a superset of the first set of inputs, determining, by the system, to omit
applying the second set of inputs to the neural network.” - The limitation recites that the second set of
inputs will further comprise/include a “superset” of the first set of inputs and instead letting the system
determine to not apply the second set onto the neural network. The limitation does not amount to more than
further limiting the input sets to a field of use/environment, and thus it cannot be integrated to a practical
application, nor can it provide significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 10 is non-patent eligible.
Regarding claim 11,
Step 1: The claim is directed to a method, which is considered to be in the category of process. The claim satisfies step 1.
Step 2A Prong 1:
(a) “The method of claim 8, wherein the respective inputs of the sets of inputs have an upper limit of inputs of the respective inputs that indicate failed tests, and wherein the upper limit is less than a number of tests in the group of tests.” - The limitation is directed to the respective inputs will have a predetermined set upper limit, that will be compared to a number of tests from the group/plurality of tests. Setting an upper limit and comparing it to gathered data is a process that can be performed in the human mind using evaluation, operation, and judgement, and thus it is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 11 is non-patent eligible.
Regarding claim 12,
Step 1: The claim is directed to a method, which is considered to be in the category of process. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The method of claim 11, wherein a number of inputs of the respective inputs that indicate failed tests vary between 1 and the upper limit.” - The limitation recites that the number of inputs of respective inputs first introduced in earlier claims will further be limited to values that will be between 1 and the upper limit, which merely limits the limitation to a particular value (environment of finite values), and thus it cannot be integrated to a practical application, nor can it provide significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 12 is non-patent eligible.
Regarding claim 13,
Step 1: The claim is directed to a method, which is considered to be in the category of process. The claim satisfies
step 1.
Step 2A Prong 1:
(a) “The method of claim 11, wherein the upper limit is proportionate to a logarithm of a number of the
respective tests.”- The limitation is directed to the upper limit to be relative to a logarithmic number of
tests. The limitation is directed to calculating a logarithmic number of tests, which would involve
mathematical calculation/operation to do. Thus the limitation is directed to math.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 13 is non-patent eligible.
Regarding claim 14,
Step 1: The claim is directed to a method, which is considered to be in the category of process. The claim satisfies
step 1.
Step 2A Prong 1:
(a) “The method of claim 8, wherein a first size of data that represents a first weight of the respective
weights is smaller than a second size of word size of the processor,”-The limitation is directed to a size
value corresponds to a weight value that has been determined to be smaller than a second word size related
to the processor. Determining a weight to be smaller than another can be performed in the human mind
using evaluation, observation, and judgement, with aid of pen and paper. Thus the limitation is directed to a
mental process.
Step 2A Prong 2 and Step 2B:
(a) “and wherein multiple weights of the respective weights are combined into a first word of the
processor and processed in parallel.” - The limitation recites merely multiple weight values to be
combined to a word representation related to the processor and will be applied to the computer by
processing in parallel. The act of combining data to be processed on a computer is recited in a high level of
generality, and it cannot be integrated to a practical application, nor can it provide significantly more than
the judicial exception (see MPEP 2106.05(f)).
Thus, claim 14 is non-patent eligible.
Regarding claim 16,
Step 1: The claim is directed to a non-transitory computer-readable medium, which is considered to be in the category of manufacture. Therefore, claim 16 satisfies Step 1.
Step 2A Prong 1:
“The non-transitory computer-readable medium of claim 15, further comprising: in response to determining that a second test of the subset of the respective tests fails, and to determining that the first test depends on the subset of the respective tests, determining to omit testing the computer code with the first test” - This limitation is directed to evaluating test-result information and determining whether to omit a test based on a dependency relationship between tests. Determining whether one test depends on a subset of tests and determining whether to omit another test based on that dependency is an abstract data-analysis and decision-making process based on test-result classifications. Therefore, this limitation recites a mental process and/or mathematical/data-analysis concept.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 16 is non-patent eligible.
Regarding claim 17,
Step 1: The claim is directed to a non-transitory computer-readable medium, which is directed to the category an
article of manufacture. The claim satisfies step 1.
Step 2A Prong 1:
(a) “The non-transitory computer-readable medium of claim 15, wherein a dimensionality of the output
of the neural network is equal to one.” - The limitation is directed to setting the dimensionality
value of the output for the neural network to be set to the value of one. Setting a mathematical value
to a finite number, 1,can be done in the human mind and/or by using pen and paper to determine the
dimensionality and set it to 1, thus the limitation is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 17 is non-patent eligible.
Regarding claim 18,
Step 1: The claim is directed to a non-transitory computer-readable medium, which is directed to the category an
article of manufacture. The claim satisfies step 1.
Step 2A Prong 1:
“The non-transitory computer-readable medium of claim 15, wherein a dimensionality of the
respective inputs of the neural network is equal to a number of the respective tests that are
separate from the first test.” - The limitation is directed to the dimensionality of the neural
network to be equal to the number of tests separate from the first test. A human mind is capable of
determining/setting the dimensionality of the neural network to have the same value of the number
of tests that have occurred using evaluation, observation, and judgement, with aid of using pen and paper to
calculate the dimensionality. Thus, the limitation is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 18 is non-patent eligible.
Regarding claim 20,
Step 1: The claim is directed to a non-transitory computer-readable medium, which is directed to the category an
article of manufacture. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The non-transitory computer-readable medium of claim 15, wherein a number inputs of the first
input is equal to a number of tests of the respective tests that are separate from the first test.” - The
limitation is directed to the number of inputs of a first input to be equal to a number of tests out of the
respective tests for which are separate from the first test. The limitation does not amount to more than
merely limiting the number of the inputs to a particular field of use/environment of being equal to a number
of tests, and it cannot be integrated to a practical application, nor can it provide significantly more than the
judicial exception (see MPEP 2106.05(h)).
Thus, claim 20 is non-patent eligible.
Regarding claim 22,
Step 1: The claim is directed to a system, which is considered to be in the category of machine. The claim satisfies
step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The system of claim 1, wherein the three outputs of the discrete set of three outputs are -1, 0, and 1.” -- The limitation recites that the three outputs of the discrete set of three outputs are between -1 and 1. The limitation amounts to no more than merely further limiting the claim to a field of use/environment, and it does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 22 is non-patent eligible.
Regarding claim 23,
Step 1: The claim is directed to a system, which is considered to be in the category of machine. Therefore, claim 23 satisfies Step 1.
Step 2A Prong 1:
(a) “The system of claim 1, wherein the operations further comprise: after determining that the first set of inputs to the neural network results in the failure, and in response to determining that a second set of inputs of the sets of inputs comprises a superset of the first set of inputs, determining to omit applying the second set of inputs to the neural network.” - This limitation is directed to evaluating input sets, determining whether one input set is a superset of another input set, and determining whether to omit applying an input set based on that relationship. Determining whether a second set is a superset of a first set is a mathematical/set relationship, and deciding whether to omit applying the second set based on that relationship is an abstract data-analysis and decision-making process. Therefore, this limitation recites a mathematical concept and/or mental process under Step 2A Prong 1.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 23 is non-patent eligible.
Regarding claim 24,
Step 1: The claim is directed to a system, which is considered to be in the category of machine. Therefore, claim 24 satisfies Step 1.
Step 2A Prong 1:
“wherein the respective inputs of the sets of inputs have an upper limit of inputs of the respective inputs that indicate failed tests, and wherein the upper limit is less than a number of tests in the group of tests.” - This limitation is directed to setting or defining a numerical upper limit for inputs that indicate failed tests and comparing that upper limit to a number of tests in a group of tests. Setting a numerical limit and comparing the limit to another numerical value is a mathematical concept, and thus the limitation is directed to math.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 24 is non-patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this
Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not
identically disclosed as set forth in section 102, if the differences between the claimed invention and the
prior art are such that the claimed invention as a whole would have been obvious before the effective filing
date of the claimed invention to a person having ordinary skill in the art to which the claimed invention
pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are
summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 1,7-10,14-15, 18, and 22-24 are rejected under 35 U.S.C. 103 as being unpatentable over Poornaki et al., US 2020/0210824 A1, “Scalable system and method for forecasting wind turbine failure with varying lead time windows” (referred herein as Poornaki), in view of NPL reference “Using a Neural Network in the Software Testing Process,” by Vanmali et al. (referred herein as Vanmali), in further view of NPL reference “DeepOrder: Deep Learning for Test Case Prioritization in Continuous Integration Testing,” by Sharif et al. (referred herein as Sharif), in further view of El-Khamy et al., US 2018/0300624 A1, “Method and apparatus for reducing computational complexity of convolutional neural networks” (referred herein as El-Khamy), and in further view of Yalla et al., US 10,949,337 B1, “Utilizing neural network and artificial intelligence models to select and execute test cases in a software development platform” (referred herein as Yalla).
Regarding claim 1, Poornaki teaches:
A system, comprising: at least one processor; and at least one memory that stores executable instructions that, when executed by the at least one processor, facilitate performance of operations, comprising … applying sets of inputs to the neural network, respective inputs of the sets of inputs identifying whether the respective tests pass or fail ([Poornaki, [0008-0009] “An example system may comprise at least one processor and memory containing instructions, the instructions being executable by the at least one processor to… applying the first selected failure prediction model to the current sensor data to generate a first failure prediction… comparing the first failure prediction to a trigger criteria…”, wherein the examiner interprets “applying the first selected failure prediction model to the current sensor data to generate a first failure prediction… comparing the first failure prediction to a trigger criteria” to be the same as “applying sets of inputs to the neural network, respective inputs…identifying whether the respective tests pass or fail”, because they are both directed to providing a set of input values to a predictive model for classification of a condition based on the model’s output.)
Poornaki does not teach generating a neural network, wherein an output of the neural network indicates whether a first test of a computer code will pass given an input of respective results of whether respective tests, of a group of tests of the computer code, pass…wherein the neural network comprises a trinary neural network...wherein respective weights of the neural network indicate one of a discrete set of three outputs, and rein one output of the discrete set of three outputs is a positive number, which indicates a positive correlation between a respective output of a respective node of the neural network and the output of the neural network, another output of the discrete set of three outputs is a negative number, which indicates a negative correlation between the respective output and the output, and a third output of the discrete set of three outputs is zero, which indicates no correlation between the respective output and the output, and a third output of the discrete set of three outputs is zero, which indicates no correlation between the respective output and the output; applying sets of inputs to the neural network, respective inputs of the sets of inputs identifying whether the respective tests pass or fail, wherein the sets of inputs mask a value of a first input to the neural network, the first input identifying whether the first test passes or fails, wherein the first input is configured to indicate one of the first test passing, the first test failing, or no indication of whether the first test passed or failed, and wherein the value of the first test is masked with a value of zero to provide no indication of whether the first test passed or failed; in response to determining that a first set of inputs of the sets of inputs to the neural network results in a failure output, storing an indication that the first test is dependent on a subset of the respective tests indicated as failing by the first set of inputs, and performing testing on the computer code based on the indication.
wherein respective weights of the neural network indicate one of a discrete set of three outputs, and wherein one output of the discrete set of three outputs is a positive number, which indicates a positive correlation between a respective output of a respective node of the neural network and the output of the neural network, another output of the discrete set of three outputs is a negative number, which indicates a negative correlation between the respective output and the output, and a third output of the discrete set of three outputs is zero, which indicates no correlation between the respective output and the output (([Vanmali, page 46, 48, 53, sec 1,2] “A multi-layer neural network is trained on the original software application by using randomly generated test data that conform to the specification…the trained neural network is used to produce a particular output when presented with an input signal…the synaptic weights of the network…” AND [Vanmali, page 51-52] “The comparison tool is employed as an independent method of comparing the results from the neural network and the results of the- tested versions of the credit app-roval application.”, and Table I:
PNG
media_image1.png
164
563
media_image1.png
Greyscale
The tool uses the output of a neural network and the output of the tested application. The distance between the outputs is taken as the absolute difference between the value of the winning node for each output and the corresponding value in the application”, wherein the examiner interprets “a positive weight represents a positive correlation while a negative weight reflects a negative correlation” to be the same as one output of the discrete set of three outputs is a positive number, which indicates a positive correlation between a respective output of a respective node of the neural network and the output of the neural network, and another output of the discrete set of three outputs is a negative number, which indicates a negative correlation between the respective output and the output, as both are directed to representing the direction of correlation between a respective node output of the neural network and the output of the neural network using signed weight values, and Table I, which defines output categories including true positive, true negative, false positive, and false negative, to be the same as a third output of the discrete set of three outputs is zero, which indicates no correlation between the respective output and the output, as both are directed to an output state in which there is no correspondence between the neural network output and the tested application output.)
in response to determining that a first set of inputs of the sets of inputs to the neural network results in a failure output, storing an indication that the first test is dependent on a subset of the respective tests indicated as failing by the first set of inputs” ([Vanmali, page 52, sec 3], “the output of the tested application is wrong, the evaluation of the comparison tool is classified as being a true negative or a category of 2, i.e., the determination that the output of the application is an actual error.”, wherein the examiner interprets “the determination that the output of the application is an actual error” to be the same as “storing an indication… indicated as failing” because they are both directed to recording a classification or flag that a specific test has failed based on the neural network’s evaluation of a given set of inputs.)
Poornaki and Vanmali do not teach wherein the sets of inputs mask a value of a first input to the neural network, the first input identifying whether the first test passes or fails, wherein the first input is configured to indicate one of the first test passing, the first test failing, or no indication of whether the first test passed or failed, and wherein the value of the first test is masked with a value of zero to provide no indication of whether the first test passed or failed; and performing testing on the computer code based on the indication.
Sharif teaches:
the first input identifying whether the first test passes or fails; wherein the first input is configured to indicate one of the first test passing, the first test failing, or no indication of whether the first test passed or failed; ([Sharif, page 2-3, sec 2-3] “
PNG
media_image2.png
73
315
media_image2.png
Greyscale
” AND “Given a test suite TS = {t1,...,tn} as a set of test cases, their historical test case execution effectiveness (pass/fail status) ES” , wherein the examiner interprets “1, if ti failed in cycle j” and “0, if ti passed in cycle j… as a set of test cases, their historical test case execution effectiveness (pass/fail status) ES” to be the same as “the first input identifying whether the first test passes or fails” because they are both directed to an input value identifying/providing indication whether a software test passed or failed.)
Poornaki, Vanmali, and Sharif do not teach wherein the sets of inputs mask a value of a first input to the neural network … and wherein the value of the first test is masked with a value of zero to provide no indication of whether the first test passed or failed; and performing testing on the computer code based on the indication.
El-Khamy teaches wherein the sets of inputs mask a value of a first input to the neural network … and wherein the value of the first test is masked with a value of zero to provide no indication of whether the first test passed or failed. ([El-Khamy, page 12-13, col 5,7, lines 62-63] “The feature map X may be punctured by applying thereto mask T using a dot product operation…while overwriting some values of the feature map and / or kernel elements with zeros” AND [El-Khamy, page 11, col 3-4, lines 62-63, 2-3] “Skipping a MAC operation is equivalent to having one of the operands in the multiplication to be zero…puncture the input feature map by overwriting some values to zero”, wherein the examiner interprets “the feature map X may be punctured by applying thereto mask T using a dot product operation” and “overwriting some values of the feature map and/or kernel elements with zeros” to be the same as “wherein the sets of inputs mask a value of a first input to the neural network” because they are both directed to applying a mask to neural-network input/feature values so selected values are zeroed or prevented from contributing to neural-network processing.)
Poornaki, Vanmali, Sharif, and El-Khamy do not teach and performing testing on the computer code based on the indication.
Yalla teaches and performing testing on the computer code based on the indication; ([Yalla, Abstract] “process the software data, with the trained neural network model, to predict a set of test cases to execute for testing the software.” and [Yalla, col. 2, lines 22-24] “cause the set of test cases to be executed in the software development platform based on the configurations, the scripts, and the test targets”, wherein the examiner interprets “predict a set of test cases to execute for testing the software in the software development platform” to be the same as “the indication” because they are both directed to output information used to determine what testing is performed, and wherein the examiner interprets “cause the set of test cases to be executed in the software development platform” to be the same as “performing testing on the computer code” because they are both directed to executing software test cases for testing software.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and the instant application are analogous art because they are all directed to computer-implemented failure prediction, neural-network-based software testing, software test-case execution, software test-case prioritization, or neural-network processing techniques for evaluating inputs and determining failure-related outputs.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the failure-prediction system disclosed by Poornaki to include the neural-network software-testing technique disclosed by Vanmali. One would be motivated to do so to effectively use a neural network as an automated software-testing model for evaluating software outputs, as suggested by Vanmali ([Vanmali, page 3] “Using an automated oracle to support the activities of human testers can reduce the actual cost of the testing process and the related maintenance costs.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the predictive-model/neural-network software-testing system disclosed by Poornaki and Vanmali to include the historical pass/fail execution-status features disclosed by Sharif. One would be motivated to do so to effectively use historical pass/fail and not-executed test-status information as neural-network input data for software test prioritization and selection, as suggested by Sharif ([Sharif, Abstract] “DeepOrder learns failed test cases”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the predictive-model/neural-network software-testing system disclosed by Poornaki, Vanmali, and Sharif to include the masking and zeroing technique disclosed by El-Khamy. One would be motivated to do so to efficiently reduce neural-network computation by zeroing selected neural-network input/feature values so that selected values do not contribute to neural-network computation, as suggested by El-Khamy ([El-Khamy, page 11, col. 3-4, lines 62-63, 2-3] “Skipping a MAC operation is equivalent to having one of the operands in the multiplication to be zero” and “puncture the input feature map by overwriting some values to zero”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the predictive-model/neural-network software-testing system disclosed by Poornaki, Vanmali, Sharif, and El-Khamy to include the software-test execution technique disclosed by Yalla. One would be motivated to do so to effectively execute selected software test cases based on neural-network-processed software data, as suggested by Yalla ([Yalla, Abstract] “process the software data, with the trained neural network model, to predict a set of test cases to execute for testing the software” and [Yalla, col. 2, lines 22-24] “cause the set of test cases to be executed in the software development platform based on the configurations, the scripts, and the test targets”). Claims 8 and 15 are analogous to claim 1, aside from claim type, and thus it would face the same rejection as above.
Regarding claim 7, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teaches The system of claim 1, (see rejection of claim 1).
Vanmali further teaches wherein the indication is a first indication, wherein the subset is a first subset,
and wherein the operations further comprise: in response to determining that a second set of inputs of the
sets of inputs to the neural network results in the failure output, storing a second indication that the first test
is dependent on a second subset of the respective tests indicated as failing by the second set of inputs.
[Vanmali, page 52, sec. 3] “the tested application itself may produce errors, which is the main reason for the testing
process. If the ANN output is correct while the output of the tested application is wrong, the evaluation of the
comparison tool is classified as being a true negative or a category of 2, i.e., the determination that the output of the
application is an actual error.” wherein the examiner interprets “the teste application itself may produce errors” to
be the same as a “first indication... first subset “and “ category of 2, i.e. the determination that the output of the
application is an actual error” to be the same as storing an indication that a failure output from a second set of
inputs is linked to a second subset of tests.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and the instant application are analogous art because they are all directed to a system for evaluating neural network outputs in relation to application performance errors.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the
invention to modify the method of claim 1 disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla
to include the “evaluation of the comparison tool is classified as being a true negative or a category of 2, i.e., the determination that the output of the application is an actual error” disclosed by Vanmali. One would be motivated to do so to efficiently improve the robustness of error detection in neural network-based systems, as suggested by Vanmali (Vanmali, [page 52, sec. 3] “the evaluation of the comparison tool is classified as being a true negative or a category of 2, i.e., the determination that the output of the application is an actual error.”)
Regarding claim 9, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teaches The method of claim 8, (claim 8 is analogous to claim 1; see claim 1). Vanmali further teaches wherein the respective weights of the neural network indicate one of the positive correlation, the negative correlation, or no correlation between the respective outputs of the respective nodes of the neural network and the output of the neural network. ([Vanmali, page 47, sec 2], “Each neuron in the network is used to perform calculations that contribute to the overall learning process, or training, of the network. The neuron interconnections are associated with synaptic weights that store the information computed during the training of the network. The neural network is thus a massive parallel information processing system that utilizes distributed control to learn and store knowledge about its environment”, wherein the examiner interprets “the synaptic weights” to be the same as “respective weights of the neural network,” as they both refer to the weights used in the neural network model and the sign of the weight can be interpreted as reflecting the direction of the relationship between an input and the output, where a positive weight represents a positive correlation and a negative weight reflects a negative correlation.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and the instant application are analogous art, because they are all directed to neural networks for evaluating test data to determine pass/fail outcomes and storing indications of test dependencies.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify method of claim 8 disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the “trained neural network is used to produce a particular output when presented with an input signal” disclosed by Vanmali. One would be motivated to do so to effectively determine test outcomes based on simulated model outputs without executing the full underlying application, as suggested by Vanmali ([Vanmali, page 47, sec 2] “The neuron interconnections are associated with synaptic weights that store the information computed during the training of the network. The neural network is thus a massive parallel information processing system that utilizes distributed control to learn and store knowledge about its environment”).
Regarding claim 10, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teaches The method of claim 8, (claim 8 is analogous to claim 1). Vanmali further teaches further comprising: after determining that the first set of inputs to the neural network results in the failure, and in response to determining that a second set of inputs of the sets of inputs, ([Vanmali, page 52, sec. 3] “the tested application itself may produce errors, which is the main reason for the testing process. If the ANN output is correct while the output of the tested application is wrong, the evaluation of the comparison tool is classified as being a true negative or a category of 2, i.e., the determination that the output of the application is an actual error.” wherein the examiner interprets the “the output of the tested application is wrong” to be the same as the step of determining a failure from the first set of neural network inputs and then providing a second set of inputs because the network was tested on an initial set then another set is determined.)
comprises a superset of the first set of inputs, determining, by the system, to omit applying the second set of inputs to the neural network. ([Vanmali, page 46, sec 1], “Third, saving an exhaustive set of test cases with the outputs of the original version may be infeasible for real-world applications.” wherein the examiner interprets “exhaustive set of test cases” to be the same as a “superset of the first set of inputs” and “saving … test cases…may be infeasible” to be the same as omit applying another set of input data due to storage constraints or for any other possible reason.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and the instant application are analogous art because they are all directed to neural network-based testing methods that determine when additional testing inputs can be omitted to reduce unnecessary execution.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify method of claim 8 disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the “saving an exhaustive set of test cases with the outputs of the original version may be infeasible for real-world applications” disclosed by Vanmali. One would be motivated to do so to efficiently conserve computational and storage resources by avoiding redundant test execution, as suggested by Vanmali ([Vanmali, page 46, sec 1] “saving an exhaustive set of test cases with the outputs of the original version may be infeasible for real-world applications.”)
Regarding claim 14, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teaches The method of claim 8, (see rejection of claim 8, analogous to claim 1).
Yalla further teaches wherein a first size of data that represents a first weight of the respective weights is smaller than a second size of word size of the processor, and wherein multiple weights of the respective weights are combined into a first word of the processor and processed in parallel. [Yalla, col 18, lines 22-30], “retraining the neural network model or the execution model based on the set of test cases and the one or more of the configurations, the scripts, or the test targets. Although FIG. 4 shows example blocks of process 400, in some implementations, process 400 may include additional blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in FIG. 4. Additionally, or alternatively, two or more of the blocks of process 400 may be performed in parallel.” wherein the examiner interprets “retraining the neural network (NN) model … based on the set of the test cases” to be the same as “first size of data that represents a first weight” since the amount of data throughput processed by the NN is limited by the network size (i.e. the weights in the input layer). The examiner further interprets “blocks of process may be performed in parallel” to be the same as “processed in parallel”.
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and the instant application are analogous art because they are all directed to methods of determining software errors using test data and neural networks.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 8 disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the process flow for running the test cases in through the NN disclosed by Yalla. One would be motivated to do so to effectively determine the best approach to train the NN model on test cases as suggested by Yalla ([Yalla, col 18, lines 22-30], “in some implementation, process may include additional blocks, fewer blocks, different blocks, “).
Regarding claim 17, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teach The non-transitory computer-readable medium of claim 15, (see rejection of claim 15).
Yalla teaches wherein a dimensionality of the output of the binary neural network is equal to one. ([Yalla, col 6, lines 32-36], “In some implementations, the neural network model may include a logistic regression model with a rectified linear unit activation for intermediate layers and a sigmoid activation for a final layer. In a neural network, an activation function is responsible for transforming a summed weighted input from a node into activation of the node or output for that input” wherein the examiner interprets “a sigmoid activation for a final layer” to be the same as “the output of the binary neural network is equal to one” because the sigmoid function of a binary neural net maps its input to a value of 0 or 1.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and the instant application are analogous art because they are all directed to an output of a neural network, e.g. a binary NN, to a have a dimensionality value of one.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of the computer-readable medium of claim 15 as disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the sigmoid activation, which would amount to a value equal to one and/or set to the value of one as further disclosed by Yalla. One would be motivated to do so to effectively output a value of a [binary] neural network to be equal to one as suggested by Yalla ([Yalla, col 6, lines 32-36], “sigmoid activation for a final layer…transforming a summed weighted input…or output for that input”).
Regarding claim 18, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teaches The non-transitory computer-readable medium of claim 15, (see rejection of claim 15).
Yalla further teaches wherein a dimensionality of the respective inputs of the binary neural network is equal to a number of the respective tests that are separate from the first test. ([Yalla, col 8, lines 55-67 and col 9 lines 1-4], “the testing platform may process the software data, with the trained neural network model, to predict a set of test cases to execute for testing the software in the software development platform. For example, a test case may include a set of test inputs, execution conditions, expected results, and/or the like developed for a particular objective. The set of test cases may include test cases to be applied to software tested by the software development platform to detect errors or faults in a software code module (e.g., program, application, script, and/or the like) before the software code module is deployed, updated, and/or the like. The set of test cases may include test cases selected from historical sets of test cases that are provided as part of the historical test configuration data, a repository of test cases that is part of the testing platform, a repository of test cases that is part of the software development system, or the like. The set of test cases may also be generated dynamically by the testing platform” wherein the examiner interprets “set of test cases” and “test cases selected from historical sets” to be the same as “respective tests”, and “The set of test cases may include test cases to be applied to software” to be the same as “dimensionality of the respective inputs” that are separate from the first test.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and the instant application are analogous art because they are all directed to respective inputs of the neural network to be equal to the number of respective tests.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of the computer-readable medium of claim 15 as disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the set of tests cases that are respective to historical sets, for which would be the inputs as further disclosed by Yalla. One would be motivated to do so to effectively have the same amount of respective tests to equal the number of inputs that are not included in the first set as suggested by Yalla ([Yalla, col 8, lines 55-67 and col 9 lines 1-4], “set of test cases to execute for testing the software in the software development platform. For example, a test case may include a set of test inputs…from historical sets of test cases that are provided as part of the historical test configuration data, a repository of test cases that is part of the testing platform, a repository of test cases that is part of the software development system, or the like”).
Regarding claim 22, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teaches The system of claim 1, (see rejection of claim 1).
Vanmali further teaches wherein the three outputs of the discrete set of three outputs are -1, 0, and 1; ([Vanmali, page 46] “A multi-layer neural network is trained on the original software application by using randomly generated test data that conform to the specification,” [Vanmali, page 58, sec 4.2] “The initial synaptic weights of the neural network were obtained randomly and covered a range between -0.5 and 0.5,” AND [Vanmali, page 51, sec 3] “The comparison tool is employed as an independent method of comparing the results from the neural network and the results of the tested versions of the credit approval application,” and Table I on page 52, wherein the examiner interprets the disclosure of synaptic weights having positive values, negative values, and values within a range (for example range said in Vanmali of -0.5 to 0.5) that includes zero to be the same as the three outputs of the discrete set of three outputs are -1, 0, and 1, as both are directed to representing three distinct correlation states between a respective output of a respective node of the neural network and the output of the neural network using signed numerical values, and wherein the examiner interprets the categorical output comparison shown in Table I to be the same as the discrete set of three outputs, as both are directed to classifying outcomes into mutually exclusive states corresponding to agreement, disagreement, or absence of correspondence between outputs.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and the instant application are analogous art because they are all directed to using neural network-based models to analyze and evaluate outcomes in order to identify failure conditions and relationships within a system.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system claim 1 disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the representation of correlation states using signed numerical values as disclosed by Vanmali, which employs neural network models with synaptic weights that take on positive values, negative values, and values within a range that includes zero. One would be motivated to do so to efficiently represent and interpret correlation states within the neural network without changing the underlying operation of the system, as suggested by Vanmali ([Vanmali, page 46] “A multi-layer neural network is trained on the original software application by using randomly generated test data that conform to the specification.”).
Regarding claim 23, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teach The system of claim 1, (see rejection of claim 1).
Sharif further teaches:
after determining that the first set of inputs to the neural network results in the failure; ([Sharif, page 2] “1, if ti failed in cycle j [the equation]”, wherein the examiner interprets “1, if ti failed in cycle j” to be the same as “determining that the first set of inputs to the neural network results in the failure” because they are both directed to a neural-network input feature indicating that a software test failed in a test cycle.)
and in response to determining that a second set of inputs of the sets of inputs comprises a superset of the first set of inputs; ([Sharif, page 2, sec 2] “Given a test suite TS = {t1, . . . , tn} as a set of test cases…the goal is to select test cases… which means to select as many test cases as possible”, wherein the examiner interprets “select test cases… which means to select as many test cases as possible” and “TS = {t1, . . . , tn}” to be the same as “a second set of inputs of the sets of inputs comprises a superset of the first set of inputs” because they are both directed to a set relationship in which a selected test set is included within a larger test suite.)
determining to omit applying the second set of inputs to the neural network; ([Sharif, page 2] “max(ES(i,j), 0) is used to discard the calculations for non-executed test cases in a given cycle (i.e., where ES(i,j) = -1).”, wherein the examiner interprets “discard the calculations for non-executed test cases” to be the same as “determining to omit applying the second set of inputs to the neural network” because they are both directed to omitting neural-network-related calculation or application for selected test-case input information.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and the instant application are analogous art because they are all directed to computer-implemented software testing, neural-network-based software test analysis, software test-case prioritization, and omitting or reducing neural-network processing for selected test-case input information.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system of claim 1 disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the set-based test selection and calculation-discarding technique disclosed by Sharif. One would be motivated to do so to efficiently avoid unnecessary neural-network calculations for non-executed test cases while prioritizing relevant software tests, as suggested by Sharif ([Sharif, page 2, sec 2] “discard the calculations for non-executed test cases”).
Regarding claim 24, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teach The system of claim 1, (see rejection of claim 1).
Sharif further teaches:
wherein the respective inputs of the sets of inputs have an upper limit of inputs of the respective inputs that indicate failed tests; ([Sharif, page 3, sec 3] “For each test case ti, DeepOrder uses 14 input features in training. The features include the test case ExecutionStatus ES(i,1), . . . , ES(i,10), and four additional features: Duration, LastRun, Distance, and ChangeInStatus”, wherein the examiner interprets “ExecutionStatus ES(i,1), . . . , ES(i,10)” to be the same as “inputs of the respective inputs that indicate failed tests” because they are both directed to execution-status input features indicating whether software tests failed, passed, or were not executed, and wherein the examiner interprets “DeepOrder uses 14 input features in training” and “ExecutionStatus ES(i,1), . . . , ES(i,10)” to be the same as “the respective inputs of the sets of inputs have an upper limit” because they are both directed to a bounded number of neural-network input features.)
wherein the upper limit is less than a number of tests in the group of tests; ([Sharif, page 1, sec 1] “Test engineers have available a set of several hundred test cases” AND [Sharif, page 3, sec 3] “For each test case ti, DeepOrder uses 14 input features in training. The features include the test case ExecutionStatus ES(i,1), . . . , ES(i,10), and four additional features: Duration, LastRun, Distance, and ChangeInStatus”, wherein the examiner interprets “DeepOrder uses 14 input features in training” and “ExecutionStatus ES(i,1), . . . , ES(i,10)” to be the same as “the upper limit” because they are both directed to a limited number of neural-network input features, and wherein the examiner interprets “a set of several hundred test cases” to be the same as “a number of tests in the group of tests” because they are both directed to the total number of tests in a software test suite.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and the instant application are analogous art because they are all directed to computer-implemented software testing, neural-network-based software test analysis, software test-case prioritization, and using a limited set of neural-network input features to process software test status information.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system of claim 1 disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the limited execution-status input features disclosed by Sharif. One would be motivated to do so to efficiently process historical test-execution status data using a bounded number of neural-network input features rather than processing an unlimited or full test-history input set, as suggested by Sharif ([Sharif, page 3, sec 3] “DeepOrder involves a multi-layer neural network . . . so that its training does not require extensive time and computing resources.”).
Claims 2 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Poornaki in view of Vanmali in view of Sharif in view of El-Khamy in view of Yalla further in view of Fong et. al., US10838848B2 “System and method for test generation” (referred herein as Fong).
Regarding claim 2, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teaches The system of claim 1, (see rejection of claim 1).
Poornaki, Vanmali, Sharif, El-Khamy, and Yalla do not teach wherein the output is a first output, and wherein the neural network comprises a number of outputs that corresponds to a number of tests of the group of tests.
Fong teaches wherein the output is a first output, and wherein the neural network comprises a number of outputs that corresponds to a number of tests of the group of tests. ([Fong, Page 19-20, co12-3, lines 60-61,62-67, 1-3] “a set of outputs based on the training received by the neural network ... the test outputs may include, for example, a test script that can be run to conduct the test described by the tester, the test script including specific actions that need to be taken and the order in which they need to be taken. In an alternate embodiment, the outputs could be compiled binaries containing object code, which when executed by a processor, cause the processor to run the actual tests and to record the outputs (e.g., storing screenshots for verification or checking for the presence of error/success codes)”,wherein the examiner interprets the “set of outputs based on the training received by the neural network... the outputs could be compiled binaries containing object code, which when executed by a processor, cause the processor to run the actual tests and to record the outputs” to be the same as a neural network that will have a number of outputs that will correspond to a group of tests.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, Fong, and the instant application are analogous art, because they are all directed to neural networks that analyze and process outputs to find errors in software based on test cases as inputs to the neural networks.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system's method of claim 1 disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the process to determine which “outputs could be compiled binaries containing object code” disclosed by Fong. One would be motivated to do so to efficiently manage and execute testcases automatically, as suggested by Fong ([Fong, Page 19-20, col2-3, lines 60-61, 62-67, 1-3]”the outputs could be compiled binaries containing object code, which when executed by a processor, cause the processor to run the actual tests and to record the outputs”).
Regarding claim 5, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teaches The system of claim 1, (see rejection of claim 1).
Poornaki, Vanmali, Sharif, El-Khamy, and Yalla does not teach wherein the neural network is a first neural network, wherein the sets of inputs are first sets of inputs, and wherein the operations further comprise: applying second sets of inputs to a second neural network to determine whether a second test is dependent on a respective second set of the second sets of inputs.
Fong teaches wherein the neural network is a first neural network, wherein the sets of inputs are first
sets of inputs, and wherein the operations further comprise: applying second sets of inputs to a second neural
network to determine whether a second test is dependent on a respective second set of the second sets of
inputs. ([Fong, Page 23, col 10, lines 36-40] “In some embodiments, the neural network can be provided the state
and the action to be taken as an input to identify one or more dependencies in the test case, and to select parameters
based at least on the identified one or more dependencies...”, wherein the examiner interprets both the instant
application and Fong use multiple neural networks, each handling different sets of inputs and determining
dependencies within test cases.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, Fong, and the instant application are analogous art, because they are all directed to
neural networks that analyze and process outputs to find errors in software based on test cases as inputs to the neural
networks.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the
invention to modify system's method of claim 1 as disclosed by Poornaki, Vanmali, Sharif, El-Khamy and Yalla to include the process to determine the process of providing the state and next action to the neural network disclosed by Fong. One would be motivated to do so to efficiently automatically select parameters, as suggested by Fong ([Fong, Page 23, co110, lines 36-40],”...and to select parameters based at least on the identified one or more dependencies”)
Claim 6 is rejected under 35 U.S.C. 103 as being un patentable over Poornaki in view of Vanmali in view
of Sharif in view of El-Khamy in view of Yalla in view of Fong further in view of Tsoukalas et. al, US10678678B1 “Ordered test execution based on test coverage” (referred herein as Tsoukalas).
Regarding claim 6, Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and Fong teaches The system of claim 5, (see rejection of claim 5).
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, Fong do not teach wherein the indication is a first indication, wherein the first sets of inputs comprises a second indication of whether the second test passes, and wherein the second sets of inputs comprises a third indication of whether the first test passes.
Tsoukalas teaches wherein the indication is a first indication, wherein the first sets of inputs comprises a second indication of whether the second test passes, and wherein the second sets of inputs comprises a third indication of whether the first test passes. ([Tsoukalas, page 13, col 3-4, lines 49-53 and 66-1] “In one embodiment, a suite of tests 180 may be determined based (at least in part) on user input. For example, a developer associated with program code 170 for a software product may supply or indicate tests that she or he deems to be relevant to the software product…various heuristics may be applied to determine whether the software product passes or fails a particular test.”, wherein the examiner interprets tests being determined based on user input and determining if a software test passes or not is the same as having indications what will determine/classify if tests passes or fails.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, Fong, Tsoukalas, and the instant application are analogous art because they are all directed to methods of determining software errors using test data and neural networks.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system’s method of claim 5 disclosed by Poornaki, Vanmali, Sharif, El-Khamy, Yalla, and Fong to include the heuristic approach to determine pass or fail of software test disclosed by Tsoukalas. One would be motivated to do so to efficiently identify which test pass or fail using user defined software tests, as suggested by Tsoukalas (Tsoukalas, [page 13, col 3-4, lines 49-53 and 66-1] “various heuristics may be applied to determine whether the software product passes or fails a particular test.”)
Claims 16 and 20 is rejected under 35 U.S.C. 103 as being un patentable over Poornaki in view of Vanmali in view
of Sharif in view of El-Khamy in view of Yalla further in view of Tsoukalas.
Regarding claim 16, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teach The non-transitory computer-readable medium of claim 15, (see rejection of claim 15).
Poornaki, Vanmali, Sharif, El-Khamy, and Yalla do not teach in response to determining that a second test of the subset of the respective tests fails, and to determining that the first test depends on the subset of the respective tests, determining to omit testing the computer code with the first test.
Tsoukalas teaches in response to determining that a second test of the subset of the respective tests fails, and to determining that the first test depends on the subset of the respective tests, determining to omit testing the computer code with the first test. ([Tsoukalas, page 13, col 3-4, lines 63-66] “Each test may be associated with an acceptable range of results, such that results outside the acceptable range may constitute a failure for that particular test. In one embodiment, various heuristics may be applied to determine whether the software product passes or fails a particular test.” AND [Tsoukalas, page 15, col 8, lines 47-52] “tests with higher failure rates may be ranked higher than tests with lower failure rates in the ordered test sequence 182, and the entire testing process may be terminated early if a test fails so that time and computing resources may be used efficiently.” AND ([Tsoukalas, col 20, lines 1-5], “…moving the particular test from the suite of tests to a suite of deprecated tests, wherein the suite of deprecated tests is excluded from consideration for the subset of tests that are likely to exercise the second set of program code…”, wherein the examiner interprets “results outside the acceptable range may constitute a failure for that particular test” and “determine whether the software product passes or fails a particular test” to be the same as “determining that a second test of the subset of the respective tests fails” because they are both directed to determining that a software test fails, and wherein the examiner interprets “the entire testing process may be terminated early if a test fails so that time and computing resources may be used efficiently” to be the same as “determining to omit testing the computer code with the first test” because they are both directed to omitting further software testing after a test failure is determined. Furthermore, the examiner interprets “moving the particular test from the suite of tests to a suite of deprecated tests” and excluding it from consideration to be the same as “determining to omit testing the computer code with the first test,” as both actions relate to excluding or omitting specific tests based on the failure of another test.)
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, Tsoukalas, and the instant application are analogous art because they are all directed to computer-implemented software testing, neural-network-based software test analysis, software test-case execution, software test-case prioritization, and omitting software tests based on software testing results.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the non-transitory computer-readable medium of claim 15 disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the early termination and omission of software tests disclosed by Tsoukalas. One would be motivated to do so to efficiently reduce the time and computing resources required for software testing after a failure is detected, as suggested by Tsoukalas ([Tsoukalas, Specification] “the entire testing process may be terminated early if a test fails so that time and computing resources may be used efficiently.”).
Regarding claim 20, Poornaki, Vanmali, Sharif, El-Khamy, and Yalla teach The non-transitory computer-readable medium of claim 15, (see rejection of claim 15).
Poornaki, Vanmali, Sharif, El-Khamy, and Yalla do not teach wherein a number inputs of the first input is equal to a number of tests of the respective tests that are separate from the first test.
Tsoukalas teaches wherein a number inputs of the first input is equal to a number of tests of the respective tests that are separate from the first test. [Tsoukalas, col 5, lines 42-49], “In one embodiment, the selected subset 181 of tests are likely to be exercised (e.g., encountered, executed, or otherwise performed) by the updated program code 171. The subset 181 of the tests may be selected based (at least in part) on the mapping 130 and on the change data associated with the updated program code 171.” wherein the examiner interprets “the selected subset of tests are likely to be exercised by the updated program code” to be the same as “number inputs of the first is equal to a number of tests of the respective tests” because in both cases the final subset of tests is determined from another set of tests.
Poornaki, Vanmali, Sharif, El-Khamy, Yalla, Tsoukalas, and the instant application are analogous art because they are all directed to methods of determining software errors using test data and neural networks.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of the computer-readable medium of claim 15 as disclosed by Poornaki, Vanmali, Sharif, El-Khamy, and Yalla to include the process of selecting a subset of test data disclosed by Tsoukalas. One would be motivated to do so to effectively eliminate unnecessary test data, as suggested by Tsoukalas (Tsoukalas, [col 5, lines 42-49] “… The subset 181 of the tests may be selected based (at least in part) on the mapping 130 and on the change data associated with the updated program code 171”).
ConclusionAny inquiry concerning this communication or earlier communications from the examiner should be directed to DEVAN KAPOOR whose telephone number is (703)756-1434. The examiner can normally be reached Monday - Friday: 9:00AM - 5:00 PM EST (times may vary).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVAN KAPOOR/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights