DETAILED ACTION
This action is in response to the claims filed March 8. 2024. Claims 1-17 and 23-24 are pending. Claims 1, 16, and 23 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because:
- Reference character “105” has been used to designate both “User Stories and Test Case Associations” in Figure 1 and “Test Data” in Figure 2.
- Reference character “109” has been used to designate both “User Stories” in Figure 1 and “Test Data” in Figure 2.
- Reference character “123” has been used to designate both “Result” in Figure 1 and “Test Case Decision” in Figures 2 and 3.
- Reference character “125” has been used to designate both “Test Plan” in Figure 1 and “Feature Creation” in Figure 3.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 1-3, 9, 10, 13, 15-17, and 23-24 objected to because of the following informalities:
- Claims 1, 9, 10, 13, 16-17, and 23-24 contain multiple instances of element numbers. For example, claim 1 includes “calculating (701)”. Element numbers should not be included in the claims.
- Claim 2 recites “The method of Claim 1”. This should likely read “The method of claim 1”.
- Claim 3 recites “The method of Claim 2”. This should likely read “The method of claim 2”.
- Claim 15 recites “displaying the test plan comprises signalling the test plan…”. This should likely read “displaying the test plan comprises signaling the test plan”.
- Claim 16 recites “for automatically performing ranking of test cases for a test of for a release of…”. This should likely read “for automatically performing ranking of test cases for a test of a release of…”.
- Claim 17 recites “for automatically performing ranking of test cases for a test of for a release of…”. This should likely read “for automatically performing ranking of test cases for a test of a release of…”.
- Claim 17 recites “and storing program code that is executed…”. This should likely read “and stores program code that is executed”.
Claims 23 and 24 recite “a non-transitory storage medium”. This should likely read “a non-transitory computer-readable storage medium”.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-17 and 23-24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1, 16, and 23, the limitations “calculating a value representing a prioritization for a test case based on (i) at least one factor having an influence on the test case, the at least one factor obtained from data for the test case and the data including a user story, (ii) a first weight assigned to the at least one factor, and (iii) a second weight for the user story based on a defect in the user story”, “deciding a decision about an importance of the test case based on the value representing a prioritization for the at least one test case” and “ranking the test case based on the decision” as drafted, are functions that under their broadest reasonable interpretation, recite the abstract idea of a mental process. The limitation encompasses a human mind carrying out the function through observation, evaluation, judgment, and/or opinion, or even with the aid of pen and paper. The limitation “calculating a value representing a prioritization for a test case based on (i) at least one factor having an influence on the test case, the at least one factor obtained from data for the test case and the data including a user story, (ii) a first weight assigned to the at least one factor, and (iii) a second weight for the user story based on a defect in the user story” under its broadest reasonable interpretation, recites the abstract idea of a mathematical concept. Thus, these limitations recites and falls within the “Mental Processes” grouping of abstract ideas under Prong 1.
Under Prong 2, this judicial exception is not integrated into a practical application. The additional elements “A method automatically performed by a network node for ranking test cases for a test of a release of an application in a communication network, the method comprising”, “A network node (100, 500) for automatically performing ranking of test cases for a test of for a release of an application in a communication network, the network node comprising: at least one processor (503); at least one memory (505) connected to the at least one processor (503) and storing program code that is executed by the at least one processor to perform operations comprising”, and “A computer program product comprising a non-transitory storage medium including program code to be executed by processing circuitry (503) of a network node (100, 500), whereby execution of the program code causes the network node to perform operations comprising” are recited at a high level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer, and/or mere computer components. See MPEP 2106.05(f). The limitation “outputting a test plan based on the ranking of the test case” does nothing more than add the insignificant extra solution activity of merely gathering and transmitting data to the judicial exception. See MPEP 2106.05(g). Accordingly, the additional elements do not integrate the recited judicial exception into a practical application and the claim is therefore directed to the judicial exception.
Under Step 2B, the claims to not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of “A method automatically performed by a network node for ranking test cases for a test of a release of an application in a communication network, the method comprising”, “A network node (100, 500) for automatically performing ranking of test cases for a test of for a release of an application in a communication network, the network node comprising: at least one processor (503); at least one memory (505) connected to the at least one processor (503) and storing program code that is executed by the at least one processor to perform operations comprising”, and “A computer program product comprising a non-transitory storage medium including program code to be executed by processing circuitry (503) of a network node (100, 500), whereby execution of the program code causes the network node to perform operations comprising” amount to no more than mere instructions, or generic computer/computer components to carry out the exception. For the limitation “outputting a test plan based on the ranking of the test case”, the courts have identified mere data gathering and transmitting as well-understood, routine, and conventional activity. See MPEP 2106.05(d). Accordingly, the claims are not patent eligible under 35 U.S.C. 101.
Regarding claim 2, the limitation “wherein the data for the test case further comprises at least one of a defect in a release of the application, and a mapping between the test case and the user story” merely further describes the “data” of the mental step of claim 1, and therefore amounts part of a mental step. No additional limitations are recited which would amount to practical application under Prong 2, nor to significantly more under Step 2B.
Regarding claim 3, the limitation “wherein the data for the test case further comprises at least one of a release identifier for the application, a priority for the test case, a value representing a story point of the user story, a creation date of the test case, a test stub for the test case, a number of sprints for the test case, a tester for the test case, and a program code for a release of the application” merely further describes the “data” of the mental step of claim 1, and therefore amounts part of a mental step. No additional limitations are recited which would amount to practical application under Prong 2, nor to significantly more under Step 2B.
Regarding claim 4, the limitation “wherein the user story comprises at least one parameter comprising a severity of a defect for the user story, a story point for the user story, a creation date for the user story, a deployed date for the user story, and a time period for a sprint for the user story” merely further describes the “user story” of the mental step of claim 1, and therefore amounts part of a mental step. No additional limitations are recited which would amount to practical application under Prong 2, nor to significantly more under Step 2B.
Regarding claim 5 the limitation “wherein the data for the test case is in a machine readable format…” merely further describes the “the data” of the mental step of claim 1, and therefore amounts part of a mental step. The limitation “…the machine readable format obtained from a process that transformed the data in at least one natural language to the machine readable format” amounts to an additional mental step. No additional limitations are recited which would amount to practical application under Prong 2, nor to significantly more under Step 2B.
Regarding claim 6 the limitation “wherein the value representing the prioritization for the test case comprises a third weight, the third weight comprising a severity based defect distribution in the in data for the test case and a number of defects found in the data for the test case.” merely further describes the “value representing the prioritization for the test case” of the mental step of claim 1, and therefore amounts part of a mental step. No additional limitations are recited which would amount to practical application under Prong 2, nor to significantly more under Step 2B.
Regarding claim 7, the limitation “wherein the second weight for the user story based on a defect in the user story is obtained from a classifier processes…that classifies the user story based on a severity of a defect in the user story” recites an additional mental step. The limitation “comprises an artificial intelligence support vector classifier” amounts to merely applying a generic computer/computer component to the judicial exception, which does not amount to practical application under Prong 2, nor to significantly more under Step 2B, as discussed above.
Regarding claim 8, the limitation “wherein the decision comprises an analysis that assigns a value to the at least one factor” is an additional mental step. No additional limitations are recited which would amount to practical application under Prong 2, nor to significantly more under Step 2B.
Regarding claim 9, the limitation “assigning a fourth weight to the new user story that is higher than the second weight” is an additional mental step. The limitation “obtaining (801) a new user story, the new user story having no discovered defect” amounts to the insignificantly extra solution activity of mere data gathering, which does not amount to practical application under Prong 2, nor to significantly more under Step 2B, as discussed above.
Regarding claim 10, the limitation “and learning (807) from the feedback, wherein the learning comprises repeating for another test case the calculating a value representing a prioritization, the deciding a decision about an importance of the test case, and the ranking the test case based on the decision” is an additional mental step. The limitation “receiving feedback on the ranking” amounts to mere data gathering, which does not amount to practical application under Prong 2, nor to significantly more under Step 2B, as discussed above.
Regarding claim 11, the limitation “wherein the application is a control system supporting the communication network” merely further describes the “application” of the generic computer component of claim 1, thus amounting to a generic computer component, which does not amount to practical application under Prong 2, nor to significantly more under Step 2B, as discussed above.
Regarding claim 12, the limitation “wherein the communication network is a wireless network” merely further describes the “communication network” of the generic computer component of claim 1 and amounts to a generic computer component, which does not amount to practical application under Prong 2, nor to significantly more under Step 2B as discussed above.
Claim 13 does not recite additional mental steps. The limitation “wherein the outputting (707) the test plan comprises at least one of displaying the test plan and executing the test plan in an automatic way” amounts to the insignificant post solution activity of merely displaying data, which does not amount to practical application under Prong 2, nor to significantly more under Step 2B. See MPEP 2106.05(g) and 2106.05(d).
Regarding claim 14, the limitation “wherein the test plan comprises a score reflecting a confidence level in the ranking of the test case” merely further describes the “test plan” of the data transmitting step of claim 14, thus amounting to data transmission which is extra solution activity.
Claim 15 does not recite additional mental steps. The limitation “wherein the displaying the test plan comprises signalling the test plan to one of a display interface and a user via an electronic notification” amounts to mere data transmission, which does not amount to practical application under Prong 2, nor to significantly more under Step 2B as discussed above.
Regarding claims 17 and 24, the computer components, “calculate…”, “decide…” and “rank…” steps are identical to claims 16 and 23. Therefore these limitations are rejected under 35 U.S.C. 101 for the same reasons as stated in the rejection of claims 16 and 23 above. The additional limitations “wherein the at least one memory is connected to the at least one processor and storing program code that is executed by the at least one processor to perform operations according to claim 2” and “wherein execution of the program code causes the network node to perform operations according the claim 2” merely recite generic computer/computer components to perform the mental step of claim 2. Therefore, the limitation amounts to nothing more than mere application of a generic computer/computer component to a judicial exception, which does not amount to practical application under Prong 2, nor to significantly more under Step 2B, as discussed above.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 4-6, 8, 13, 15-17, and 23-24 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by US 20180046570 A1 (hereinafter “Kaulgud”).
1. (Original) A method automatically performed by a network node for ranking test cases for a test of a release of an application in a communication network, the method comprising (Fig. 1, Fig. 5, Fig. 10, Fig. 18):
- calculating (701) a value representing a prioritization for a test case based on (i) at least one factor having an influence on the test case, the at least one factor obtained from data for the test case and the data including a user story, (ii) a first weight assigned to the at least one factor, and (iii) a second weight for the user story based on a defect in the user story (Paragraph [0043], “The task prioritizer 128 may determine, based on the weightage 130 for each criterion of the criteria 126 associated with the specified release of the application 106, a task priority for each task of the plurality of tasks 120 to be applied to a different release (e.g., release (n+1)) of the application 106 [calculating a value representing a prioritization for a test case…]”; Paragraph [0104], “With respect to location “6” in FIG. 10, for determination of priority between the criteria, the task prioritizer 128 may prioritize the tasks based on the multi-criteria availability. The task level statistics (e.g., issue_weightage, popularity_weightage, crash_weightage, error_weightage, and task-usage_weightage), and the criteria priority may be used to prioritize tasks (which have associated test cases), and hence provide a task priority index (referred as task_priority_releasedata) based on the release data. The task prioritizer 128 may apply an epic_weight (i.e., organizational priority) to the task_priority_releasedata to determine the final task priority (task_priority). The issue_weightage, popularity_weightage, crash_weightage, error_weightage, and task-usage_weightage criteria may be used to determine the task priority based on release data, and feature priority may be applied to derive the final task priority. The final task priority may be used to determine the prioritized test cases which are derived from the prioritized tasks due to its associativity [based on (i) at least one factor having an influence on the test case the at least one factor obtained from data for the test case and data including a user story, (ii) a first weight assigned to the at least one factor]”; Paragraph [0082], “Referring to location “3” FIG. 5 (and location “4” of FIG. 10 discussed below), with respect to user feedback data 124, the task prioritizer 128 may identify and map the potential defects from negative reviews to tasks [L3], use-cases [L2], and features [L1]. For each negative review, the task prioritizer 128 may create a vector of keywords (i.e., an Issue_vector) describing the issue. Thus, for each negative review, the task prioritizer 128 may determine the issue_weightage and map the issue_weightage to the task. The Issue_vector may include the set of keywords mentioned in the user feedback data 124 [(iii) a second weight for the user story based on a defect in the user story]”) [Examiner’s remarks: A value representing prioritization is calculated using at least one factor (weight based on epic which is user story, weights based on release data, etc) having to do with test case and user story data which has assigned weights, and at least one factor based on user story defects (defects and issue_weights derived from negative reviews).];
- deciding (703) a decision about an importance of the test case based on the value representing a prioritization for the at least one test case (Paragraph [0043], “The task prioritizer 128 may determine, based on the weightage 130 for each criterion of the criteria 126 associated with the specified release of the application 106, a task priority for each task of the plurality of tasks 120 to be applied to a different release (e.g., release (n+1)) of the application 106. The task prioritizer 128 may apply, based on the determination of the task priority for each task of the plurality of tasks 120 to be applied to the different release of the application 106, test cases 132 associated with a highest priority task of the plurality of tasks 120 to the different release of the application 106. Further, the task prioritizer 128 may apply, based on the determination of the task priority for each task of the plurality of tasks 120 to be applied to the different release of the application 106, further test cases associated with remaining tasks of the plurality of tasks 120 in order of the task priority to the different release of the application 106”) [Examiner’s Remarks: The prioritizer may determine (decide) a priority of a plurality of tests based on weights and task priority determined in the ranking process.];
- ranking (705) the test case based on the decision (Paragraph [0087], “The task prioritizer 128 may prioritize the tasks based on the priority index (i.e., task_priority) of a task. In this regard, a task may include associated test cases, which in turn may be prioritized”) [Examiner’s remarks: The test cases are prioritized (ranked) based on the decisions of the prioritizer.]; and
- outputting (707) a test plan based on the ranking of the test case (Paragraph [0087], “The task prioritizer 128 may prioritize the tasks based on the priority index (i.e., task_priority) of a task. In this regard, a task may include associated test cases, which in turn may be prioritized”; Paragraph [0088], “Referring to location “5” of FIG. 5, the output that includes the prioritized tests for the next release of the application 106 based on current release data (i.e., the development data 112 and the production data 114) may be rendered on a user interface and/or downloaded as a test plan document”) [Examiner’s remarks: the test plan with the ranked test cases is output to a display.].
Regarding claim 2, the rejection of claim 1 is incorporated; and Kaulgud further discloses:
- wherein the data for the test case further comprises at least one of a defect in a release of the application, and a mapping between the test case and the user story (Paragraph [0077], “Referring to location “1” FIG. 5 (and location “1” of FIG. 10 discussed below), with respect to task prioritization, the development data (from the application data 200 of FIG. 2) may include the requirements (i.e., user stories, specified as use-case-1 (UC1), use-case-2 (UC2), etc.) and test cases (specified as TC1, TC2, etc.). These requirements and test cases may be modeled and stored as the data structure depicted in FIG. 6. Assuming that a requirement is captured using, for example, an agile methodology, whereas an epic (i.e., feature of the application 106) forms the Level-1 (L1) of the requirement, epic may be subsequently broken down into Level-2 (L2) which represents use-cases which are further broken into more granular levels known as tasks as Level-3 (L3). The test cases may be derived and/or written at the tasks level and map back to use-cases, and then to the feature.”) [Examiner’s remarks: Data includes a mapping between test cases and user story (use-cases).].
Regarding claim 4, the rejection of claim 1 is incorporated; and Kaulgud further discloses:
- wherein the user story comprises at least one parameter comprising a severity of a defect for the user story, a story point for the user story, a creation date for the user story, a deployed date for the user story, and a time period for a sprint for the user story (Paragraph [0082], “Referring to location “3” FIG. 5 (and location “4” of FIG. 10 discussed below), with respect to user feedback data 124, the task prioritizer 128 may identify and map the potential defects from negative reviews to tasks [L3], use-cases [L2], and features [L1]. For each negative review, the task prioritizer 128 may create a vector of keywords (i.e., an Issue_vector) describing the issue. Thus, for each negative review, the task prioritizer 128 may determine the issue_weightage and map the issue_weightage to the task. The Issue_vector may include the set of keywords mentioned in the user feedback data 124”) [Examiner’s remarks: The user story comprises “issue weightage” mapped to the task (test) which describes a severity of the defect of the user story.].
Regarding claim 5, the rejection of claim 1 is incorporated; and Kaulgud further discloses:
- wherein the data for the test case is in a machine readable format, the machine readable format obtained from a process that transformed the data in at least one natural language to the machine readable format (Paragraphs [0078] – [0079], “For each LI (epic), the task prioritizer 128 may create a vector of keywords (i.e., an Epic_vector) describing the epic. The Epic_vector may include all keywords present in the epic description. Similarly, the task prioritizer 128 may create subset vectors for use-cases and tasks to obtain Use-case_vectors and Task_vectors. The task prioritizer 128 may perform natural language processing on the stored requirements (from the development data) to identify keyword vectors. For each task and use-case, the task prioritizer 128 may retrieve the set of associated test cases, and create a vector of application screens (i.e., Task_screen_vectors) implemented for the task. The Task_screen_vectors may include all of the application screens present in the set of test cases”) [Examiner’s Remarks: Data for a test case in natural language may be converted to vectors (machine readable format).].
Regarding claim 6, the rejection of claim 1 is incorporated; and Kaulgud further discloses:
- wherein the value representing the prioritization for the test case comprises a third weight, the third weight comprising a severity based defect distribution in the in data for the test case and a number of defects found in the data for the test case (Paragraph [0082], “Based on the keyword match, the weightage may be assigned to the epic. Further, the task prioritizer 128 may traverse the graph that includes the epic, use-case, and task (e.g., see graphs of FIGS. 6-9), aggregate the weightage starting from epic, use-case, and task, and determine the issue_weightage for each task of the application”) [Examiner’s Remarks: A third weight (weightage) may be calculated by aggregating the weights of each use-case (story) and task which includes test cases, thereby creating a weight representing the distribution and number of defects of the test case.].
Regarding claim 8, the rejection of claim 1 is incorporated; and Kaulgud further discloses:
- wherein the decision comprises an analysis that assigns a value to the at least one factor (Paragraph [0080], “A user, such as an application owner, may provide the prioritization to the epics and features for determining the test strategy. The prioritization to the epics and features may be ascertained as the Epic_weight. The task prioritizer 128 may capture the feature priority based, for example, on the user input, and store the feature priority”; Paragraph [0085], “These criteria and associated weightages may be used by the task prioritizer 128 as disclosed herein with respect to determination of priority between the criteria”; Paragraph [0086], “The task prioritizer 128 may apply an epic_weight (i.e., organizational priority) to the task_priority_releasedata to determine the final task priority (task_priority)”) [Examiner’s remarks: A relative value is assigned to each factor (weight) by the task prioritizer for making a ranking decision.].
Regarding claim 13, the rejection of claim 1 is incorporated; and Kaulgud further discloses:
- wherein the outputting (707) the test plan comprises at least one of displaying the test plan and executing the test plan in an automatic way (Paragraph [0088], “Referring to location “5” of FIG. 5, the output that includes the prioritized tests for the next release of the application 106 based on current release data (i.e., the development data 112 and the production data 114) may be rendered on a user interface and/or downloaded as a test plan document”) [Examiner’s remarks: The test plan may be displayed or downloaded as a document.].
Regarding claim 15, the rejection of claim 13 is incorporated; and Kaulgud further discloses:
- wherein the displaying the test plan comprises signalling the test plan to one of a display interface and a user via an electronic notification (Paragraph [0088], “Referring to location “5” of FIG. 5, the output that includes the prioritized tests for the next release of the application 106 based on current release data (i.e., the development data 112 and the production data 114) may be rendered on a user interface and/or downloaded as a test plan document”) [Examiner’s remarks: The test plan is displayed on the user interface, which serves as a notification. Since it is displayed on an electronic screen, it must be sent via signal.].
Claim 16 is a system claim corresponding to the method claim hereinabove (claim 1). Therefore, claim 16 is rejected for the same reasons as set forth in the rejection of claim 1.
Claim 23 is a computer product claim corresponding to the method claim hereinabove (claim 1). Therefore, claim 23 is rejected for the same reasons as set forth in the rejection of claim 1.
Claim 17 combines the limitations of claim 16 and claim 2. The limitations of claim 17 are functionally identical to the combined limitations of claim 16 and claim 2. Therefore, claim 17 is rejected for the same reasons as set forth in the rejection of claim 16 and claim 2.
Claim 24 combines the limitations of claim 23 and claim 2. The limitations of claim 24 are functionally identical to the combined limitations of claim 23 and claim 2. Therefore, claim 24 is rejected for the same reasons as set forth in the rejection of claim 23 and claim 2.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3, 7, and 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over US 20180046570 A1 (hereinafter “Kaulgud”), in view of “System-Level Test Case Prioritization Using Machine Learning” by Lachmann et. al (hereinafter “Lachmann”).
Regarding claim 3, the rejection of claim 2 is incorporated; and Kaulgud does not explicitly disclose:
- wherein the data for the test case further comprises at least one of a release identifier for the application, a priority for the test case, a value representing a story point of the user story, a creation date of the test case, a test stub for the test case, a number of sprints for the test case, a tester for the test case, and a program code for a release of the application.
However, Lachmann discloses:
- wherein the data for the test case further comprises at least one of a release identifier for the application, a priority for the test case, a value representing a story point of the user story, a creation date of the test case, a test stub for the test case, a number of sprints for the test case, a tester for the test case, and a program code for a release of the application (Page 362-363, “All failures which have been found by a test case are tracked. Hence, the failure count (FC) of a test case is used as another attribute for learning. This is particularly interesting for failure retests to ensure old failures have been fixed. Moreover, for every reported failure, the failure age (FA) (i.e., date when it was found) and the assigned failure priority (FP) are recorded”) [Examiner’s remarks: Lachmann discloses data for a test case being a priority of the test case, in this case, the priority being a failure priority.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Lachmann into the teachings of Kaulgud to include “wherein the data for the test case further comprises at least one of a release identifier for the application, a priority for the test case, a value representing a story point of the user story, a creation date of the test case, a test stub for the test case, a number of sprints for the test case, a tester for the test case, and a program code for a release of the application”. As stated in Lachmann, “We employ supervised machine learning (ML) and natural language processing (NLP) to rank test cases. As a result, the failure detection rate of testing is improved by finding failures earlier. This increases testing effectiveness compared to random and manual prioritization, especially, as prioritizing test cases manually a priori is infeasible and done in an ad-hoc fashion” (Page 361). Both Lachmann and Kaulgud are in the area of test case prioritization. Using test case data as an input to a model allows relevant attributes of the test to be considered when ranking the test case using an automated model. Therefore, it would be obvious to one of ordinary skill in the art to combine automated ranking of test cases with test case data.
Regarding claim 7, the rejection of claim 1 is incorporated; and Kaulgud discloses:
- wherein the second weight for the user story based on a defect in the user story is obtained from a classifier process … that classifies the user story based on a severity of a defect in the user story (Paragraph [0082], “Referring to location “3” FIG. 5 (and location “4” of FIG. 10 discussed below), with respect to user feedback data 124, the task prioritizer 128 may identify and map the potential defects from negative reviews to tasks [L3], use-cases [L2], and features [L1]. For each negative review, the task prioritizer 128 may create a vector of keywords (i.e., an Issue_vector) describing the issue. Thus, for each negative review, the task prioritizer 128 may determine the issue_weightage and map the issue_weightage to the task. The Issue_vector may include the set of keywords mentioned in the user feedback data 124”) [Examiner’s remarks: The user story comprises “issue weightage” mapped to the task (test) which describes a severity of the defect of the user story.].
Kaulgud does not explicitly disclose:
- …obtained from a classifier process that comprises an artificial intelligence support vector classifier …
However, Lachmann discloses:
- …obtained from a classifier process that comprises an artificial intelligence support vector classifier (Page 361, “A realization of our concept using a ranked classification ML algorithm, the ranked support vector machine (SVM RANK), obtaining a ranked classification according to the priority of test cases”; Pages 362-363, “Revealed Failures: All failures which have been found by a test case are tracked. Hence, the failure count (FC) of a test case is used as another attribute for learning. This is particularly interesting for failure retests to ensure old failures have been fixed. Moreover, for every reported failure, the failure age (FA) (i.e., date when it was found) and the assigned failure priority (FP) are recorded. The FA attribute is of interest if only new failures shall be considered for a retest. We define FP on a range from 1 to 3, where 1 indicates a high severity failure that absolutely has to be fixed before release, 2 are test cases of normal importance and 3 are cosmetic failures. Each attribute is a feature of ML, the value FC representing the sum of failures revealed by test case, FA their average age and FP the sum of the failure priorities, respectively”) [Examiner’s remarks: Lachmann discloses classifying failure severity using a support vector machine. One of ordinary skill understands that the same model may be applied to the same task.]….
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Lachmann into the teachings of Kaulgud to include “obtained from a classifier process that comprises an artificial intelligence support vector classifier”. As stated in Lachmann, “We employ supervised machine learning (ML) and natural language processing (NLP) to rank test cases. As a result, the failure detection rate of testing is improved by finding failures earlier. This increases testing effectiveness compared to random and manual prioritization, especially, as prioritizing test cases manually a priori is infeasible and done in an ad-hoc fashion” (Page 361). Both Lachmann and Kaulgud are in the area of test case prioritization. Using an artificial intelligence model to perform classification allows for automation of the process and reduces the need for human intervention on large datasets, thereby saving time in the testing process. Therefore, it would be obvious to one of ordinary skill in the art to combine automated ranking of test cases with AI driven classification of data.
Regarding claim 9, the rejection of claim 1 is incorporated; and Kaulgud does not explicitly disclose:
- obtaining (801) a new user story, the new user story having no discovered defects; and assigning (803) a fourth weight to the new user story that is higher than the second weight.
However, Lachmann discloses:
- obtaining (801) a new user story, the new user story having no discovered defects; and assigning (803) a fourth weight to the new user story that is higher than the second weight (Page 364, “To cope with high dimensionality of EC and FA, we change the FA feature from the concrete number of days to a 9-point scale, based on the time passed since their detection, leading to the following values: ”today” (1), ”last week” (2), ”two weeks” (3), ”last month” (4), last ”three months” (5), within ”six months”(6) and ”last year”(7) or older than that (8). If no failures have been found, a value of 0 is given”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Lachmann into the teachings of Kaulgud to include “obtaining (801) a new user story, the new user story having no discovered defects; and assigning (803) a fourth weight to the new user story that is higher than the second weight”. As stated in Lachmann, “We employ supervised machine learning (ML) and natural language processing (NLP) to rank test cases. As a result, the failure detection rate of testing is improved by finding failures earlier. This increases testing effectiveness compared to random and manual prioritization, especially, as prioritizing test cases manually a priori is infeasible and done in an ad-hoc fashion” (Page 361). Both Lachmann and Kaulgud are in the area of test case prioritization. Assigning a weight to user stories with no defects allows its relative stability to be considered in the ranking process. Therefore, it would be obvious to one of ordinary skill in the art to combine automated ranking of test cases with consideration of historical defects.
Regarding claim 10, the rejection of claim 1 is incorporated; and Kaulgud discloses:
- …the calculating a value representing a prioritization, the deciding a decision about an importance of the test case, and the ranking the test case based on the decision (Paragraph [0043], “The task prioritizer 128 may determine, based on the weightage 130 for each criterion of the criteria 126 associated with the specified release of the application 106, a task priority for each task of the plurality of tasks 120 to be applied to a different release (e.g., release (n+1)) of the application 106. The task prioritizer 128 may apply, based on the determination of the task priority for each task of the plurality of tasks 120 to be applied to the different release of the application 106, test cases 132 associated with a highest priority task of the plurality of tasks 120 to the different release of the application 106. Further, the task prioritizer 128 may apply, based on the determination of the task priority for each task of the plurality of tasks 120 to be applied to the different release of the application 106, further test cases associated with remaining tasks of the plurality of tasks 120 in order of the task priority to the different release of the application 106”; Paragraph [0087], “The task prioritizer 128 may prioritize the tasks based on the priority index (i.e., task_priority) of a task. In this regard, a task may include associated test cases, which in turn may be prioritized”).
Kaulgud does not explicitly disclose:
- receiving (805) feedback on the ranking; and
- learning (807) from the feedback, wherein the learning comprises repeating for another test case ...
However, Lachmann discloses:
- receiving (805) feedback on the ranking (Page 362, “For evaluation, we apply k-fold cross validation [31]. Here, the training set is split in k parts, where each part is used once as validation set, whereas the other k ´1 parts are used as input for the training. The training is repeated k times, such that each fold is used for validation once”; Page 363, “The main goal of our technique is to rank test cases based on a given set of training data so that it corresponds to the intuition of a human tester creating a test set. To this end, we employ SVM RANK (cf. Sec. II-B) to learn a ranked classification model (classifier) based on given supervised training data”) [Examiner’s Remarks: One of ordinary skill in the art understands that k-fold cross validation gives feedback on the validity of a model after each iteration, and supervised learning presents human labels of training test cases which are compared to the model results during testing of the model, in this case, for test classification.]; and
- learning (807) from the feedback, wherein the learning comprises repeating for another test case (Page 362, “For evaluation, we apply k-fold cross validation [31]. Here, the training set is split in k parts, where each part is used once as validation set, whereas the other k ´1 parts are used as input for the training. The training is repeated k times, such that each fold is used for validation once”; Page 363, “The main goal of our technique is to rank test cases based on a given set of training data so that it corresponds to the intuition of a human tester creating a test set. To this end, we employ SVM RANK (cf. Sec. II-B) to learn a ranked classification model (classifier) based on given supervised training data”) [Examiner’s remarks: One of ordinary skill in the art understands that the training of an SVM model requires the model learn from the feedback obtained from a true label and the label derived from running the model, and that learning steps are completed for each data point (test case) during training. One of ordinary skill in the art may combine the steps of Kaulgud with the training of Lachmann to achieve the present invention.]...
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Lachmann into the teachings of Kaulgud to include “receiving (805) feedback on the ranking” and “learning (807) from the feedback, wherein the learning comprises repeating for another test case”. As stated in Lachmann, “We employ supervised machine learning (ML) and natural language processing (NLP) to rank test cases. As a result, the failure detection rate of testing is improved by finding failures earlier. This increases testing effectiveness compared to random and manual prioritization, especially, as prioritizing test cases manually a priori is infeasible and done in an ad-hoc fashion” (Page 361). Both Lachmann and Kaulgud are in the area of test case prioritization. Training a machine learning model allows it to more accurately predict for the desired parameters. Therefore, it would be obvious to one of ordinary skill in the art to combine automated ranking of test cases with feedback based training.
Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over US 20180046570 A1 (hereinafter “Kaulgud”), in view of US 20020162059 A1 (hereinafter “McNeely”).
Regarding claim 11, the rejection of claim 1 is incorporated; and Kaulgud does not explicitly disclose:
- wherein the application is a control system supporting the communication network.
However, McNeely discloses:
- wherein the application is a control system supporting the communication network (Paragraph [0003], “Wireless and wireline communications networks have evolved to encompass a variety of services, switching platforms, applications, data processing system hardware and equipment that are collectively referred to as network products. Before deploying such network products within a live network environment, it is important to first test these network products to ensure that their operation is as expected. Typical telecommunications environments rely upon network elements (e.g., packet switches, database nodes, routers, etc.) produced by a variety of manufacturers, and such network elements may each further include a number of operating system or application software revisions. Consequently, one of the primary objectives or goals of the testing process prior to deployment is to identify and resolve any potential problems in the provisioning of network services that may result from the diverse nature of the collection of network elements. Such problems may include, for example, compatibility problems between newly developed network elements and existing or legacy equipment/software”) [Examiner’s remarks: The application may be part of a communications network environment which supports the communication network. The application may be tested. One of ordinary skill in the art understands that the test ranking procedures of Kaulgud may be used for testing of any application, including that of communications networks.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of McNeely into the teachings of Kaulgud to include “wherein the application is a control system supporting the communication network”. As stated in McNeely, “Consequently, one of the primary objectives or goals of the testing process prior to deployment is to identify and resolve any potential problems in the provisioning of network services that may result from the diverse nature of the collection of network elements. Such problems may include, for example, compatibility problems between newly developed network elements and existing or legacy equipment/software” (Paragraph [0003]). As noted, efficient testing of software is important to ensuring the maintained compatibility of existing communications networks. Therefore, a procedure to efficiently perform software tests may be applied to the testing of communication networks software. Therefore, it would be obvious to one of ordinary skill in the art to combine test case prioritization with applications supporting networks.
Regarding claim 12, the rejection of claim 1 is incorporated; and Kaulgud does not explicitly disclose:
- wherein the communication network is a wireless network.
However, McNeely discloses:
- wherein the communication network is a wireless network (Paragraph [0003], “Wireless and wireline communications networks have evolved to encompass a variety of services, switching platforms, applications, data processing system hardware and equipment that are collectively referred to as network products. Before deploying such network products within a live network environment, it is important to first test these network products to ensure that their operation is as expected. Typical telecommunications environments rely upon network elements (e.g., packet switches, database nodes, routers, etc.) produced by a variety of manufacturers, and such network elements may each further include a number of operating system or application software revisions. Consequently, one of the primary objectives or goals of the testing process prior to deployment is to identify and resolve any potential problems in the provisioning of network services that may result from the diverse nature of the collection of network elements. Such problems may include, for example, compatibility problems between newly developed network elements and existing or legacy equipment/software”) [Examiner’s remarks: The application may be part of a wireless communications network environment which supports the communication network. The application may be tested. One of ordinary skill in the art understands that the test ranking procedures of Kaulgud may be used for testing of any application, including that of communications networks.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of McNeely into the teachings of Kaulgud to include “wherein the communication network is a wireless network”. As stated in McNeely, “Consequently, one of the primary objectives or goals of the testing process prior to deployment is to identify and resolve any potential problems in the provisioning of network services that may result from the diverse nature of the collection of network elements. Such problems may include, for example, compatibility problems between newly developed network elements and existing or legacy equipment/software” (Paragraph [0003]). As noted, efficient testing of software is important to ensuring the maintained compatibility of existing communications networks. Therefore, a procedure to efficiently perform software tests may be applied to the testing of wireless communication networks software. Therefore, it would be obvious to one of ordinary skill in the art to combine test case prioritization with applications supporting wireless communication networks.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over US 20180046570 A1 (hereinafter “Kaulgud”), in view of US 20200371902 A1 (hereinafter “Mangione-Tran”).
Regarding claim 14, Kaulgud does not explicitly disclose:
- wherein the test plan comprises a score reflecting a confidence level in the ranking of the test case.
However, Mangione-Tran discloses:
- wherein the test plan comprises a score reflecting a confidence level in the ranking of the test case (Paragraph [0059], “FIG. 7 is a flow diagram illustrating an example embodiment of a process 470 whereby the TCP server 352 uses the TCP model 360 to determine a minimum set of test cases that would provide a minimal or basic assessment of whether or not a regression has been introduced in one or more software files modified since previous software testing (e.g., a previous preflight verification). The illustrated process 470 begins with the TCP server 352 receiving, as input, a list 472 indicating one or more software files (e.g., a list of software IDs) that have been modified since the previous software testing. In certain embodiments, the TCP server 352 may also receive a confidence threshold 474, which is a numerical value (e.g., between 1 and 100) indicating how thoroughly the TCP server 352 should test the software files for regressions. For such embodiments, the TCP server 352 will select a greater number of test cases to be applied to the modified software files for a larger confidence threshold 474”) [Examiner’s remarks: The test plan calculated by the TCP model includes a confidence interval, which may be used to determine how likely the ranked test cases are to identifying a regression.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Mangione-Tran into the teachings of Kaulgud to include “wherein the test plan comprises a score reflecting a confidence level in the ranking of the test case”. As stated in Mangione-Tran, “In certain embodiments, the TCP server 352 may also receive a confidence threshold 474, which is a numerical value (e.g., between 1 and 100) indicating how thoroughly the TCP server 352 should test the software files for regressions” (Paragraph [0059]). Allowing for calculations and use of confidence intervals allows for customization of settings to ensure that no time is wasted on extra test cases, and the optimal accuracy for a given scenario is met. Therefore, it would be obvious to one of ordinary skill in the art to combine test case prioritization with use of confidence intervals.
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. - US 20190196950 A1 discloses a method of test prioritization using testing results and pattern-mining.
- US 20180260312 discloses a method of selecting tests for an application commit based on the relevant commit characteristics.
- “A Regression Test Selection Technique by Optimizing User Stories in an Agile Environment” by Chauhan discloses optimizing User stories for test ranking and selection.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIVIAN WEIJIA DUAN whose telephone number is (703)756-5442. The examiner can normally be reached Monday-Friday 8:30AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wei Y Mui can be reached at (571) 272-3708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/V.W.D./Examiner, Art Unit 2191 /WEI Y MUI/Supervisory Patent Examiner, Art Unit 2191
Prosecution Insights