Detailed
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. CN202111545266.0 , filed on 12/15/2021.
Status of Claims
Claims 1-12, 15-16,18-19,21-23 are currently amended. Claims 13, 17, 20 have been canceled. Claims 1 – 12, 14 – 16, 18 – 19, 21 – 23 are pending and examined herein.
Claims 1 – 12, 14 – 16, 18 – 19, 21 – 23 are rejected under 35 U.S.C. 103.
Response to Arguments
Applicant's arguments filed February 4th, 2026 regarding the rejections under 35 U.S.C. 103 have been fully considered but are not persuasive. Applicant argues, on pages 12-14, that the cited references fail to disclose the amended claim features arranged as recited and that there is no teaching, suggestion, or motivation to combine the references. In view of applicant’s amendments and arguments, the prior rejection has been modified. The amended limitations are taught or suggested by the combined references, and the amendments and arguments do not overcome the prior art rejection.
Claim Objections
Claims 1,5,15,18 are objected to because of the following informalities
Claim 5 recites “current iteration of performing testing”. “current iteration of performance testing” seems more appropriate.
Claims 1, 15, 18 recites “two or more different sets of parameter values being associated with two or more different ones of the application workloads that collectively meet one or more testing goals”. Correction would be suggested to clarify whether the testing goals are collectively met by the selected sets of parameter values, by the associated application workloads, or by the performance testing resulting from user of the selected sets of parameter values.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 4 – 10, 14 – 16, 18 – 19, 21 – 23 are rejected under 35 U.S.C. 103 as being unpatentable over Mahshid et al. (NPL: "Performance Testing Using a Smart Reinforcement Learning-Driven Test Agent) in view of Cady et al. (U.S. Pub. 11392315), further in view of Doni et al. (U.S. Pub. 2020/0293835).
Regarding Claim 1, Mahshid teach
An apparatus comprising: at least one processing device comprising a processor coupled to a memory; the at least one processing device being configured to perform steps of: (Pg. 2386. I. Introductions describes RELOAD: self-adaptive model-free reinforcement learning load testing agent. It is inherently executed on a computing system that works with software under test(SUT) and execution platform (pg.2388), which will include processor and memory.)
detecting a request for parameter values for a set of parameters to be utilized in performance testing of a given information technology asset type in an information technology infrastructure; (Pg. 2385 Introduction of Mahshid states “Load testing is a type of performance testing that focuses on analyzing the performance of the system when subjected to workloads.” and Algorithm 2 Learning_Episode states tuning the workload to use the modified workload on the SUT. As the algorithm is repeated through loop until meeting the test objective, algorithm detects the need to modify the workload which is parameter for load testing. The need to modify workload would be requested to meet the target SUT, such as example on pg.2389, IV. Method section)
determining, utilizing a reinforcement learning framework, the parameter values for the set of parameters to be utilized in the given iteration of the performance testing of the information technology asset based at least in part on the current state of the information technology asset; (Pg. 2388 states “In this study the research problem, i.e., generating an effective workload to meet an intended test objective” and Equation 6,7 describing the Q-learning, which is one of the approach they use for Reinforcement Learning. Pg. 2389 also states “workload that contains all the transactions with the same number of users per each transaction, then increases the number of users in fixed steps by 33% until accomplishing the test objective.” These workload values to represent load characteristics are generated based on learned Q-values and current state using equation above. )
performing the current iteration of performance testing of the given information technology asset utilizing the determined one or more changes for the two or more input-output parameters, (Algorithm 1 Adaptive Reinforcement Learning-Driven load testing contains while loop running algorithm 2 Learning Episode. These iterations are ran using the modified workload generated with Q-learning using Equation 7. )
and updating the reinforcement learning framework based at least in part on a subsequent state of the given information technology asset following the current iteration of performance testing of the given information technology asset. (Algorithm 2 shows “Detect the new state (Sn+1) of the SUT” to “Update the Q-value of the pair of previous state and taken action”.)
… the set of parameters comprising two or more input-output parameters utilized in modeling two or more different application workloads … (Pg. 2385 I. Introduction section of Mahshid states “Workload is often configured as a set of concurrent (virtual) users doing different transactions on the software under test (SUT)… Different transactions do not have the same impact on the performance, and generating an effective test workload in an optimal way is challenging.” Things like what users run the transactions and what transactions are included in workload determine the IO pattern)
one or more changes in the two or more input-output parameters (Pg. 2386 I. Introduction section of Mahshid states “The proposed reinforcement learning-driven load testing agent identifies the effects of different transactions involved in the workload and learns how to adjust the transactions to meet the test objective.” Pg. 2387 III. RELOAD test agent for optimal test workload generation section of Mahshid states “We define the actions as adjusting the load of constituent transactions in the workload, in terms of numbers of virtual users running each transaction.” The agent adjust these loads comprising multiple transactions. )
wherein performing the current iteration of performance testing of the given information technology asset comprises configuring one or more input-output testing tools utilizing the one or more changes in the two or more input- output parameters(Pg. 2385 Introduction of Mahshid states “Load testing is a type of performance testing that focuses on analyzing the performance of the system when subjected to workloads.” and Algorithm 2 Learning Episode states tuning the workload to use the modified workload on the SUT. Pg. 2386 Introduction of Mahshid staets “RELOAD uses a well-known load test actuator, i.e., Apache JMeter [23], to execute the designed workload on the SUT.” Pg. 2387 III. RELOAD test agent for optimal test workload generation section of Mahshid states “After the agent decides on an action, a test plan is generated by the agent, and then is executed on the SUT by the test actuator, i.e., Apache JMeter… We define the actions as adjusting the load of constituent transactions in the workload, in terms of numbers of virtual users running each transaction.” JMeter is a load test actuator, which is an input output testing tool configured with a test plan and used in each performance testing iteration. RL in Mahshid adjusts workload parameters and runs the modified workload. These workloads comprises different transactions and concurrent users, which forms combination of IO request patterns.)
performing two or more iterations of evaluating parameter values for the set of parameters to be utilized in the performance testing of the given information technology asset type, wherein each of the two or more iterations comprises: (Pg. 2385 Introduction of Mahshid states “Load testing is a type of performance testing that focuses on analyzing the performance of the system when subjected to workloads.” and Algorithm 2 Learning_Episode states tuning the workload to use the modified workload on the SUT. As the algorithm is repeated through loop until meeting the test objective, algorithm detects the need to modify the workload which is parameter for load testing. The need to modify workload would be requested to meet the target SUT, such as example on pg.2389, IV. Method section. Therefore, Mahshid teaches two or more iterations of performance testing in which action/parameter changes are selected, evaluated, and used to update the reinforcement learning framework.)
Mahshid does not explicitly teach that
determining a current state of a given information technology asset of the information technology asset type, the current state of the given information technology asset (i) current values of the two or more input-output parameters and (ii) current values for two or more performance metric for the given information technology asset;
the information technology asset comprising a storage system,
… configured to generate input-output operations for processing by the storage system;
to simulate a combination of input-output patterns in an input-output path of the storage system to test an effect of the one or more changes in the current values of the two or more input-output parameters on the two or more performance metrics for the given information technology asset;
selecting, based at least in part on the two or more iterations of evaluating parameter values for the set of parameters to be utilized in the performance testing of the given information technology asset type, two or more different sets of parameter values for the two or more input- output parameters, the two or more different sets of parameter values being associated with two or more different ones of the application workloads that collectively meet one or more testing goals for the given information technology asset type:
and utilizing the selected two or more different sets of parameters values for the two or more input-output parameters for performance testing of one or more additional information technology assets of the given information technology asset type in the information technology infrastructure.
However, Cady teaches that
the information technology asset comprising a storage system, (Column 1 Lines 57 – 60 of Cady states “using a Deep Reinforcement Learning (DRL) agent to automatically tune Quality of Service (QoS) settings of volumes in a distributed storage system (DSS).”)
… configured to generate input-output operations for processing by the storage system; (Column 1 Lines 29 – 34 of Cady states “a distributed storage architecture configured to service storage requests issued by one or more clients of the cluster. The storage requests are directed to data stored on storage devices coupled to one or more of the storage nodes of the cluster.” This ties the “application workloads” to I/O requests that are processed by the storage system handled by the DSS in Cady)
to simulate a combination of input-output patterns in an input-output path of the storage system (Column 6 lines 49 – 54 of Cady states “The simulated environment 122 may include one or more client systems directing input/output operations to a distributed storage system configured similarly to one to which the trained DRL agent 124 will ultimately be deployed to facilitate automated tuning of QoS settings.” Column 9-10 Lines 66 – 3 of Cady states “the DRL agent 210 may be trained (caused to learn) in a simulated environment (e.g., a distributed storage system running in a controlled environment that exposes the distributed storage system to various workload characteristics).”)
to test an effect of the one or more changes in the two or more input-output parameters on performance (Column 1- 2 Lines 63 – 7 of Cady states “A current state of the DSS is determined, including a value of the QoS setting that is currently applied to a volume of the DSS for the client, information indicative of a workload to which the DSS is exposed, and a value of a system metric indicative of a current load on the DSS as a result of the value of the QoS setting and the workload. An action is selected to be performed from among multiple predefined actions based on the current state and previous learning regarding minimizing the value of the system metric for one or more types of workloads using the QoS setting performed during iterative training on a second DSS.” Column 2 Lines 23 – 27 of Cady states “Responsive to selection of a first action, the DRL agent identifies a new value of the QoS parameter, applies the new value to the volume for the first client, and receives a reward when application of the new value lessens the value of the system metric.” Fig. 6 of Cady shows
PNG
media_image1.png
374
534
media_image1.png
Greyscale
DRL is trained in simulation with QoS settings, workload info, system metrics and learn to select actions that achieve one or more performance standard.)
of at least one of the two or more different application workloads configured to generate input-output operations for processing by the storage system; (Column 2 Lines 5 – 7 of Cady states “minimizing the value of the system metric for one or more types of workloads using the QoS setting performed during iterative training on a second DSS.” Column 6 Lines 49 – 54 of Cady states “The simulated environment 122 may include one or more client systems directing input/output operations to a distributed storage system configured similarly to one to which the trained DRL agent 124 will ultimately be deployed to facilitate automated tuning of QoS settings.”)
determining a current state of a given information technology asset of the information technology asset type, the current state of the given information technology asset (i) current values of the two or more input-output parameters and (ii) current values for two or more performance metric for the given information technology asset; (Column 2 Lines 13 - 19 of Cady states “The DRL agent is caused to select an action to be performed during a current iteration of the training from among multiple predefined actions based on the state. The state includes a value of a Quality of Service (QoS) parameter, representing a defined QoS threshold relating to usage of the DSS by a client and that is currently applied to a volume of the DSS for the client;” Column 10 Lines 4 – 15 of Cady states “The DRL agent 210 may be iteratively trained by placing the DRL agent 210 into various scenarios (e.g., states of the distributed storage system, represented by a current set of one or more QoS settings, information indicative of a current workload, and current values of a set of one or more system metrics) and causing the DRL agent to select an action (e.g., update the current client QoS settings or maintain the current client QoS settings) to be performed. Over time, the DRL agent 210 learns to select the actions that achieve one or more performance standards (e.g., minimizing one or more system metrics), for example, by applying various policies.” Column 8 Lines 22 – 29 of Cady states “System metrics may be calculated over a period of time (which may be referred to herein as a sample period), e.g., 250 milliseconds (ms), 500 ms, 1 second (s), etc. Accordingly, different values such as a min, max, standard deviation, average, etc., can be calculated for each system metric. One or more of the metrics may directly represent and/or be used to calculate a value that represents a load of the distributed storage system.” Cady maps both parts of the current state.
Doni teaches that
selecting, based at least in part on the two or more iterations of evaluating parameter values for the set of parameters to be utilized in the performance testing of the given information technology asset type, two or more different sets of parameter values for the two or more input- output parameters, the two or more different sets of parameter values being associated with two or more different ones of the application workloads that collectively meet one or more testing goals for the given information technology asset type: (Column 10 Lines 4 – 15 of Cady states “The DRL agent 210 may be iteratively trained by placing the DRL agent 210 into various scenarios (e.g., states of the distributed storage system, represented by a current set of one or more QoS settings, information indicative of a current workload, and current values of a set of one or more system metrics) and causing the DRL agent to select an action (e.g., update the current client QoS settings or maintain the current client QoS settings) to be performed. Over time, the DRL agent 210 learns to select the actions that achieve one or more performance standards (e.g., minimizing one or more system metrics), for example, by applying various policies.” Column 15 Lines 1 – 7 of Cady states “For example, the distributed storage system may initially be configured in accordance with a number of different sets of QoS settings and exposed to workload characteristics in accordance with Table 1 (below) by programmatically causing one of more clients to direct IO operations to the distributed storage system.” Column 19 Lines 24 – 34 of Cady states “Those skilled in the art will appreciate additional colunms may be added to Table 1 for other workload characteristics (e.g., I/O size, proportions ofread IOPS to write IOPS, etc.). For example, for each of the scenarios represented in Table 1, the training may again be repeated for I/O sizes of 4 KB, 8 KB, 16 KB, and 32 KB and/or for varying proportions of read IOPS to write IOPS. In alternative embodiments, the training may involve exposing the DRL agent to only those workload characteristics expected to be experienced within a particular production environment.” [0024] of Doni states “In the remainder, a candidate configuration is identified as a set of values to be applied to the parameters of the SUT which might affect the performance of such a system.” [0025] of Doni states “According to an additional embodiment, when many IT systems have been tuned, the optimized configurations are stored in a central storage (for example either a local memory storage, or a remote memory storage or a cloud storage), so that the collected knowledge can be later exploited to speed up the tuning process of novel IT systems by looking for similar application components and workloads in the previously stored history data.” [0049] of Doni states “The Knowledge Base 102 holds the information of many performance experiments or tests executed on the SUT as well as historical data gathered from similar systems. This information is used in the optimization loop to enrich the knowledge of the ML algorithm and derive more promising configurations, as will be detailed below.“ [0055] of Doni states “The analyzer module 106 also computes scalar scores representing the performance of the applied parameter vector—i.e. the vector representing the current values of parameters—which will be consequently used by the optimizer module 101 for future configurations.” It would have been obvious to select successful storage I/O parameter sets from Cady’s evaluated parameter workload combinations as Doni teaches scoring and retraining optimized configurations based on performance results and workload history.)
and utilizing the selected two or more different sets of parameters values for the two or more input-output parameters for performance testing of one or more additional information technology assets of the given information technology asset type in the information technology infrastructure. (Column 3 Lines 32 – 41 of Cady states “According to one embodiment, a DRL agent may be trained in a simulated environment replicating cluster performance and a target production environment with respect to latency and QoS. The trained DRL agent may then be deployed to one or more clusters to constantly update QoS settings in an optimal manner so as to minimize a selected measure of load on the cluster. In this manner, the DRL agent is expected to learn and adapt to new trends in volume utilization and make adjustments to QoS settings on the fly accordingly.” Column 6 lines 62 – Column 7 lines 3 of Cady states “While for sake of brevity, only a single cluster is shown in the context of the present example, it is to be appreciated that DRL agents (e.g., DRL agent 124) may be trained and deployed to multiple clusters owned by or leased by the same or different companies. Those skilled in the art will appreciate DRL agents may be trained specifically for states of a cluster expected to operate within a particular target production environment having specific workload characteristics.” [0025] of Doni states “According to an additional embodiment, when many IT systems have been tuned, the optimized configurations are stored in a central storage (for example either a local memory storage, or a remote memory storage or a cloud storage), so that the collected knowledge can be later exploited to speed up the tuning process of novel IT systems by looking for similar application components and workloads in the previously stored history data.” [0049] of Doni states “The Knowledge Base 102 holds the information of many performance experiments or tests executed on the SUT as well as historical data gathered from similar systems. This information is used in the optimization loop to enrich the knowledge of the ML algorithm and derive more promising configurations, as will be detailed below.“ It would have been obvious to use the selected or learned storage I/O parameter information for performance testing of additional storage assets of the same type as Cady teaches deployment to additional DSSs and Doni teaches reusing stored configuration knowledge for similar systems.)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Mahshid, Doni, and Cady. Mahshid teaches a reinforcement learning based performance testing process in which a test agent detects a current state, selects an action, runs a modified workload, observes a subsequent state, computes a reward, and updates learned state action information. Cady teaches applying deep reinforcement learning in a distributed storage system to tune storage QoS and I/O parameter values based on workload characteristics and system performance metrics. Doni teaches a performance testing and tuning framework in which candidate parameter configurations are applied to a system under test, a load generator running performance test, collecting performance metrics, applying parameter vectors that are scored, and optimized configurations to reuse them. One with ordinary skill in the art would be motivated to incorporate the teaching of Cady and Doni into that of Mahshid to provide an automated storage performance testing system that evaluates storage I/O parameter values across workload conditions and reuse successful configurations. It would have been predictable combination as each reference uses known performance testing or tuning techniques for improving IT system asset performance, which improves testing efficiency across multiple storage workload cases.
Regarding Claim 4, the rejection of claim 1 is incorporated herein. The combination of Mahshid, Doni, and Cady teaches
determining the one or more changes in the two or more input-output parameters to be utilized in the current iteration of the performance testing of the given information technology asset is further based at least in part on learned experience of the reinforcement learning framework, (pg. 2388, Learning Procedure section of Mahshid states “Then, in the proposed model-free RL-driven solution, the agent finds (learns) the optimal policy to generate an effective workload to accomplish the test objective through a built-in iterative policy evaluation-improvement process.” And Algorithm 1&2 shows iterative process which uses Q-learning method to keep record of learned experiences.)
the learned experience comprising characterizations of whether different sets of one or more actions that modify the two or more input-output parameters, taken from the current state of the given information technology asset, meet the one or more testing goals for the performance testing of the given information technology asset type. (Equation 6,7 of Mahshid are used to obtain the optimal policy by storing Q-values and considering the experience of RELOAD agent. Pg. 2388 Learning procedure section of Mahshid states “The optimal action-value function, Q(s, a), gives the expected long-term return, given state s, taking an arbitrary action a, and then following the optimal policy.” And “In the Q-learning algorithm, the agent learns an optimal value function, i.e., an action-value function Q (s,a), from which the optimal policy can be obtained.” Thus, learned experiences from given state with action are stored and used to decide whether optimal policy could be met. )
Regarding Claim 5, the rejection of claim 4 is incorporated herein. The combination of Mahshid, Doni, and Cady teaches
the reinforcement learning framework utilizes a reward function which assigns a reward to the one or more changes in the two or more input-output parameters utilized in the current iteration of performing testing of the given information technology asset based at least in part on whether the subsequent state of the given information technology asset following the current iteration of performance testing of the given information technology asset advances the one or more testing goals for the performance testing of the given information technology asset type. (Pg. 2387 Reward Signal section of Mahshid states “After taking the selected action and running the tuned workload, the test agent receives a reward signal which shows how effective the applied action was in leading the test agent to reaching the test objective.” And Equation 3 defines a function to determine reward signal. )
Regarding Claim 6, the rejection of claim 4 is incorporated herein. The combination of Mahshid, Doni, and Cady teaches
the one or more testing goals for the performance testing of the given information technology asset type comprise target utilization values for the two or more performance metrics. (Pg.2390 V. Results and Discussion section of Mahshid states “The test objective is reaching a performance status under which 1) the response time of the SUT exceeds 1, 500ms or 2) the error rate in the received responses exceeds 20%.” These are performance metrics to see if the testing goal was met. )
Regarding Claim 7, the rejection of claim 4 is incorporated herein. The combination of Mahshid, Doni, and Cady teaches
the request for the parameter values for the set of parameters to be utilized in the performance testing of the given information technology asset type is detected responsive to determining that a previous iteration of the performance testing of the given information technology asset did not meet the one or more testing goals for the performance testing of the given information technology asset type. (Algorithm 2 Learning_episode of Mahshid states “repeated until meeting the stopping criteria (reaching the test objective).” If the goal was not met, the learning will be repeated and the request to tune the workload for next iteration is detected responsive to it. )
Regarding Claim 8, the rejection of claim 1 is incorporated herein. The combination of Mahshid, Doni, and Cady teaches
determining the one or more changes in the current values of the two or more input-output parameters to be utilized in the current iteration of the performance testing of the given information technology asset comprises determining whether the current state of the given information technology asset matches any of a plurality of state-action records of learned experience maintained by the reinforcement learning framework, each of the plurality of state-action records specifying a given value characterizing an extent to which taking a given set of one or more actions for modifying the two or more input-output parameters from a given state of the given information technology asset meets the one or more testing goals for the performance testing of the given information technology asset type. (Pg. 2388 Learning procedure section of Mahshid describes how RL learn the optimal policy to accomplish the objective. They use “Q learning, each entry in the Q-table corresponds to a specific (state, action) pair and Q values are considered the experience learned.“ Equation 4 describes the action-value function for “The optimal action-value function, Q(s, a), gives the expected long-term return, given state s, taking an arbitrary action a, and then following the optimal policy.” Therefore, these Q-values calculated based on the state and action of the given state presents extent to whether they are close to meeting designated goal. Algorithm 2 Learning_Episode further describes the process of calculating new Q-value based on state and action and assessing whether updated Q-value based on modified workload meet the goal.)
Regarding Claim 9, the rejection of claim 8 is incorporated herein. The combination of Mahshid, Doni, and Cady teaches
responsive to determining that the current state of the given information technology asset does not match any of the plurality of state-action records, selecting a set of one or more actions for modifying the current values of the two or more input-output parameters randomly from an action space, the action space defining permissible modifications to respective ones of the two or more input-output parameters. (Pg. 2388 Learning Procedure of Mahshid shows Q-values stored in form of Q-table or a neural network to consider whether current Q-value for current state with given action is resulting the optimal policy. Mahshid also states “In model-free RL, e-greedy is a well-known method for action selection, when RL is used to find the optimal policy in a decision-making problem. It guarantees the sufficient continual exploration required for finding the optimal policy.” This value e, represent epsilon, is used to “adjusts the degree of exploration versus exploitation, as it leads the agent to select a high-value action based on the learned value function with probability (1-e) or a random possible action with probability e, given a certain state.” )
Regarding Claim 10, the rejection of claim 8 is incorporated herein. The combination of Mahshid, Doni, and Cady teaches
responsive to determining that the current state of the given information technology asset matches a given one of the plurality of state-action records: selecting, with a first probability, a first set of one or more actions specified in the given one of the plurality of state-action records matching the current state of the given information technology asset; and selecting, with a second probability, a second set of one or more actions for modifying the current values of the two or more input-output parameters randomly from an action space, the action space defining permissible modifications to respective ones of the two or more input-output parameters. (pg. 2388 Learning Procedure section and algorithm 1 of Mahshid shows that RELOAD agent uses Q-table to determine whether the current state has associated action for Q-value. Mahshid states “In e-greedy, the value of e adjusts the degree of exploration versus exploitation, as it leads the agent to select a high-value action based on the learned value function with probability (1-e) or a random possible action with probability e, given a certain state.” First probability 1-e will be used to choose the action with the highest value in known state and second probability e will be used to random choose the action.)
Regarding claim 14, the rejection of claim 1 is incorporated herein. The combination of Mahshid, Doni, and Cady teaches
the two or more input-output parameters comprise at least two of input-output size, a read/write ratio, a random/sequential ratio, and an input-output thread number. (Column 7, 54 – 59 of Cady states “Other examples of system metrics include measured or aggregated metrics such as read latency, write latency, IOPS, read IOPS, write IOPS, I/O size, write cache capacity, dedupe-ability, compressibility, total bandwidth, read bandwidth, write bandwidth, read/write ratio, workload type, data content, data type, etc.”)
Regarding claim 15, the combination of Mahshid, Doni, and Cady teaches
A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform steps of: (Fig 1 of Mahshid shows the overall picture of RELOAD, an RL-driven load testing agent. It comprises a computer program to perform recited functions when executed by such computing agent.)
The rest of claims 15 - 16 recite substantially similar subject matter as claim 1, 4 respectively, and is rejected with the same rationale, mutatis mutandis.
Regarding claim 18, the combination of Mahshid, Doni, and Cady teaches
A method comprising: (Pg. 2389 IV Method section of Mahshid shows the run through of running functions they recited in Algorithm 1,2.)
The rest of claims 18 - 19 recite substantially similar subject matter as claim 1, 4 respectively, and is rejected with the same rationale, mutatis mutandis.
Regarding claim 21, the rejection of claim 18 is incorporated herein. Regarding claims 22 – 23, the rejection of claim 21 is incorporated herein. Claims 21 - 23 recite substantially similar subject matter as claims 8 – 10 respectively, and are rejected with the same rationale, mutatis mutandis.
Claims 2 - 3 are rejected under 35 U.S.C. 103 as being unpatentable over Mahshid et al. (NPL: "Performance Testing Using a Smart Reinforcement Learning-Driven Test Agent) in view of Cady et al. (U.S. Pub. 11392315), Doni et al. (U.S. Pub. 2020/0293835), further in view of Naito (U.S. Pub. 2019/0129398).
Regarding Claim 2, the rejection of claim 1 is incorporated herein. The combination of Mahshid, Cady, and Doni teaches
the current state of the given information technology asset further comprises testing information associated with the performance testing of the given information technology asset (Fig.2 and pg.2387 State Detection section of Mahshid shows the performance state comprising testing information, such as response time and error rate. )
The combination does not explicitly teach
and configuration information for the given information technology asset.
However, Naito teach
and configuration information for the given information technology asset. ([0007] of Naito states “a state observation unit that observes test item data representing the test item, manufacturing machine operation state data representing an operation state of the manufacturing machine, and manufacturing machine specification data representing specifications of the manufacturing machine, as a state variable representing a current state of an environment;” and Fig 2. Illustrates the configuration including the learning unit which executes reinforcement learning. [0047] states “the configuration is employed in which when an environment (that is, the state s) is changed as a result of selection of the action a in the state s”. )
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Mahshid, Cady, Doni and Naito. Mahshid teaches a reinforcement learning based performance testing process in which a test agent detects a current state, selects an action, runs a modified workload, observes a subsequent state, computes a reward, and updates learned state action information. Cady teaches applying deep reinforcement learning in a distributed storage system to tune storage QoS and I/O parameter values based on workload characteristics and system performance metrics. Doni teaches a performance testing and tuning framework in which candidate parameter configurations are applied to a system under test, a load generator running performance test, collecting performance metrics, applying parameter vectors that are scored, and optimized configurations to reuse them. Naito teaches using testing information, operation state information, and hardware and software specification information as state variables in a ML based testing system. The learning agent considers the current configuration and operational context of the asset under test when selecting the next test action. One with ordinary skill in the art would be motivated to incorporate the teaching of Naito into the combination of Mahshid, Doni, and Cady so that the learning framework would consider additional context about the asset being tested when selecting parameter changes. It would have been predictable combination as adding known testing and specification information to the state representation would allow the learning framework to distinguish between different asset configurations and make better decisions on choosing parameter values.
Regarding Claim 3, the rejection of claim 2 is incorporated herein. The combination of Mahshid, Cady, Doni, and Naito teaches
the configuration information for the given information technology asset comprises at least one of a hardware configuration of the given information technology asset and a software configuration of the given information technology asset. ([0043-0044] of Naito describes configuration containing an operation state of the manufacturing machine and specifications of the manufacturing machine. Specifications of the machine could reflect hardware and software configuration of the system. )
Claims 11 - 12 are rejected under 35 U.S.C. 103 as being unpatentable over Mahshid et al. (NPL: "Performance Testing Using a Smart Reinforcement Learning-Driven Test Agent), in view of Cady et al. (U.S. Pub. 11392315), Doni et al. (U.S. Pub. 2020/0293835), further in view of Jeong et al. (U.S. Pub. 2022/0271999).
Regarding claim 11, the rejection of claim 10 is incorporated herein. The combination of Mahshid, Cady, and Doni teaches
responsive to determining that the given value specified in the given one of the plurality of state-action records matching the current state of the given information technology asset indicates that the given set of one or more actions for modifying the current values of the two or more input-output parameters will meet the one or more testing goals for the given information technology asset type(Pg. 2388 Learning procedure section describes how RL learn the optimal policy to accomplish the objective. They use “Q learning, each entry in the Q-table corresponds to a specific (state, action) pair and Q values are considered the experience learned.“ Equation 4 describes the action-value function for “The optimal action-value function, Q(s, a), gives the expected long-term return, given state s, taking an arbitrary action a, and then following the optimal policy.” Therefore, these Q-values calculated based on the state and action of the given state presents extent to whether they are close to meeting designated goal. Algorithm 2 Learning_Episode further describes the process of calculating new Q-value based on state and action and assessing whether updated Q-value based on modified workload meet the goal.)
The combination does not explicitly teach
within a threshold number of iterations of performance testing of the given information technology asset, setting the second probability to zero.
However, Jeong explicitly teach
within a threshold number of iterations of performance testing of the given information technology asset, setting the second probability to zero. ([0067] of Jeong states “decrease the value of the exploration rate when the performance as indicated by the performance indicator is between the first and second thresholds. The first and second thresholds may be set such that the performance is considered acceptable when the performance is between the first and second thresholds.” And [0068] states “adjust the value of the exploration rate such that the reinforcement learning agent takes a (more) conservative exploration strategy (or a primarily exploitative strategy) when the performance as indicated by the performance indicator is between the first and second thresholds… For example, in embodiments where the RL agent employs an epsilon greedy strategy, the exploration rate (e.g. epsilon) may be set to a value less than about 0.3, a value less than about 0.2, a value less than about 0.1 or a value less than about 0.05.” As exploration rate e decreases while performance indicator is within the acceptable threshold, it could be set to 0. )
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Mahshid, Cady, Doni and Jeong. Mahshid teaches a reinforcement learning based performance testing process in which a test agent detects a current state, selects an action, runs a modified workload, observes a subsequent state, computes a reward, and updates learned state action information. Cady teaches applying deep reinforcement learning in a distributed storage system to tune storage QoS and I/O parameter values based on workload characteristics and system performance metrics. Doni teaches a performance testing and tuning framework in which candidate parameter configurations are applied to a system under test, a load generator running performance test, collecting performance metrics, applying parameter vectors that are scored, and optimized configurations to reuse them. Jeong teaches dynamically adjusting the exploration rate in an epsilon greedy reinforcement learning process based on performance conditions, which reduces random exploration when the learned action is expected to achieve the performance objective and increase when additional parameter searching is needed to meet the performance goal. One with ordinary skill in the art would be motivated to incorporate the teaching of Jeong into the combination of Mahshid, Doni, and Cady to reduce unnecessary random parameter exploration when learned state action information indicates that the testing goal can be met, while keeping exploration available when the learned actions are not expected to satisfy the testing goal. It would have been predictable combination as adjusting the exploration probability in an epsilon greedy reinforcement learning is a known way to balance exploitation of learned actions with exploration of other possible actions to improve convergence efficiency in the iterative testing process.
Regarding claim 12, the rejection of claim 10 is incorporated herein. The combination of Mahshid, Cady, Doni, and Jeong teaches
responsive to determining that the given value specified in the given one of the plurality of state-action records matching the current state of the given information technology asset indicates that the given set of one or more actions for modifying the current values of the two or more input-output parameters will not meet the one or more designated testing goals (Pg. 2388 Learning procedure section describes how RL learn the optimal policy to accomplish the objective. They use “Q learning, each entry in the Q-table corresponds to a specific (state, action) pair and Q values are considered the experience learned.“ Equation 4 describes the action-value function for “The optimal action-value function, Q(s, a), gives the expected long-term return, given state s, taking an arbitrary action a, and then following the optimal policy.” Therefore, these Q-values calculated based on the state and action of the given state presents extent to whether they are close to meeting designated goal. Algorithm 2 Learning_Episode further describes the process of calculating new Q-value based on state and action and assessing whether updated Q-value based on modified workload meet the goal. It will further continue if goal has not been met)
within a threshold number of iterations of performance testing of the given information technology asset, setting the second probability to a non-zero value. ([0065] of Jeong states “In some embodiments, the node may be configured to increase the value of the exploration rate when the performance as indicated by the performance indicator decreases below a second threshold, the second threshold being lower than the first threshold. The second threshold may be indicative, for example of poor or low performance and may be set in a configuration phase by a user dependent on the particular communications network, network conditions and/or SLA requirements.” And [0066] states “adjust the value of the exploration rate such that the reinforcement learning agent undertakes a more aggressive exploration strategy when the performance as indicated by the performance indicator decreases below the second threshold…. For example, in embodiments where the RL agent employs an epsilon greedy strategy, the exploration rate (e.g. epsilon) may be set to a value greater than about 0.7, a value greater than about 0.8, a value greater than about 0.9 or a value greater than about 0.95.” As the epsilon increase, it can only be set to non-zero value. )
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BYUNGKWON HAN whose telephone number is (571)272-5294. The examiner can normally be reached M-F: 9:00AM-6PM PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYUNGKWON HAN/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights