DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The instant application having application No. 18/469,315 filed on September 18, 2023, presents claims 1-20 for examination.
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 3, 4, 9, 10, 11, 16, 17, and 18 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
With respect to claims 2, 9, and 16, line 4 of claim 2 recites “the telemetry information.” It is unclear if this means “telemetry information associated with workloads executing on a plurality of endpoints,” as recited on lines 2-3 of claim 1, “telemetry information generated prior to deploying a first deployment to the first endpoint,” as recited on lines 7-8 of claim 1, “telemetry information generated after deploying the first deployment to the first endpoint,” as recited on lines 10-11 of claim 1, or something else. The scope of claim 2 is therefore indefinite. For purposes of compact prosecution only, Examiner has interpreted line 4 of claim 2 as reciting “the telemetry information associated with the workloads executing on the plurality of endpoints.”
Claims 9 and 16 recite limitations similar to claim 2, and each depends from a claim that recites limitations similar to claim 1. Claims 9 and 16 and are therefore also indefinite for the same reason as claim 2.
With respect to claim 3, 10, and 17, line 2 of claim 3 recites “the telemetry information.” It is unclear if this means “telemetry information associated with workloads executing on a plurality of endpoints,” as recited on lines 2-3 of claim 1, “telemetry information generated prior to deploying a first deployment to the first endpoint,” as recited on lines 7-8 of claim 1, “telemetry information generated after deploying the first deployment to the first endpoint,” as recited on lines 10-11 of claim 1, or something else. The scope of claim 3 is therefore indefinite. For purposes of compact prosecution only, Examiner has interpreted line 2 of claim 3 as reciting “the telemetry information associated with the workloads executing on the plurality of endpoints.”
Claims 10 and 17 recite limitations similar to claim 3, and each depends from a claim that recites limitations similar to claim 1. Claims 10 and 17 and are therefore also indefinite for the same reason as claim 17.
With respect to claims 4, 11, and 18, line 2 of claim 4 recites “the telemetry information.” It is unclear if this means “telemetry information associated with workloads executing on a plurality of endpoints,” as recited on lines 2-3 of claim 1, “telemetry information generated prior to deploying a first deployment to the first endpoint,” as recited on lines 7-8 of claim 1, “telemetry information generated after deploying the first deployment to the first endpoint,” as recited on lines 10-11 of claim 1, or something else. The scope of claim 4 is therefore indefinite. For purposes of compact prosecution only, Examiner has interpreted line 2 of claim 2 as reciting “the telemetry information associated with the workloads executing on the plurality of endpoints.”
Claims 11 and 18 recite limitations similar to claim 4, and each depends from a claim that recites limitations similar to claim 1. Claims 11 and 18 and are therefore also indefinite for the same reason as claim 4.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention recites a judicial exception, is directed to that judicial exception, specifically an abstract idea, as it has not been integrated into a practical application and the claims further do not recite significantly more than the judicial exception. Examiner has evaluated the claims under the framework provided in the 2019 Patent Eligibility Guidance published in the Federal Register 01/07/2019 and has provided such analysis below.
Step 1: Claims 1-7 are directed to a computer implemented method and falls within the statutory category of processes; Claims 8-14 are directed to a system and falls within the statutory category of machines; Claims 15-20 are directed to a computer-readable storage medium and falls within the statutory category of articles of manufacture1. Therefore, “Are the claims to a process, machine, manufacture or composition of matter?” Yes.
In order to evaluate the Step 2A inquiry “Is the claim directed to a law of nature, a natural phenomenon or an abstract idea?” we must determine, at Step 2A Prong 1, whether the claim recites a law of nature, a natural phenomenon, or an abstract idea (see MPEP § 2106.04).
Step 2A Prong 1:
With respect to claims 1, 8, and 15, The limitations of “A method comprising: analyzing telemetry information associated with workloads … determining, based on the telemetry information, a first endpoint of the plurality of endpoints associated with a first periodic workload; determining first values of a performance metric for the first endpoint, the first values associated with telemetry information generated prior to deploying a first deployment to the first endpoint; determining second values of the performance metric for the first endpoint, the second values associated with telemetry information generated after deploying the first deployment to the first endpoint; determining a first post-deployment impact of deploying the first deployment to the first endpoint by comparing the first values of the performance metric to the second values of the performance metric,” as claimed, is a process that, but for the recitation of generic computing components and under its broadest reasonable interpretation, covers performance of the limitation in the mind with no more than pen and paper. For example, a person could (1) analyze human-readable telemetry information for database queries to identify a database that gets sufficient use to be selected to receive a deployment for testing; (2) based on the analysis, select a database to receive the deployment; (3) analyze historical pre-deployment human-readable telemetry information for the database to determine a historical pre-deployment query success rate; (4) analyze post-deployment human-readable telemetry information to determine a post-deployment query success rate; (5) determine an impact of the deployment by comparing the historical pre-deployment query success rate with the post-deployment query success rate.
Therefore, Yes, claims 1, 8, and 15 recite limitations that fall within the “Mental Processes” grouping of abstract ideas.
As the claims have been identified as reciting a judicial exception, Step 2A Prong 2 will evaluate whether the claim as a whole integrates the recited judicial exception into a practical application (see MPEP § 2106.04(d)).
Step 2A Prong 2:
With respect to claims 1, 8, and 15, The judicial exception is not integrated into a practical application. In particular, the claims recite the following additional elements: “executing on a plurality of endpoints,” “A system, comprising: a processor; and a memory device stores program code structured to cause the processor to:,” “A computer-readable storage medium comprising computer-executable instructions that, when executed by a processor, cause the processor to,” which merely recite instructions to implement an abstract idea on a generic computer, or merely uses a generic computer or computer components to perform the abstract idea, which does not integrate a judicial exception into a practical application (see MPEP § 2106.05(f)).
The claims further recite the following additional element(s): “performing a first action based at least on the first post-deployment impact,” which is/are merely insignificant extra-solution activity such as displaying data2 that does not integrate the judicial exception into a practical application (see MPEP § 2106.05(g)), and will be analyzed further below in Step 2B as being well-understood, routine, and conventional.
Therefore, “Do the claims recite additional elements that integrate the judicial exception into a practical application? No, even when viewed in combination, these additional elements do not integrate the abstract idea into a practical application and they do not impose any meaningful limits on practicing the abstract idea.
After having evaluated the inquires set forth in Steps 2A Prong 1 and 2, it has been concluded that claims 1, 8, and 15 not only recite a judicial exception but are directed to the judicial exception as the judicial exception has not been integrated into a practical application. Accordingly, Step 2B will evaluate whether the claim as a whole amounts to significantly more than the recited exception, i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP § 2106.05.
Step 2B:
With respect to claims 1, 8, and 15, The claims do not include additional elements, alone or in combination, that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements amount to no more than generic computing components applying the abstract idea and insignificant extra-solution activity such as displaying data, which is well-understood, routine, and conventional (see MPEP § 2106.05(d)(II) for court decisions recognizing that this activity is well-understood, routine, and conventional.).
Therefore, “Do the claims recite additional elements that amount to significantly more than the judicial exception?” No, these additional elements, alone or in combination, do not amount to significantly more than the judicial exception.
Having concluded the analysis within the provided framework, claims 1, 8, and 15 do not recite patent eligible subject matter under 35 U.S.C. § 101.
With respect to claims 2, 19, and 16, the limitations recite “wherein said determining, based on the telemetry information, a first endpoint of the plurality of endpoints associated with a first periodic workload comprises: determining, based on the telemetry information, a plurality of query hashes associated with the first endpoint; determining that a first query hash of the plurality of query hashes repeats on a periodic basis with a periodicity that satisfies a predetermined relationship with a periodicity threshold; determining a variation in query data associated with queries associated with the first query hash; and determining that the variation in query data satisfies a predetermined relationship with a data variation threshold,” which is a process that, but for the recitation of generic computing components and under its broadest reasonable interpretation, covers performance of the limitations in the mind with no more than pen and paper. For example, a person could (1) calculate a query hash for SQL query text in human readable telemetry information, (2) determine that a query hash meets a periodicity threshold (e.g., query repeats every day) by counting occurrences of the query hash; (3) look at database query text to see if there are any differences between the queries and compare the different to a difference threshold. These limitations therefore also fall within the Mental Processes grouping of abstract ideas identified above. Thus, the claims are directed to the judicial exception and do not have elements amounting to significantly more than the abstract idea itself. Therefore, the claims do not recite patent eligible subject matter under 35 U.S.C. § 101.
With respect to claims 3, 10, and 17, the limitations recite “determining, based on the telemetry information, a second endpoint of the plurality of endpoints associated with a second periodic workload; determining third values of the performance metric for the second endpoint, the third values associated with telemetry information generated prior to deploying the first deployment to the second endpoint; determining fourth values of the performance metric for the second endpoint, the fourth values associated with telemetry information generated after deploying the first deployment to the second endpoint; determining a second post-deployment impact of deploying the first deployment to the second endpoint by comparing the third values of the performance metric to the fourth values of the performance metric; and determining an overall impact of deploying the first deployment to the plurality of endpoints based at least on the first post-deployment impact and the second post-deployment impact,” which is a process that, but for the recitation of generic computing components and under its broadest reasonable interpretation, covers performance of the limitations in the mind with no more than pen and paper. For example, a person could repeat the mental process described above with respect to claims 1, 8, and 15, for a second endpoint to determine a second impact of the deployment and then determine an overall impact based on the first and second impacts. These limitations therefore also fall within the Mental Processes grouping of abstract ideas identified above. Thus, the claims are directed to the judicial exception and do not have elements amounting to significantly more than the abstract idea itself. Therefore, the claims do not recite patent eligible subject matter under 35 U.S.C. § 101.
With respect to claims 4, 11, and 18, the limitations recite “determining, based on the telemetry information, a second endpoint of the plurality of endpoints associated with a second periodic workload; determining third values of the performance metric for the second endpoint, the third values associated with telemetry information generated prior to deploying a second deployment to the second endpoint, the second deployment differing from the first deployment in at least one feature; determining fourth values of the performance metric for the second endpoint, the fourth values associated with telemetry information generated after deploying the second deployment to the second endpoint; determining a second post-deployment impact of deploying the second deployment to the second endpoint by comparing the third values of the performance metric to the fourth values of the performance metric; and determining an impact of the at least one feature based at least on the first post-deployment impact and the second post-deployment impact,” which is a process that, but for the recitation of generic computing components and under its broadest reasonable interpretation, covers performance of the limitations in the mind with no more than pen and paper. For example, a person could repeat the mental process described above with respect to claims 1, 8, and 15, for a second deployment with a different feature. These limitations therefore also fall within the Mental Processes grouping of abstract ideas identified above. Thus, the claims are directed to the judicial exception and do not have elements amounting to significantly more than the abstract idea itself. Therefore, the claims do not recite patent eligible subject matter under 35 U.S.C. § 101.
With respect to claims 5, 12, and 19, the limitations recite “wherein the performance metric comprises at least one of: a query duration; a query success rate; a login success rate; a memory utilization of frontend applications associated with the first endpoint; a memory utilization of backend applications associated with the first endpoint; a processor utilization of frontend applications associated with the first endpoint; or a processor utilization of backend applications associated with the first endpoint,” which merely provides details for the performance metric that is part of the mental process identified above with respect to claims 1, 8, and 15. Furthermore, a person could determine a login success rate simply by analyzing the human-readable telemetry information for the database. These limitations therefore also fall within the Mental Processes grouping of abstract ideas identified above. Thus, the claims are directed to the judicial exception and do not have elements amounting to significantly more than the abstract idea itself. Therefore, the claims do not recite patent eligible subject matter under 35 U.S.C. § 101.
With respect to claims 6, 13, and 20, the limitations recite “wherein the first action comprises at least one of: undeploying the first deployment from the first endpoint; modifying the first deployment based at least on the first post-deployment impact; deploying the first deployment to additional endpoints of the plurality of endpoints; or generating a user interface element based at least on the first post-deployment impact.” Since only one of these actions is required by the claims and “generating a user interface element based at least on the first post-deployment impact” is insignificant extra-solution activity such as displaying data (see MPEP § 2106.05(g)) that is well-understood, routine, and conventional (see § 2106.05(d)(II) for court decisions recognizing that this activity is well-understood, routine, and conventional), the claims are directed to the judicial exception and do not have elements amounting to significantly more than the abstract idea itself. Therefore, the claims do not recite patent eligible subject matter under 35 U.S.C. § 101.
With respect to claim 7, the limitation recites “wherein the first periodic workload comprises a workload associated with a serverless structured query language (SQL) pool,” which merely provides additional details of the periodic workload that was identified above as being part of the abstract idea in claim 1, and this does not impact a person’s ability to mentally (or with pen and paper) perform the process described above with respect to claim 1. These limitations therefore also fall within the Mental Processes grouping of abstract ideas identified above. Thus, the claim is directed to the judicial exception and does not have elements amounting to significantly more than the abstract idea itself. Therefore, the claim does not recite patent eligible subject matter under 35 U.S.C. § 101.
With respect to claim 14, the limitations recite “wherein the first endpoint comprises at least one of: a service endpoint; or a private endpoint,” which merely provides additional details of the endpoint determined from the telemetry information. Furthermore, a person is capable of mentally analyzing human-readable telemetry information determine a service endpoint such as a database service. These limitations therefore also fall within the Mental Processes grouping of abstract ideas identified above. Thus, the claim is directed to the judicial exception and does not have elements amounting to significantly more than the abstract idea itself. Therefore, the claim does not recite patent eligible subject matter under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6, 8, 13, 14, 15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Demarne et al. (US 20220283793, hereinafter Demarne; see IDS dated 12/4/24) in view of Zhang et al. “Deploying a Steered Query Optimizer in Production at Microsoft” (hereinafter Zhang; see IDS dated 12/4/24).
With respect to claim 1, Demarne discloses A method comprising:
analyzing telemetry information associated with workloads executing on a plurality of endpoints (e.g., Figs. 1-7 along with associated text, e.g., [0046], In step 202, a sample set of cloud computing resources is selected for testing software based at least on telemetry data; [0053], resources 308 [plurality of endpoints] may comprise databases in an SQL database management system. Data store 110 may comprise an analytic database that may monitor and/or process telemetry data from telemetry database 312 … and store the data in telemetry dataset 112… for use in selecting a portion of resources 308 for participation in a software test … Telemetry dataset 112 may include telemetry data such as database login information … query information.);
determining, based on the telemetry information, a first endpoint of the plurality of endpoints associated with a first workload (e.g., Figs. 1-7 along with associated text, e.g., [0067], a candidate subset of the cloud computing resources [plurality of endpoints] is selected based on the … telemetry data indicating workloads that exercise relevant aspects of the new software release; [0071], after selecting the candidate subset of the cloud computing resources for testing in the new software release, selection manager 104 may be configured to utilize a method of random selection to select a sample set [first endpoint] of cloud computing resources 308 from among the candidate subset of cloud computing resources 308 for testing the new software release.);
determining first values of a performance metric for the first endpoint, the first values associated with telemetry information generated prior to deploying a first deployment to the first endpoint (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0048], Monitoring agents 310 may log any suitable telemetry information such as service measurements or metrics from resources 308 and/or underlying infrastructure or platforms; [0055], the software is deployed to the sample set of cloud computing resources and the sample set is designated as a B resource group [first endpoint]; [0059], software test metrics comparisons may cover metrics before … the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308.);
determining second values of the performance metric for the first endpoint, the second values associated with telemetry information generated after deploying the first deployment to the first endpoint (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0059], software test metrics comparisons may cover metrics … after the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308.);
determining a first post-deployment impact of deploying the first deployment to the first endpoint by comparing the first values of the performance metric to the second values of the performance metric (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0059], when the resources 308 comprise databases, the software test results may be based on whether (1) an increase in, or new, process crash signatures are observed in the B resource group resources 308, (2) specific increases in, or new, system error codes (e.g., for internal failures) are returned to the customer for the B resource group resources 308, (3) a significant negative change in resource usage is observed in the B resource group resources 308 … software test metrics comparisons may cover metrics before and after the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308.); and
performing a first action based at least on the first post-deployment impact (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0062] In step 208, in instances when results of the deployment to the B resource group are not successful, control flows to step 214. In step 214, the software deployment to the B resource group is rolled back.).
Demarne does not appear to disclose the following, which is taught in analogous art, Zhang: periodic (e.g., p. 2300, right col., 1st full para., The importance of recurring jobs. More than 60% of SCOPE jobs are recurring, i.e., periodically arriving template-scripts.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Demarne with the technique of Zhang, such that the workload is periodic, because “recurring jobs are overall interesting because we can use historical information on previous executions to improve future occurrences,” as suggested by Zhang (see p. 2300, right col., 1st full para.) and also because periodic workloads are more predictable and thus the comparison of the before and after performance metrics would be more likely to identify issues caused by the deployment itself.
With respect to claim 8, Demarne discloses A system, comprising: a processor; and a memory device stores program code structured to cause the processor to (e.g., Figs. 1 and 13 and associated text, e.g., [0099], Processor circuit 1302 may execute program code stored in a computer readable medium.):
analyze telemetry information associated with workloads executing on a plurality of endpoints (e.g., Figs. 1-7 along with associated text, e.g., [0046], In step 202, a sample set of cloud computing resources is selected for testing software based at least on telemetry data; [0053], resources 308 [plurality of endpoints] may comprise databases in an SQL database management system. Data store 110 may comprise an analytic database that may monitor and/or process telemetry data from telemetry database 312 … and store the data in telemetry dataset 112… for use in selecting a portion of resources 308 for participation in a software test … Telemetry dataset 112 may include telemetry data such as database login information … query information.);
determine, based on the telemetry information, a first endpoint of the plurality of endpoints associated with a first workload (e.g., Figs. 1-7 along with associated text, e.g., [0067], a candidate subset of the cloud computing resources [plurality of endpoints] is selected based on the … telemetry data indicating workloads that exercise relevant aspects of the new software release; [0071], after selecting the candidate subset of the cloud computing resources for testing in the new software release, selection manager 104 may be configured to utilize a method of random selection to select a sample set [first endpoint] of cloud computing resources 308 from among the candidate subset of cloud computing resources 308 for testing the new software release.);
determine first values of a performance metric for the first endpoint, the first values associated with telemetry information generated prior to deploying a first deployment to the first endpoint (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0048], Monitoring agents 310 may log any suitable telemetry information such as service measurements or metrics from resources 308 and/or underlying infrastructure or platforms; [0055], the software is deployed to the sample set of cloud computing resources and the sample set is designated as a B resource group [first endpoint]; [0059], software test metrics comparisons may cover metrics before … the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308.);
determine second values of the performance metric for the first endpoint, the second values associated with telemetry information generated after deploying the first deployment to the first endpoint (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0059], software test metrics comparisons may cover metrics … after the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308.);
determine a first post-deployment impact of deploying the first deployment to the first endpoint by comparing the first values of the performance metric to the second values of the performance metric (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0059], when the resources 308 comprise databases, the software test results may be based on whether (1) an increase in, or new, process crash signatures are observed in the B resource group resources 308, (2) specific increases in, or new, system error codes (e.g., for internal failures) are returned to the customer for the B resource group resources 308, (3) a significant negative change in resource usage is observed in the B resource group resources 308 … software test metrics comparisons may cover metrics before and after the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308.); and
perform a first action based at least on the first post-deployment impact (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0062] In step 208, in instances when results of the deployment to the B resource group are not successful, control flows to step 214. In step 214, the software deployment to the B resource group is rolled back.).
Demarne does not appear to disclose the following, which is taught in analogous art, Zhang: periodic (e.g., p. 2300, right col., 1st full para., The importance of recurring jobs. More than 60% of SCOPE jobs are recurring, i.e., periodically arriving template-scripts.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Demarne with the technique of Zhang, such that the workload is periodic, because “recurring jobs are overall interesting because we can use historical information on previous executions to improve future occurrences,” as suggested by Zhang (see p. 2300, right col., 1st full para.) and also because periodic workloads are more predictable and thus the comparison of the before and after performance metrics would be more likely to identify issues caused by the deployment itself.
With respect to claim 15, Demarne discloses A computer-readable storage medium comprising computer-executable instructions that, when executed by a processor, cause the processor to (e.g., Figs. 1 and 13 along with associated text, e.g., [0094], For example, embodiments described herein may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer readable storage medium.):
analyze telemetry information associated with workloads executing on a plurality of endpoints (e.g., Figs. 1-7 along with associated text, e.g., [0046], In step 202, a sample set of cloud computing resources is selected for testing software based at least on telemetry data; [0053], resources 308 [plurality of endpoints] may comprise databases in an SQL database management system. Data store 110 may comprise an analytic database that may monitor and/or process telemetry data from telemetry database 312 … and store the data in telemetry dataset 112… for use in selecting a portion of resources 308 for participation in a software test … Telemetry dataset 112 may include telemetry data such as database login information … query information.);
determine, based on the telemetry information, a first endpoint of the plurality of endpoints associated with a first workload (e.g., Figs. 1-7 along with associated text, e.g., [0067], a candidate subset of the cloud computing resources [plurality of endpoints] is selected based on the … telemetry data indicating workloads that exercise relevant aspects of the new software release; [0071], after selecting the candidate subset of the cloud computing resources for testing in the new software release, selection manager 104 may be configured to utilize a method of random selection to select a sample set [first endpoint] of cloud computing resources 308 from among the candidate subset of cloud computing resources 308 for testing the new software release.);
determine first values of a performance metric for the first endpoint, the first values associated with telemetry information generated prior to deploying a first deployment to the first endpoint (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0048], Monitoring agents 310 may log any suitable telemetry information such as service measurements or metrics from resources 308 and/or underlying infrastructure or platforms; [0055], the software is deployed to the sample set of cloud computing resources and the sample set is designated as a B resource group [first endpoint]; [0059], software test metrics comparisons may cover metrics before … the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308.);
determine second values of the performance metric for the first endpoint, the second values associated with telemetry information generated after deploying the first deployment to the first endpoint (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0059], software test metrics comparisons may cover metrics … after the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308.);
determine a first post-deployment impact of deploying the first deployment to the first endpoint by comparing the first values of the performance metric to the second values of the performance metric (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0059], when the resources 308 comprise databases, the software test results may be based on whether (1) an increase in, or new, process crash signatures are observed in the B resource group resources 308, (2) specific increases in, or new, system error codes (e.g., for internal failures) are returned to the customer for the B resource group resources 308, (3) a significant negative change in resource usage is observed in the B resource group resources 308 … software test metrics comparisons may cover metrics before and after the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308.); and
perform a first action based at least on the first post-deployment impact (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0062] In step 208, in instances when results of the deployment to the B resource group are not successful, control flows to step 214. In step 214, the software deployment to the B resource group is rolled back.).
Demarne does not appear to disclose the following, which is taught in analogous art, Zhang: periodic (e.g., p. 2300, right col., 1st full para., The importance of recurring jobs. More than 60% of SCOPE jobs are recurring, i.e., periodically arriving template-scripts.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Demarne with the technique of Zhang, such that the workload is periodic, because “recurring jobs are overall interesting because we can use historical information on previous executions to improve future occurrences,” as suggested by Zhang (see p. 2300, right col., 1st full para.) and also because periodic workloads are more predictable and thus the comparison of the before and after performance metrics would be more likely to identify issues caused by the deployment itself.
With respect to claims 6, 13, and 20, Demarne also discloses wherein the first action comprises at least one of: undeploying the first deployment from the first endpoint (e.g., Fig. 2 and associated text, e.g., [0062] In step 208, in instances when results of the deployment to the B resource group are not successful, control flows to step 214. In step 214, the software deployment to the B resource group is rolled back); deploying the first deployment to additional endpoints of the plurality of endpoints (e.g., Fig. 2 and associated text, e.g., [0061] In step 208, in instances when results of the software deployment to the B resource group are successful, control flows to step 212. In step 212, the software is deployed to the A resource group; [0055], The A resource group may include resources that were selected as part of the candidate subset of resources 308 but were not randomly selected as part of the sample set.); or .
With respect to claim 14, Demarne also discloses wherein the first endpoint comprises at least one of: a service endpoint (e.g., [0025], The resources may vary depending on the type of cloud computing service being offered. Some types of cloud services include infrastructure as a service (IaaS), platform as a service (Paa); [0047], Resources 308 in a PaaS type of cloud service 304 may include, for example, scalable operating systems, database management systems … Resources 308 in a SaaS type of cloud service 304 may include applications that may be subscribed to by customers 330 and delivered via the Internet.);
Claims 3, 4, 10, 11, 17, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Demarne in view of Zhang, as applied to claims 1, 8, and 15 above, and further in view of Tuffs et al. (US 20140282422, hereinafter Tuffs).
With respect to claims 3, 10, and 17, Demarne also discloses determining, based on the telemetry information, a second endpoint of the plurality of endpoints associated with a second workload (e.g., Figs. 1-7 along with associated text, e.g., [0067], a candidate subset of the cloud computing resources [plurality of endpoints] is selected based on the … telemetry data indicating workloads that exercise relevant aspects of the new software release; [0060], steps 202-206 may be repeated for testing the new software release in a different sample set of cloud computing resources [a second endpoint of the plurality of endpoints associated with a second workload] from the plurality of resources 308.);
determining third values of the performance metric for the second endpoint, the third values associated with telemetry information generated prior to deploying the first deployment to the second endpoint (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0055], the software is deployed to the sample set of cloud computing resources and the sample set is designated as a B resource group; [0059], software test metrics comparisons may cover metrics before … the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308; [0060], steps 202-206 may be repeated for testing the new software release in a different sample set of cloud computing resources.);
determining fourth values of the performance metric for the second endpoint, the fourth values associated with telemetry information generated after deploying the first deployment to the second endpoint (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0059], software test metrics comparisons may cover metrics … after the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308; [0060], steps 202-206 may be repeated for testing the new software release in a different sample set of cloud computing resources.);
determining a second post-deployment impact of deploying the first deployment to the second endpoint by comparing the third values of the performance metric to the fourth values of the performance metric (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0059], software test metrics comparisons may cover metrics before and after the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308; [0060], steps 202-206 may be repeated for testing the new software release in a different sample set of cloud computing resources.); and
Zhang further teaches periodic (e.g., p. 2300, right col., 1st full para., The importance of recurring jobs. More than 60% of SCOPE jobs are recurring, i.e., periodically arriving template-scripts.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Demarne with the technique of Zhang for the same reason set forth above.
Demarne does not appear to disclose the following, which is taught in analogous art, Tuffs: determining an overall impact of deploying the first deployment to the plurality of endpoints based at least on the first post-deployment impact and the second post-deployment impact (e.g., Figs. 3-4 and associated text, e.g., [0008], the plurality of canary instances are running the second version of the software … monitoring the plurality of software instances to collect performance data for a plurality of performance metrics … for each of the plurality of canary instances [first and second deployments], calculating a relative performance value [first and second deployment impacts] … calculating a final overall measure of performance for the second version of software, based on the relative performance values [overall impact].).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the invention of Demarne with the invention of Tuffs, such that an overall impact of the deployment is calculated, because it can address the problem that “when testing a new version of a particular software service, it can be difficult to ascertain whether any variation in the new version's performance efficiency, relative to the previous version of the service, are in fact caused by the new version of the code,” as suggested by Tuffs (see [0019]).
With respect to claims 4, 11, and 18, Demarne also discloses determining, based on the telemetry information, a second endpoint of the plurality of endpoints associated with a second workload (e.g., Figs. 1-7 along with associated text, e.g., [0067], a candidate subset of the cloud computing resources [plurality of endpoints] is selected based on the … telemetry data indicating workloads that exercise relevant aspects of the new software release; [0060], steps 202-206 may be repeated for testing the new software release in a different sample set of cloud computing resources [a second endpoint of the plurality of endpoints associated with a second workload] from the plurality of resources 308.);
determining third values of the performance metric for the second endpoint, the third values associated with telemetry information generated prior to deploying a second deployment to the second endpoint (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0055], the software is deployed to the sample set of cloud computing resources and the sample set is designated as a B resource group; [0059], software test metrics comparisons may cover metrics before … the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308; [0060], steps 202-206 may be repeated for testing the new software release in a different sample set of cloud computing resources.), ;
determining fourth values of the performance metric for the second endpoint, the fourth values associated with telemetry information generated after deploying the second deployment to the second endpoint (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0059], software test metrics comparisons may cover metrics … after the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308; [0060], steps 202-206 may be repeated for testing the new software release in a different sample set of cloud computing resources.);
determining a second post-deployment impact of deploying the second deployment to the second endpoint by comparing the third values of the performance metric to the fourth values of the performance metric (e.g., Figs. 1-3 and 10-12 along with associated text, e.g., [0059], software test metrics comparisons may cover metrics before and after the deployment to the B resource group; [0089], Monitoring in the software test may be performed by measuring behavioral differences between before and after the software was deployed to the sample set of resources 308; [0060], steps 202-206 may be repeated for testing the new software release in a different sample set of cloud computing resources.); and
.
Zhang further teaches periodic (e.g., p. 2300, right col., 1st full para., The importance of recurring jobs. More than 60% of SCOPE jobs are recurring, i.e., periodically arriving template-scripts.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Demarne with the technique of Zhang for the same reason set forth above.
Demarne does not appear to disclose the following, which is taught in analogous art, Tuffs: the second deployment differing from the first deployment in at least one feature (e.g., Figs. 3-4 and associated text, e.g., [0034], baseline instances running a first version of code are deployed [first deployment]. Canary instances running a second version of code are also deployed [second deployment] (block 315).) … determining an impact of the at least one feature based at least on the first post-deployment impact and the second post-deployment impact (e.g., Figs. 3-4 and associated text, e.g., [0036] The canary analysis component 150 also monitors a plurality of performance metrics across the plurality of baseline instances and the plurality of canary instances (block 325); [0040], The canary analysis component 150 then calculates an aggregate metric for each of the measured performance metrics, using the calculated measures of variance between the data collected for the canary instances and the aggregate baseline metrics (block 420) [based at least on the first post-deployment impact and the second post-deployment impact]; [0041] The canary analysis component 150 then calculates a final measure of performance for the version of software running on the canary instances, based on the aggregated metrics (block 425) [determining an impact of the at least one feature]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the invention of Demarne with the invention of Tuffs, such that the impact of a new feature is determined, because “Advantageously, doing so provides a measure of the variability between the performance metrics for the version of software running on the canary instances ... relative to the performance metrics for the baseline version of software … Such a value could then be used to determine whether the new version of the software has affected the software's performance,” as suggested b