DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Application filed on 6/16/2023. Claims 1-20 are pending in the case. Claims 1, 8, and 15 are independent claims.
Claim 12 is objected to because of the following informalities:
Claim 12 recites “validate” where “validating” was apparently intended.
Claim 12 recites “determine” where “determining” was apparently intended.
Claim 12 recites “deploy” where “deploying” was apparently intended.
Appropriate correction is required.
Claim Rejections - 35 U.S.C. § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. § 103 as being unpatentable over Murthy et al. (US 2023/0049611 A1, hereinafter Murthy) in view of Shaffer et al. (US 2021/0263932 A1, hereinafter Shaffer).
As to independent claim 1, Murthy teaches a system comprising:
a processor (“Computer system 500 performs specific operations by processor(s) 512 and other components by executing one or more sequences of instructions contained in system memory component 514,” paragraph 0072 lines 1-7); and
a memory comprising computer program code (“Computer system 500 performs specific operations by processor(s) 512 and other components by executing one or more sequences of instructions contained in system memory component 514,” paragraph 0072 lines 1-7), the memory and the computer program code configured to cause the processor to:
generate a validation dataset using a trained model of a first version and sampled input data as input to the trained model (“a resident software application, or bundle of applications and/or operations for ML model building, training, testing, validating, and/or deploying,” paragraph 0027 lines 25-28);
deploy the trained model of the first version [] to a stage environment (“pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” paragraph 0052 lines 2-5);
generate test output data using the deployed model of the first version in the stage environment [] (“In order to determine non-performant compute items in model upgrade 306, pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” paragraph 0052 lines 1-5);
validate the deployed model of the first version in the stage environment using the generated test output data [] (“In order to determine non-performant compute items in model upgrade 306, pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” paragraph 0052 lines 1-5);
determine that results of the validation indicate that the trained model of the first version is invalid (“If non-performant compute items are detected …,” paragraph 0053 line 1); and
perform an invalidity action associated with the trained model of the first version based on determining that the results indicate that the trained model of the first version is invalid (“If non-performant compute items are detected in model upgrade 306, … a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. These non-performant compute items may be determined from the live production traffic, and configuration service server 314 may generate exclusion or skip flags for an operation of production pool 316 to skip such non-performant compute items. The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” paragraph 0053 lines 1-17).
Murthy does not appear to expressly teach a system comprising code to:
deploy the trained model of the first version and the generated validation dataset, including the sampled input data mapped to validation output data, to a stage environment;
generate test output data using the deployed model of the first version in the stage environment and the sampled input data of the generated validation dataset as input to the deployed model of the first version; and
validate the deployed model of the first version in the stage environment using the generated test output data and the validation output of the generated validation dataset.
Shaffer teaches a system comprising code to:
deploy the trained model of the first version and the generated validation dataset, including the sampled input data mapped to validation output data, to a stage environment (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” paragraph 0029 lines 2-7);
generate test output data using the deployed model of the first version in the stage environment and the sampled input data of the generated validation dataset as input to the deployed model of the first version (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” paragraph 0029 lines 2-7); and
validate the deployed model of the first version in the stage environment using the generated test output data and the validation output of the generated validation dataset (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” paragraph 0029 lines 2-7).
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the stage environment of Murthy to comprise the validation dataset of Shaffer. (1) The Examiner finds that the prior art included each claim element listed above, although not necessarily in a single prior art reference, with the only difference between the claimed invention and the prior art being the lack of actual combination of the elements in a single prior art reference. (2) The Examiner finds that one of ordinary skill in the art could have combined the elements as claimed by known software development methods, and that in combination, each element merely performs the same function as it does separately. (3) The Examiner finds that one of ordinary skill in the art would have recognized that the results of the combination were predictable, namely validating using a sample of the input data (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 2-7). Therefore, the rationale to support a conclusion that the claim would have been obvious is that the combining prior art elements according to known methods to yield predictable results to one of ordinary skill in the art. See MPEP § 2143(I)(A).
As to dependent claim 2, the rejection of claim 1 is incorporated. Murthy/Shaffer further teaches a system:
wherein deploying the trained model of the first version and the generated validation dataset to the stage environment includes providing a model schema of the trained model of the first version to the stage environment (“The IRs and query plan graphs generated in step 502 are provided as input data to featurizer 212, which joins, featurizes, and formats the data to be used by difference model 318 and model executor 310,” Shaffer paragraph 0070 lines 3-6); and
wherein generating the test output data includes providing the sampled input data of the generated validation dataset to the deployed model of the first version according to the provided model schema (“The IRs and query plan graphs generated in step 502 are provided as input data to featurizer 212, which joins, featurizes, and formats the data to be used by difference model 318 and model executor 310,” Shaffer paragraph 0070 lines 3-6).
As to dependent claim 3, the rejection of claim 1 is incorporated. Murthy/Shaffer further teaches a system wherein the memory and the computer program code are configured to cause the processor to further:
store the trained model (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10) and the generated validation dataset (“an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 5-7) in a model registry (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10); and
wherein deploying the trained model of the first version to the stage environment includes:
determining that the trained model of the first version has not been validated (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed,” Shaffer paragraph 0029 lines 2-4) using the model registry (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10); and
deploying the trained model of the first version (“pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” Murthy paragraph 0052 lines 2-5) from the model registry (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10) to the stage environment (“pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” Murthy paragraph 0052 lines 2-5).
As to dependent claim 4, the rejection of claim 3 is incorporated. Murthy/Shaffer further teaches a system wherein the memory and the computer program code are configured to cause the processor to further:
train a second model of a second version using training data (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5);
generate a second (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) validation dataset (“an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 5-7) using the trained second model of the second version and sampled input data as input to the trained second model (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5); and
wherein at least one of training the second model or generating the second validation dataset is performed simultaneously with validating the deployed model of the first version in the stage environment (“At step 404, the ML models of the ML model package are tested,” Murthy paragraph 0063 lines 1-2).
As to dependent claim 5, the rejection of claim 1 is incorporated. Murthy/Shaffer further teaches a system wherein the memory and the computer program code are configured to cause the processor to further:
validate a second model of a second version (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) in the stage environment using an associated second validation dataset (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 2-7);
determine that results of the validation indicate that the second model of the second version is valid (“If non-performant compute items are detected in model upgrade 306, … a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. … The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” Murthy paragraph 0053 lines 1-17 – i.e. the non-flagged items are determined valid); and
deploy the second model of the second version (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) to a production environment based on determining that the results indicate that the second model of the second version is valid (“If non-performant compute items are detected in model upgrade 306, … a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. … The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” Murthy paragraph 0053 lines 1-17 – i.e. the non-flagged items are deployed to production).
As to dependent claim 6, the rejection of claim 5 is incorporated. Murthy/Shaffer further teaches a system wherein deploying the second model of the second version to the production environment includes:
initially validating the second model of the second version (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) in the production environment using the associated second validation dataset (“Production pool 316 may include … an alert/monitor 322 may be used to determine if a data source degrades for an ML model and/or variable, which may then communicate with configuration service server 314 to further flag non-performant compute items causing this degradation and/or issues with ML model execution,” Murthy paragraph 0055 lines 1-15); and
performing inference operations in the production environment using the validated second model of the second version (“a service provider system may include a production computing system and/or environment having live production traffic that executes a latest or most current version for an ML model engine and build,” Murthy paragraph 0016 lines 1-4).
As to dependent claim 7, the rejection of claim 1 is incorporated. Murthy/Shaffer further teaches a system wherein validating the deployed model of the first version in the stage environment using the generated validation dataset includes:
performing a defined quantity of inference operations with the deployed model of the first version and using the sampled input data as input to the deployed model of the first version (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 2-7);
measuring latency of the deployed model of the first version during the performance of the defined quantity of inference operations (“If non-performant compute items are detected in model upgrade 306, such as if a variable is failing or causing inconsistent or inaccurate result or if an ML model itself is failing, inaccurate, or causing timeouts …,” Murthy paragraph 0053 lines 1-4, emphasis added);
determining that the measured latency of the deployed model of the first version falls outside of a defined latency validation range (“If non-performant compute items are detected in model upgrade 306, such as if a variable is failing or causing inconsistent or inaccurate result or if an ML model itself is failing, inaccurate, or causing timeouts …,” Murthy paragraph 0053 lines 1-4, emphasis added); and
based on the determination that the measured latency falls outside the defined latency validation range, generate a validation result indicating that the deployed model of the first version is invalid (“If non-performant compute items are detected in model upgrade 306, such as if a variable is failing or causing inconsistent or inaccurate result or if an ML model itself is failing, inaccurate, or causing timeouts, a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. These non-performant compute items may be determined from the live production traffic, and configuration service server 314 may generate exclusion or skip flags for an operation of production pool 316 to skip such non-performant compute items. The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” Murthy paragraph 0053 lines 1-17, emphasis added).
As to independent claim 8, Murthy teaches a method comprising:
generating a validation dataset using a trained model of a first version and sampled input data as input to the trained model (“a resident software application, or bundle of applications and/or operations for ML model building, training, testing, validating, and/or deploying,” paragraph 0027 lines 25-28);
deploying the trained model of the first version [] to a stage environment (“pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” paragraph 0052 lines 2-5);
validating the deployed model of the first version in the stage environment [] (“In order to determine non-performant compute items in model upgrade 306, pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” paragraph 0052 lines 1-5);
determining that results of the validation indicate that the trained model of the first version is invalid (“If non-performant compute items are detected …,” paragraph 0053 line 1); and
performing an invalidity action associated with the trained model of the first version based on determining that the results indicate that the trained model of the first version is invalid (“If non-performant compute items are detected in model upgrade 306, … a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. These non-performant compute items may be determined from the live production traffic, and configuration service server 314 may generate exclusion or skip flags for an operation of production pool 316 to skip such non-performant compute items. The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” paragraph 0053 lines 1-17).
Murthy does not appear to expressly teach a method comprising:
deploy the trained model of the first version and the generated validation dataset to a stage environment; and
validate the deployed model of the first version in the stage environment using the generated validation dataset.
Shaffer teaches a method comprising:
deploy the trained model of the first version and the generated validation dataset to a stage environment (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” paragraph 0029 lines 2-7); and
validate the deployed model of the first version in the stage environment using the generated validation dataset (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” paragraph 0029 lines 2-7).
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the stage environment of Murthy to comprise the validation dataset of Shaffer. (1) The Examiner finds that the prior art included each claim element listed above, although not necessarily in a single prior art reference, with the only difference between the claimed invention and the prior art being the lack of actual combination of the elements in a single prior art reference. (2) The Examiner finds that one of ordinary skill in the art could have combined the elements as claimed by known software development methods, and that in combination, each element merely performs the same function as it does separately. (3) The Examiner finds that one of ordinary skill in the art would have recognized that the results of the combination were predictable, namely validating using a sample of the input data (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 2-7). Therefore, the rationale to support a conclusion that the claim would have been obvious is that the combining prior art elements according to known methods to yield predictable results to one of ordinary skill in the art. See MPEP § 2143(I)(A).
As to dependent claim 9, the rejection of claim 8 is incorporated. Murthy/Shaffer further teaches a method:
wherein deploying the trained model of the first version and the generated validation dataset to the stage environment includes providing a model schema of the trained model of the first version to the stage environment (“The IRs and query plan graphs generated in step 502 are provided as input data to featurizer 212, which joins, featurizes, and formats the data to be used by difference model 318 and model executor 310,” Shaffer paragraph 0070 lines 3-6); and
wherein validating the deployed model of the first version includes providing the sampled input data of the generated validation dataset to the deployed model of the first version according to the provided model schema (“The IRs and query plan graphs generated in step 502 are provided as input data to featurizer 212, which joins, featurizes, and formats the data to be used by difference model 318 and model executor 310,” Shaffer paragraph 0070 lines 3-6).
As to dependent claim 10, the rejection of claim 8 is incorporated. Murthy/Shaffer further teaches a method comprising:
storing the trained model (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10) and the generated validation dataset (“an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 5-7) in a model registry (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10); and
wherein deploying the trained model of the first version to the stage environment includes:
determining that the trained model of the first version has not been validated (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed,” Shaffer paragraph 0029 lines 2-4) using the model registry (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10); and
deploying the trained model of the first version (“pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” Murthy paragraph 0052 lines 2-5) from the model registry (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10) to the stage environment (“pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” Murthy paragraph 0052 lines 2-5).
As to dependent claim 11, the rejection of claim 10 is incorporated. Murthy/Shaffer further teaches a method comprising:
train a second model of a second version using training data (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5);
generate a second (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) validation dataset (“an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 5-7) using the trained second model of the second version and sampled input data as input to the trained second model (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5); and
wherein at least one of training the second model or generating the second validation dataset is performed simultaneously with validating the deployed model of the first version in the stage environment (“At step 404, the ML models of the ML model package are tested,” Murthy paragraph 0063 lines 1-2).
As to dependent claim 12, the rejection of claim 8 is incorporated. Murthy/Shaffer further teaches a method comprising:
validate a second model of a second version (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) in the stage environment using an associated second validation dataset (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 2-7);
determine that results of the validation indicate that the second model of the second version is valid (“If non-performant compute items are detected in model upgrade 306, … a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. … The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” Murthy paragraph 0053 lines 1-17 – i.e. the non-flagged items are determined valid); and
deploy the second model of the second version (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) to a production environment based on determining that the results indicate that the second model of the second version is valid (“If non-performant compute items are detected in model upgrade 306, … a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. … The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” Murthy paragraph 0053 lines 1-17 – i.e. the non-flagged items are deployed to production).
As to dependent claim 13, the rejection of claim 12 is incorporated. Murthy/Shaffer further teaches a method comprising:
initially validating the second model of the second version (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) in the production environment using the associated second validation dataset (“Production pool 316 may include … an alert/monitor 322 may be used to determine if a data source degrades for an ML model and/or variable, which may then communicate with configuration service server 314 to further flag non-performant compute items causing this degradation and/or issues with ML model execution,” Murthy paragraph 0055 lines 1-15); and
performing inference operations in the production environment using the validated second model of the second version (“a service provider system may include a production computing system and/or environment having live production traffic that executes a latest or most current version for an ML model engine and build,” Murthy paragraph 0016 lines 1-4).
As to dependent claim 14, the rejection of claim 8 is incorporated. Murthy/Shaffer further teaches a method wherein validating the deployed model of the first version in the stage environment using the generated validation dataset includes:
performing a defined quantity of inference operations with the deployed model of the first version and using the sampled input data as input to the deployed model of the first version (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 2-7);
measuring latency of the deployed model of the first version during the performance of the defined quantity of inference operations (“If non-performant compute items are detected in model upgrade 306, such as if a variable is failing or causing inconsistent or inaccurate result or if an ML model itself is failing, inaccurate, or causing timeouts …,” Murthy paragraph 0053 lines 1-4, emphasis added);
determining that the measured latency of the deployed model of the first version falls outside of a defined latency validation range (“If non-performant compute items are detected in model upgrade 306, such as if a variable is failing or causing inconsistent or inaccurate result or if an ML model itself is failing, inaccurate, or causing timeouts …,” Murthy paragraph 0053 lines 1-4, emphasis added); and
based on the determination that the measured latency falls outside the defined latency validation range, generate a validation result indicating that the deployed model of the first version is invalid (“If non-performant compute items are detected in model upgrade 306, such as if a variable is failing or causing inconsistent or inaccurate result or if an ML model itself is failing, inaccurate, or causing timeouts, a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. These non-performant compute items may be determined from the live production traffic, and configuration service server 314 may generate exclusion or skip flags for an operation of production pool 316 to skip such non-performant compute items. The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” Murthy paragraph 0053 lines 1-17, emphasis added).
As to independent claim 15, Murthy teaches a computer storage medium has computer-executable instructions that, upon execution by a processor, cause the processor to (“Computer system 500 performs specific operations by processor(s) 512 and other components by executing one or more sequences of instructions contained in system memory component 514,” paragraph 0072 lines 1-7) at least:
generate a validation dataset using a trained model of a first version and sampled input data as input to the trained model (“a resident software application, or bundle of applications and/or operations for ML model building, training, testing, validating, and/or deploying,” paragraph 0027 lines 25-28);
deploy the trained model of the first version[] to a stage environment (“pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” paragraph 0052 lines 2-5);
validate the deployed model of the first version in the stage environment [] (“In order to determine non-performant compute items in model upgrade 306, pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” paragraph 0052 lines 1-5);
determine that results of the validation indicate that the trained model of the first version is invalid (“If non-performant compute items are detected …,” paragraph 0053 line 1); and
perform an invalidity action associated with the trained model of the first version based on determining that the results indicate that the trained model of the first version is invalid (“If non-performant compute items are detected in model upgrade 306, … a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. These non-performant compute items may be determined from the live production traffic, and configuration service server 314 may generate exclusion or skip flags for an operation of production pool 316 to skip such non-performant compute items. The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” paragraph 0053 lines 1-17).
Murthy does not appear to expressly teach a medium comprising instructions to:
deploy the trained model of the first version, the generated validation dataset, and an associated model schema to a stage environment; and
validate the deployed model of the first version in the stage environment using the generated validation dataset.
Shaffer teaches a medium comprising instructions to:
deploy the trained model of the first version, the generated validation dataset (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” paragraph 0029 lines 2-7), and an associated model schema (“The IRs and query plan graphs generated in step 502 are provided as input data to featurizer 212, which joins, featurizes, and formats the data to be used by difference model 318 and model executor 310,” paragraph 0070 lines 3-6) to a stage environment (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” paragraph 0029 lines 2-7); and
validate the deployed model of the first version in the stage environment using the generated validation dataset (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” paragraph 0029 lines 2-7).
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the stage environment of Murthy to comprise the validation dataset of Shaffer. (1) The Examiner finds that the prior art included each claim element listed above, although not necessarily in a single prior art reference, with the only difference between the claimed invention and the prior art being the lack of actual combination of the elements in a single prior art reference. (2) The Examiner finds that one of ordinary skill in the art could have combined the elements as claimed by known software development methods, and that in combination, each element merely performs the same function as it does separately. (3) The Examiner finds that one of ordinary skill in the art would have recognized that the results of the combination were predictable, namely validating using a sample of the input data (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 2-7). Therefore, the rationale to support a conclusion that the claim would have been obvious is that the combining prior art elements according to known methods to yield predictable results to one of ordinary skill in the art. See MPEP § 2143(I)(A).
As to dependent claim 16, the rejection of claim 15 is incorporated. Murthy/Shaffer further teaches a medium wherein validating the deployed model of the first version includes validating at least one of a latency metric of the deployed model of the first version or an accuracy metric of the deployed model of the first version (“If non-performant compute items are detected in model upgrade 306, such as if a variable is failing or causing inconsistent or inaccurate result or if an ML model itself is failing, inaccurate, or causing timeouts …,” Murthy paragraph 0053 lines 1-4, emphases added) using the generated validation dataset (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 2-7):
As to dependent claim 17, the rejection of claim 15 is incorporated. Murthy/Shaffer further teaches a medium wherein the computer-executable instructions, upon execution by a processor, further cause the processor to at least:
store the trained model (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10) and the generated validation dataset (“an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 5-7) in a model registry (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10); and
wherein deploying the trained model of the first version to the stage environment includes:
determining that the trained model of the first version has not been validated (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed,” Shaffer paragraph 0029 lines 2-4) using the model registry (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10); and
deploying the trained model of the first version (“pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” Murthy paragraph 0052 lines 2-5) from the model registry (“Shared volume 326 and/or another database or data repository may further include a model catalog of model upgrade 306 having model upgrade graphs 324. Further, the model catalog may document and/or record the model upgrade, release, and versioning of the ML models and corresponding DAGs for each model, as well as changes over time to the DAGs. Thus, shared volume 326 may retain the model catalog for a versioning of the ML models in an association map directory for the catalog, which may further may to the corresponding DAGs,” Murthy paragraph 0058 lines 1-10) to the stage environment (“pre-production environment 308 may be utilized, which may provide a testing environment for a performance test of model upgrade 306 on a non-production set of machines or other computes,” Murthy paragraph 0052 lines 2-5).
As to dependent claim 18, the rejection of claim 17 is incorporated. Murthy/Shaffer further teaches a medium wherein the computer-executable instructions, upon execution by a processor, further cause the processor to at least:
train a second model of a second version using training data (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5);
generate a second (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) validation dataset (“an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 5-7) using the trained second model of the second version and sampled input data as input to the trained second model (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5); and
wherein at least one of training the second model or generating the second validation dataset is performed simultaneously with validating the deployed model of the first version in the stage environment (“At step 404, the ML models of the ML model package are tested,” Murthy paragraph 0063 lines 1-2).
As to dependent claim 19, the rejection of claim 15 is incorporated. Murthy/Shaffer further teaches a medium wherein the computer-executable instructions, upon execution by a processor, further cause the processor to at least:
validate a second model of a second version (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) in the stage environment using an associated second validation dataset (“a testing of their runtime behavior in a pre-production environment that emulates production behavior at a much smaller scale is performed, according to embodiments. That is, an experimentation pipeline is configured to select and rerun a subset of workloads which validates the accuracy of the learned/optimized models,” Shaffer paragraph 0029 lines 2-7);
determine that results of the validation indicate that the second model of the second version is valid (“If non-performant compute items are detected in model upgrade 306, … a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. … The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” Murthy paragraph 0053 lines 1-17 – i.e. the non-flagged items are determined valid); and
deploy the second model of the second version (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) to a production environment based on determining that the results indicate that the second model of the second version is valid (“If non-performant compute items are detected in model upgrade 306, … a configuration service server 314 may interface with pre-production environment 308 to generate flags, such as exclusion flags and/or an exclusion list, of the non-performant compute items in model upgrade 306. … The flags may also delay or prevent deployment of the ML models and/or non-performant compute items (e.g., by deploying other ML models in model upgrade 306 but holding back or skipping deployment or execution of the non-performant compute items and/or ML model),” Murthy paragraph 0053 lines 1-17 – i.e. the non-flagged items are deployed to production).
As to dependent claim 20, the rejection of claim 19 is incorporated. Murthy/Shaffer further teaches a medium wherein deploying the second model of the second version to the production environment includes:
initially validating the second model of the second version (“The ML model package may include multiple ML models, each of which is for a new deployment or ML model upgrade, such as to a new version of the ML model,” Murthy paragraph 0062 lines 2-5) in the production environment using the associated second validation dataset (“Production pool 316 may include … an alert/monitor 322 may be used to determine if a data source degrades for an ML model and/or variable, which may then communicate with configuration service server 314 to further flag non-performant compute items causing this degradation and/or issues with ML model execution,” Murthy paragraph 0055 lines 1-15); and
performing inference operations in the production environment using the validated second model of the second version (“a service provider system may include a production computing system and/or environment having live production traffic that executes a latest or most current version for an ML model engine and build,” Murthy paragraph 0016 lines 1-4).
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant’s disclosure:
US 2020/0401491 A1 disclosing validating machine learning in a stage environment
Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
In the interests of compact prosecution, Applicant is invited to contact the examiner via electronic media pursuant to USPTO policy outlined MPEP § 502.03. All electronic communication must be authorized in writing. Applicant may wish to file an Internet Communications Authorization Form PTO/SB/439. Applicant may wish to request an interview using the Interview Practice website: http://www.uspto.gov/patent/laws-and-regulations/interview-practice.
Applicant is reminded Internet e-mail may not be used for communication for matters under 35 U.S.C. § 132 or which otherwise require a signature. A reply to an Office action may NOT be communicated by Applicant to the USPTO via Internet e-mail. If such a reply is submitted by Applicant via Internet e-mail, a paper copy will be placed in the appropriate patent application file with an indication that the reply is NOT ENTERED. See MPEP § 502.03(II).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ryan Barrett whose telephone number is 571 270 3311. The examiner can normally be reached 9:00am to 5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Michelle Bechtold can be reached at 571 431 0762. The fax phone number for the organization where this application or proceeding is assigned is 571 273 8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ryan Barrett/
Primary Examiner, Art Unit 2148
Prosecution Insights