DETAILED ACTION
This Action is a response to the filing received 20 June 2024. Claims 1-20 are presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-10 and 12-20 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Dasgupta et al., U.S. 10,719,301 B1 (“Dasgupta”).
Regarding claim 1, Dasgupta teaches: A computer implemented method for evaluating module performance under various conditions based on obtained data (Dasgupta, e.g., 4:20-26, “MDE provides a model experimentation interface … tests the model to determine its performance … model diagnosis interface to present the model’s performance metrics …”), the method comprising:
obtaining, by one or more processors supporting a development environment (Dasgupta, e.g., 4:10-13, “model development environment (MDE) … implements an interactive and iterative workflow to allow users to develop models through repeated model experiences … provides a number of graphical user interfaces that greatly simplify model development tasks and automates steps in the model development process …”), an indication of a module under evaluation (Dasgupta, e.g., 4:20-23, “MDE provides a model experimentation interface that allows users to configure and run model experiences, which performs a training run of a model and then tests the model to determine its performance …”);
configuring, by the one or more processors, a customer environment to provide an evaluation compute, separate from a customer compute executing customer-directed jobs, for executing the module under evaluation (Dasgupta, e.g., 10:10-16, “model may be tested 165 using a model tester 170 … using one or more other computing nodes in the MDE service …” See also, e.g., 10:26-35, “MLS (machine learning service) … provided by the service provider network … to perform discrete jobs specified by a client … MDE 130 may programmatically submit jobs to the MLS using an orchestrator …”);
deploying, by the one or more processors, the module under evaluation in the customer environment, wherein deploying the module under evaluation causes the customer environment to execute the module under test using the evaluation compute (Dasgupta, e.g., 10:26-48, “MLS may be used to perform discrete jobs specified by a client … model experiment interface 144 may allow the user to specify a variety of model experiment parameters, and then launch a model experiment … select a model for the experiment … one or more data sets to use for the experiment … one or more validation runs for the model …”);
configuring, by the one or more processors, the evaluation compute to operate under predetermined test conditions based on a script associated with the module under evaluation (Dasgupta, e.g., 10:36-48, “model experiment interface 144 may allow the user to specify a variety of model experiment parameters, and then launch a model experiment … interface may allow a user to … select one or more data sets to use for the experiment …” See also, e.g., 10:60-11:10, “MDE 130 may implement an auditing process for production models, that may periodically test production model against real or artificial input data … once an experiment has been launched, the model experiment interface 144 may allow users to view different properties of a running job …”); and
obtaining, by the one or more processors, an evaluation metric from the customer environment, wherein the evaluation metric is representative of an execution of the module under evaluation based on the predetermined test conditions (Dasgupta, e.g., 11:38-49, “MDE 130 may provide a model diagnosis interface 148 … used to generate a body of performance metrics from the model performance in the repository 168 …” See also, e.g., 10:13-25, “model may be tested 165 using a model tester 170 … implemented using one or more other computing nodes in the MDE service … test generates test results, which are saved as model performance metrics to be analyzed … performance metrics may be stored in a model perform metrics repository 168 …”).
Claim 12 is rejected for the reasons given in the rejection of claim 1 above. Examiner notes that with respect to claim 12, Dasgupta further teaches: A system configured for evaluating module performance under various conditions based on obtained data, the system comprising: one or more processors supporting a development environment; and a memory storing machine-readable instructions that, when executed (Dasgupta, e.g., 60:52-62, “example computer system that can be used to one or more portions of an MDE that allows users to develop models through iterative model experiments … computer system 3000 includes one or more processors 3010 coupled to a system memory 3020 …” See also, e.g., 61:7-24, “System memory 3020 may be configured to store instructions and data accessible by processor(s) 3010 … program instructions and data implementing one or more desired functions, such as those methods … described above, are shown stored within system memory 3020 as code 3025 …”), cause the one or more processors to: [[[perform the method of claim 1]]].
Regarding claim 2, the rejection of claim 1 is incorporated, and Dasgupta further teaches: wherein the script includes historical data maintained within the customer environment (Dasgupta, e.g., 10:60-67, “model experiment interface 144 may allow a user to specify a model experiment on a production model instance that is currently running in a production environment … implement an auditing process for production models, that may periodically test production model against real or artificial input data …” See also, e.g., 6:31-34, “MDE allows users to perform periodic audits on production traffic in order to help the production model to adapt to data whose characteristics change over time …”).
Regarding claim 3, the rejection of claim 2 is incorporated, and Dasgupta further teaches: wherein the historical data is associated with an audit log indicating a sequence of customer interactions with the customer environment (Dasgupta, e.g., 6:31-34, “MDE allows users to perform periodic audits on production traffic in order to help the production model to adapt to data whose characteristics change over time …” See also, e.g., 1:6-11, “machine-learned image models are increasingly being used in applications such as facial recognition, text and speech processing, computer-aided medical diagnosis …” Examiner’s note: “historical data may include and/or be associated with an audit log indicating a sequence of customer interactions with the particular customer environment 104. Depending on the embodiment, the audit log may be or include information such as labeling decisions applied to a plurality of documents maintained at the customer environment” (Spec. at ¶30). The labeling of documents in a customer environment is like the labeling of images or documents in facial recognition, text processing, and computer-aided medical diagnosis functionalities).
Regarding claim 4, the rejection of claim 3 is incorporated, and Dasgupta further teaches: wherein the audit log is an audit log for a first customer and the predetermined test condition is representative of one or more second customers sharing physical resources with the first customer while performing one or more large second customer-directed jobs (Dasgupta, e.g., 5:17-24, “model training jobs may be executed using provisioned machines and a rich layer of sophisticated software tools … MDE may be implemented as a fully managed cloud-based solution …” See also, e.g., 6:31-37, “MDE employs a federated model, allowing users to share their model by marking them public and encouraging other users to join the platform.” See also, e.g., 6:59-7:5, “MDE enables users to perform simulations based on historical production data to ensure model regression … MDE provides user interfaces that allow users to compare the ongoing model performance with its peers and facilitate production deployment through a pipeline …”).
Regarding claim 5, the rejection of claim 3 is incorporated, and Dasgupta further teaches: wherein the audit log is a first audit log, and the deploying further causes the customer environment to generate a second audit log representative of one or more modified past actions based on the predetermined test conditions (Dasgupta, e.g., 11:38-67, “MDE 130 may provide a model diagnosis interface 148 … used to generate a body of performance metrics from the model performance metrics in the repository … allow users to view the performance metrics in different ways, for example, organized according to media class, compared to other models … diagnosis interface 148 also allows a user to visually analyze the performance of the model that was the subject of the experiment … diagnosis interface 148 may obtain analytical feedback from the user, and then synthesize the feedback to infer certain corrective actions to take for a next iteration of model experiment … one or more adjustments to the training data set, or one or more changes to the model architecture of parameter values.” See also, e.g., 11:11-23, “model experiment interface 144 may also allow user to display model experiments in an organized way … ordering some experiments in developmental order … resulting performance of related groups of experiments may be plotted in a graph over iterations, so that the user can see progress in the development process in terms of improvements in a variety of performance metrics”).
Claims 13-16 are rejected for the additional reasons given in the rejections of claims 2-5 above.
Regarding claim 6, the rejection of claim 7 is incorporated, and Dasgupta further teaches: wherein the evaluation metric is based on one or more differences between the first audit log and the second audit log (Dasgupta, e.g., 11:11-37, “model experiment interface 144 may also allow user to display model experiments in an organized way … ordering some experiments in developmental order … resulting performance of related groups of experiments may be plotted in a graph over iterations, so that the user can see progress in the development process in terms of improvements in a variety of performance metrics … a notification may be sent to a user … when a model experiment has been completed … for other types of events, for example … when certain training conditions are met (e.g., when a threshold accuracy level is reached) …”).
Regarding claim 7, the rejection of claim 1 is incorporated, and Dasgupta further teaches: wherein the obtaining the evaluation metric includes obtaining an anonymized evaluation metric (Dasgupta, e.g., 6:59-7:5, “MDE enables users to perform simulations based on historical production data to ensure model regression … allow users to compare the ongoing model performance with its peers … provide adapters to listen to production audit traffic, create necessary alarms, and notifications. The MDE may then aggregate and surface production aberrations as an anomaly …” Examiner’s note: see dependent claim 8, wherein the anonymized evaluation metric includes an aggregation of evaluation metrics from a plurality of customer environments).
Regarding claim 8, the rejection of claim 7 is incorporated, and Dasgupta further teaches: wherein the customer environment is a first customer environment of a plurality of customer environments and the anonymized evaluation metric includes an aggregation of evaluation metrics from each of the plurality of customer environments (Dasgupta, e.g., 6:59-7:5, “MDE enables users to perform simulations based on historical production data to ensure model regression … allow users to compare the ongoing model performance with its peers … provide adapters to listen to production audit traffic, create necessary alarms, and notifications. The MDE may then aggregate and surface production aberrations as an anomaly …”).
Regarding claim 9, the rejection of claim 1 is incorporated, and Dasgupta further teaches: wherein the evaluation metric includes at least one of: (i) reliability, (ii) stability, (iii) training time, (iv) precision, (v) recall, (vi) area under a model curve, (vii) accuracy, (viii) adjusted mutual information, (ix) explained variance, (x) maximum error, (xi) mean absolute error, (xii) root mean squared error, (xiii) depth of recall, (xiv) throughput, (xv) latency, (xvi) resource usage, (xvii) memory, (xviii) CPU usage, (xix) network usage, (xx) IOPS usage, or (xxi) success or failure rates (Dasgupta, e.g., 5:4-13, “MDE may implement functionality to compute an array of different accuracy metrics to be used to track model performance … may include metrics such as precision, recall, and F1 scores, AUC (Area Under the Curve), ROC (Receiver Operating Characteristic), MAP (Mean Average Precision) scores … system is extensible to allow a user to introduce additional customized metrics logic”).
Regarding claim 10, the rejection of claim 1 is incorporated, and Dasgupta further teaches: updating, by the one or more processors, a module utilized in a customer environment based upon an analysis of the evaluation metric, wherein the updating includes deploying an updated module that modifies one or more parameters of the module based on the analysis (Dasgupta, e.g., 11:38-67, “MDE 130 may provide a model diagnosis interface 148 … used to generate a body of performance metrics from the model performance metrics … allows a user to visually analyze the performance of the model that was the subject of the experiment … may obtain analytical feedback from the user, and then synthesize the feedback to infer certain corrective actions to take … one or more adjustments to the training data set, one or more changes to the model architecture of parameter values.” See also, e.g., 10:49-51, “model validation runs may be used to perform tasks such as to auto-tune the model’s hyperparameters …” See also, e.g., 15:36-41, “model update layer 240 may implement a model deployment pipeline 249 … Once approved the deployment pipeline may package the resulting model for deployment into a production environment”).
Claims 17-20 are rejected for the additional reasons given in the rejections of claims 7-10 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. § 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 11 is rejected under 35 U.S.C. § 103 as being unpatentable over Dasgupta in view of Acharya et al., U.S. 11,494,171 B1 (“Acharya”).
Regarding claim 11, the rejection of claim 1 is incorporated, but Dasgupta does not more particularly teach that the predetermined test conditions are representative at least one of four variants, one of said variants comprising one or more additional large jobs by the customer environment. However, Acharya does teach: wherein the predetermined test conditions are representative of at least one of: (i) one or more additional large jobs by the customer environment, (ii) network latency in communications to and from the customer environment, (iii) memory leak in the customer environment, or (iv) packet loss in communications to and from the customer environment (Acharya, e.g., 13:30-51, “decentralized platform 108 facilitates the at least one AI model to have a public naming scheme to be found and translated to an API end-point … establishing the right endpoint to contact for model/prediction execution … datasets used for the validation are generated by the model validators 104 and are required to meet one or more predefined criteria before being implemented for validation. These pre-defined criteria include datasets that: … are free of private information, are not align to the distribution of the test dataset in the model artefact (i.e., no overlap with the data that has been used by the model publisher 102) … and stress-test the model for compute time performance.” Examiner’s note: “Depending on the embodiment, the predetermined test conditions may additionally or alternatively stress the evaluation compute by: (i) utilizing available computing and/or memory resources representative of a single user performing one or more additional large jobs (e.g., jobs that require 10%, 25%, 50%, 70%, 80%, 90%, etc. of resources for performing customer-directed jobs available for the evaluation compute …” (Spec. at ¶51). That is, by providing a testing condition whereby validation datasets meet a compute time performance stress-testing threshold, Acharya is consistent with the stress test of the evaluation compute of the invention wherein the test conditions are representative of one or more additional large jobs) for the purpose of providing storage of and access to a plurality of models over the Internet, including a plurality of validators for validating performance of the models, wherein validations may include testing, hardening and performance evaluations such as stress testing (Acharya, e.g., 2:11-30, 13:13-51).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the model development environment including model experiment and diagnosis / evaluation interfaces as taught by Dasgupta to provide that the predetermined test conditions are representative at least one of four variants, one of said variants comprising one or more additional large jobs by the customer environment because the disclosure of Acharya shows that it was known to those of ordinary skill in the pertinent art to improve a method and system for decentralized model deployment and evaluation to provide that the predetermined test conditions are representative at least one of four variants, one of said variants comprising one or more additional large jobs by the customer environment for the purpose of providing storage of and access to a plurality of models over the Internet, including a plurality of validators for validating performance of the models, wherein validations may include testing, hardening and performance evaluations such as stress testing (Acharya, Id.).
Conclusion
Examiner has identified particular references contained in the prior art of record within the body of this action for the convenience of Applicant. Although the citations made are representative of the teachings in the art and are applied to the specific limitations within the enumerated claims, the teaching of the cited art as a whole is not limited to the cited passages. Other passages and figures may apply. Applicant, in preparing the response, should consider fully the entire reference as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art and/or disclosed by Examiner.
Examiner respectfully requests that, in response to this Office Action, support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line number(s) in the specification and/or drawing figure(s). This will assist Examiner in prosecuting the application.
When responding to this Office Action, Applicant is advised to clearly point out the patentable novelty which he or she thinks the claims present, in view of the state of the art disclosed by the references cited or the objections made. He or she must also show how the amendments avoid such references or objections. See 37 C.F.R. 1.111(c).
Examiner interviews are available via telephone and video conferencing using a USPTO-supplied web-based collaboration tool. Applicant is encouraged to submit an Automated Interview Request (AIR) which may be done via https://www.uspto.gov/patent/uspto-automated-interview-request-air-form, or may contact Examiner directly via the methods below.
Any inquiry concerning this communication or earlier communication from Examiner should be directed to Andrew M. Lyons, whose telephone number is (571) 270-3529, and whose fax number is (571) 270-4529. The examiner can normally be reached Monday to Friday from 10:00 AM to 6:00 PM ET. If attempts to reach Examiner by telephone are unsuccessful, Examiner’s supervisor, Wei Mui, can be reached at (571) 272-3708. Information regarding the status of an application may be obtained from the Patent Center system. For more information about the Patent Center system, see https://www.uspto.gov/patents/apply/patent-center. If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call (800) 786-9199 (in USA or Canada) or (571) 272-1000.
/Andrew M. Lyons/Primary Examiner, Art Unit 2191
Prosecution Insights