DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Style
In this action unitalicized bold is used for claim language, while italicized bold is used for emphasis.
Applicant Reply
“The claims may be amended by canceling particular claims, by presenting new claims, or by rewriting particular claims as indicated in 37 CFR 1.121(c). The requirements of 37 CFR 1.111(b) must be complied with by pointing out the specific distinctions believed to render the claims patentable over the references in presenting arguments in support of new claims and amendments. . . . The prompt development of a clear issue requires that the replies of the applicant meet the objections to and rejections of the claims. Applicant should also specifically point out the support for any amendments made to the disclosure. See MPEP § 2163.06. . . . An amendment which does not comply with the provisions of 37 CFR 1.121(b), (c), (d), and (h) may be held not fully responsive. See MPEP § 714.” MPEP § 714.02. Generic statements or listing of numerous paragraphs do not “specifically point out the support for” claim amendments. “With respect to newly added or amended claims, applicant should show support in the original disclosure for the new or amended claims. See, e.g., Hyatt v. Dudas, 492 F.3d 1365, 1370, n.4, 83 USPQ2d 1373, 1376, n.4 (Fed. Cir. 2007) (citing MPEP § 2163.04 which provides that a ‘simple statement such as ‘applicant has not pointed out where the new (or amended) claim is supported, nor does there appear to be a written description of the claim limitation ‘___’ in the application as filed’ may be sufficient where the claim is a new or amended claim, the support for the limitation is not apparent, and applicant has not pointed out where the limitation is supported.’)” MPEP § 2163(II)(A).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7-10 and 17-19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Generally: separately listed claim elements are construed as distinct components, that all claim terms must be given weight, there is presumed to be a difference in meaning and scope when different words or phrases are used in separate claims, and repeated and consistent descriptions in the specification indicate the proper scope of a claimed term. “[C]laims must ‘conform to the invention as set forth in the remainder of the specification and the terms and phrases used in the claims must find clear support or antecedent basis in the description so that the meaning of the terms in the claims may be ascertainable by reference to the description.’ 37 C.F.R. § 1.75(d)(1).” Phillips v. AWH Corp., 415 F.3d 1303, 1316 (Fed. Cir. 2005) (as cited in MPEP § 2111). Therefore, use of two different terms in the claims that both rely on the description of a single structure in the Specification may render at least one term indefinite because there is no way to determine which term should be construed in view of the description of the single structure.
Claims 7 and 17 substantially recite “wherein each of the plurality of production machine learning models receives a different percentage of the input data relative to each other production machine learning models of the plurality of machine learning models.” It is not clear how to interpret the redundant language “relative to each other” and “relative to . . . other production machine learning models.” Presumably this is a typo, but that is not a basis for reading out duplicate terms from claim. The use of two different terms in reference to what appears to be the same claim element in the same configuration, renders the claim indefinite because it is not clear whether one or two claim elements are required by the claim language.
Claims 9-10 and 18-19 recite “allocating, on the model serving platform, a predetermined number of production model execution slots and a predetermined number of shadow model execution slots.” Both “production model execution slots” and “shadow model execution slots” appear to be undefined, applicant invented terms. The Specification uses the similar terms “production slots” and “shadow slots,” but it is not clear from context if they refer to physical slots (i.e. in a server rack where the code running each type of model is being executed) or if the “slots” refer to any set of computing resources, or even an addressable location used for the models. The claimed allocation does not indicate whether existing physical slots in the server racks are being allocated or computing resources. Ultimately, all attempts at determining the meaning of these applicant invented terms are nothing more than guesses. It is submitted that claim language that requires guessing at the meaning of terms fails to meet the requirements of “claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.” 35 U.S.C. § 112b.
All dependent claims are rejected as containing the limitations of the claims from which they depend.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 9-11, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Christopher (Deploying Machine Learning Models in Shadow Mode; 2020), Hilton (CRE life lessons (parts 1 and 2); 2017) and Tranquillin (Taking a practical approach to BigQuery slot usage analysis; 2020).
1. A method comprising: executing, on a model serving platform, a plurality of production machine learning models in a plurality of production model execution slots and a plurality of shadow machine learning models in a plurality of shadow model execution slots (Christopher teaches “There are two fundamental approaches to Shadow Mode: Application level 1. Implementations 2. Infrastructure level implementations[.]” Christopher P. 5. Implementation at either the application level or the infrastructure level each independently teach “a model serving platform.” “The strategies you adopt when deploying software have the potential to save you from expensive and insidious mistakes. This is particularly true for machine learning systems, where detecting subtle data munging, feature engineering or model bugs in production can be very challenging, particularly when the production data inputs are hard to replicate exactly. “Shadow Mode” is one such deployment strategy, and in this post I will examine this approach and its trade-offs.” Christopher p. 1. “In scenarios where an ML system is conducting multiple model deployments a day, manual batch testing may be unrealistic - instead a service may be required to check that predictions fall within expected bounds, which can be configured depending on the expectations from the research environment.” Cristopher p. 7. Based on the general nature of the explanation in Christopher (e.g. “for machine learning systems”) and the teaching of using a service to evaluate multiple models per day for deployment, one of ordinary skill in the art would understand the teaching of Christopher as applying to a plurality of shadow and production (i.e. deployed) models. Further, a mere duplication of parts has been found to be obvious. See MPEP § 2144.04. Here, the claims merely recite “a plurality” with no specific relationship or interaction between the models in the plurality of models, and no criticality attributed to the duplication.
Christopher does not explicitly teach “slots.”
The “slots” are interpreted as reading on computing resources in general. The previously cited art does not teach allocating resources for models.
Tranquillin teaches “Each task, executed on an ad-hoc microservice, requires an adequate amount of computational power in order to be fulfilled. The slot is the computational capacity unit to measure that power. The BigQuery engine dynamically identifies the amount of slots needed to perform a single query, and background processes will transparently allocate the adequate computation power needed to accomplish the task. So, it’s essential to understand how to monitor and analyze slot usage, because that lets your technical team understand if there are any bottlenecks, then allows the business to choose the best pricing model (on-demand vs. flat-rate)” Tranquillin p. 2.
It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the teaching of Tranquillin to use “slots” for each data structure (i.e. model) because this allows monitoring of resources used by the models to determine whether more resources are needed, or to purchase less resources from the cloud.) routing input data received from a production client application to the plurality of production machine learning models and to the plurality of shadow machine learning models in parallel, wherein each shadow machine learning model receives the input data and generates output that is not returned to the production client application; (Cristopher teaches ““Shadow Mode” or “Dark Launch” as Google calls it is a technique where production traffic and data is run through a newly deployed version of a service or machine learning model, without that service or model actually returning the response or prediction to customers/other systems. Instead, the old version of the service or model continues to serve responses or predictions, and the new version’s results are merely captured and stored for analysis.” Christopher p. 2. “This is where some of the realities of maintaining environments designed to replicate production come into play, namely: . . . Replicating the inbound traffic realistically.” Christopher p. 3.) receiving, at a model monitoring engine, production output data from a first production machine learning model of the plurality of production machine learning models; receiving, at the model monitoring engine, offline output data from a first shadow machine learning model of the plurality of shadow machine learning models; performing, by the model monitoring engine, analytics on the production output data and the offline output data to evaluate performance of the first production machine learning model and the first shadow machine learning model in a comparative environment; (This pair of limitations reads on receiving output data from both a production (i.e. online) machine learning model and from a shadow (i.e. offline) machine learning model at “a model monitoring engine.” The “model monitoring engine” is not expressly defined in the Specification and nothing in the Specification limits this claim element to any particular structure. Further, the Specification describes this element as performing a plethora of operations related to monitoring of models. See Spec. ¶¶3-65. Consistent with this disclosure, the model monitoring engine is interpreted as any combination of computing resources carrying out some aspect of monitoring the model. This interpretation of the term “model monitoring engine” applies throughout the claim set. Christopher teaches: “Once your new model is deployed in shadow mode, it’s time to reap the benefits. In addition to the standard service-level monitoring you should be conducting at all times (HTTP response codes, latency, memory usage etc.), you are now able to compare model inputs and outputs. This comparison will be with both the research environment, and also over time to make sure inputs and outputs do not suddenly change (perhaps due to a change in an external data source). . . . Key things to analyze include . . . raw data . . . features being generated as inputs to the model . . . predictions being generated by the model . . . The time you wait before conducting this analysis depends on the business requirements and the amount of traffic coming in.” Christopher p. 7. One of ordinary skill in the art would understand the analysis taught in Cristopher, including the analysis of predictions being generated by the model, to be performed using some computing resources. As explained above, the claimed “model monitoring engine” reads on computing resources carrying out some aspect of monitoring the models. Note that Cristopher also explicitly teaches recording data from each model for later analysis: “You should already be recording all inputs and outputs to your model, either in logs or a database, for reproducibility. Shadow mode introduces the need to be able to distinguish between predictions from the current (customer-facing) model and the model in shadow mode. You should design your logging and/or database schema accordingly, such that this distinction can be made, for example by including a column in a database for recording the model version.” Christopher p. 5. “Your data collection techniques also need to evolve so that you can easily distinguish between shadow and non-shadow model inputs and outputs. The good news is that once a model is running in shadow mode, the switch to make it live should be relatively simple – forking all traffic to the new model, or toggling a feature flag.” Christopher p. 7-8.) promoting the first shadow machine learning model to a production machine learning model based on the offline output data and the analytics performed by the model monitoring engine; and demoting the first production machine learning model based on the production output data and the analytics performed by the model monitoring engine. (“production traffic and data is run through a newly deployed version of a service or machine learning model, without that service or model actually returning the response or prediction to customers/other systems. Instead, the old version of the service or model continues to serve responses or predictions, and the new version’s results are merely captured and stored for analysis.” Christopher p. 2. “Fundamentally, “Shadow Mode” is a form of testing in production. . . . “This is where some of the realities of maintaining environments designed to replicate production come into play, namely: . . . Replicating the inbound traffic realistically. . . . Investing in the same quality of monitoring, metrics, tracing and analysis for non-production environments.” Christopher p. 3. “Which is to say, you can make the distinction between having a new version of your system and all its dependencies running on production infrastructure, and switching customer-facing systems to using that new version. . . . [F]or the rest of this post I will use the terms like so: Deployment (“in production but not affecting customers/users”) and release (“in production and affecting customers/users”).” Christopher p. 4. “The good news is that once a model is running in shadow mode, the switch to make it live should be relatively simple – forking all traffic to the new model, or toggling a feature flag. In our hypothetical scenario above, if the bank had launched Jenny’s model in shadow mode and then performed the appropriate analysis once enough data had been collected, then no customers, investors or regulators would have been affected.” Christopher p. 7-8.
Christopher does not expressly teach demoting the production model.
Hilton teaches “In theory, once you’ve dark-launched 100% of your traffic to the new service, making it go “live” is almost trivial. At the point where the traffic is forked to the original and new service, you’ll return the new service response instead of the original service response. If you have an enforced timeout on the new service, you’ll change that to be a timeout on the old service. Job done! Now you can disable monitoring of your original service, turn it off, reclaim its compute resources, and delete it from your source code repository. (A team meal celebrating the turn-down is optional, but strongly recommended.) Every service running in production is a tax on support and reliability, and reducing the service count by turning off a service is at least as important as adding a new service.” Hilton p. 9.
It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the teaching of Hilton because disabling the previously deployed software saves computing resources.)
9. The method of claim 1, comprising: allocating, on the model serving platform, a predetermined number of production model execution slots and a predetermined number of shadow model execution slots, wherein each of the plurality of production machine learning models occupies one of the predetermined number of production slots, and wherein each of the plurality of shadow machine learning models occupies one of the predetermined number of shadow slots. (The “slots” are interpreted as reading on computing resources in general. The previously cited art does not teach allocating resources for models.
Tranquillin teaches “Each task, executed on an ad-hoc microservice, requires an adequate amount of computational power in order to be fulfilled. The slot is the computational capacity unit to measure that power. The BigQuery engine dynamically identifies the amount of slots needed to perform a single query, and background processes will transparently allocate the adequate computation power needed to accomplish the task. So, it’s essential to understand how to monitor and analyze slot usage, because that lets your technical team understand if there are any bottlenecks, then allows the business to choose the best pricing model (on-demand vs. flat-rate)” Tranquillin p. 2.
It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the teaching of Tranquillin to use “slots” for each data structure (i.e. model) because this allows monitoring of resources used by the models to determine whether more resources are needed, or to purchase less resources from the cloud.)
10. The method of claim 9, wherein the first shadow machine learning model is upgraded to a one of the predetermined number of production slots previously occupied by the first production machine learning model. (See rejection of claim1. Note that upgrading the model run on the “slot” of resources teaches upgrading the model.)
For rejections of claims 11 and 20, see rejection of claim 1.
For rejections of claims 18-19, see rejections of claims 9-10.
Claims 2-6 and 12-16 are rejected under 35 U.S.C. 103 as being unpatentable over Christopher Hilton, Tranquillin, and AWS (What is pub/sub messaging?; Feb 2023).
2. The method of claim 1, comprising: routing the input data to an event streaming platform; and publishing the input data to a first topic. (As best understood, this claim is directed to using pub-sub messaging protocols to transfer data.
Christopher teaches “In scenarios where performance is a concern (systems that give real-time predictions, or that have algorithms which are time-intensive), then best practice is to pass the inputs and record the outputs on the new model asynchronously (perhaps using threads or by passing the information to a distributed task queue). More advanced systems might pass the inputs to a separate Kafka topic for the new model.” Christopher p. 5. (Note that a Kafka topic refers a “topic” in the sense used with pub/sub messaging, taught in the reference below. See Alluri Kafka Topics and Partitions — A Complete Guide; 2019, cited ONLY as evidentiary support for the interpretation of the term “topic” as used in Christopher and NOT as prior art.)
The previously cited art does not teach the operations of routing input data to an event streaming platform and publishing the input data to a first topic, or generally teach details of pub-sub messaging.
AWS teaches “Publish-subscribe messaging, or pub/sub messaging, is an asynchronous communication model that makes it easy for developers to build highly functional and architecturally complex applications in the cloud. In modern cloud architecture, applications are decoupled into smaller, independent building blocks called services. Pub/sub messaging provides instant event notifications for these distributed systems. It supports scalable and reliable communication between independent software modules. . . . How does pub/sub messaging work? The publish-subscribe (pub/sub) system has four key components. Messages A message is communication data sent from sender to receiver. Message data types can be anything from strings to complex objects representing text, video, sensor data, audio, or other digital content. Topics Every message has a topic associated with it. The topic acts like an intermediary channel between senders and receivers. It maintains a list of receivers who are interested in messages about that topic. Subscribers A subscriber is the message recipient. Subscribers have to register (or subscribe) to topics of interest. They can perform different functions or do something different with the message in parallel. Publishers The publisher is the component that sends messages. It creates messages about a topic and sends them once only to all subscribers of that topic. This interaction between the publisher and subscribers is a one-to-many relationship. The publisher doesn’t need to know who is using the information it is broadcasting, and the subscribers don’t need to know where the message comes from.” AWS pp. 1-2. One of ordinary skill in the art would understand video/audio/sensor as teaching streaming data. Further, AWS expressly teaches using this technique with “applications that rely on real-time events” and “instantaneous, push-based delivery,” which one of ordinary skill in the art would understand to teach “routing input data to the event streaming platform.” See AWS p. 5. The “platform” reads on the pub/sub system as a whole.
It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the teaching of AWS to modify the teaching of the previously cited art to include the type of pub-sub messaging pattern in a system including updating of production and shadow models as an instance of applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; The prior art contained a "base" device (method, or product) upon which the claimed invention can be seen as an "improvement” (the prior art teaches techniques of using shadow models and production models to improve models without the risk of placing untested models into service). The prior art contained a known technique that is applicable to the base device (method, or product) (as shown above, the prior art also contained the known technique of using a pub/sub system for communication between components, which is applicable to both shadow models and production models). One of ordinary skill in the art would have recognized that applying the known technique would have yielded predictable results and resulted in an improved system (one of ordinary skill in the art would have recognized that applying the pub/sub techniques to the techniques associated with training of shadow models would result in a system with increased throughput and scalability and would eliminate the need for polling when acquiring real time (steaming) data for the models. See MPEP § 2143(I)(D). See also AWS p. 5. This motivation applies to all combinations of shadow modeling techniques with pub/sub techniques.
3. The method of claim 2, comprising: subscribing, by each of the plurality of shadow machine learning models, to the first topic, and consuming the input data from the event streaming platform. (AWS teaches that pub/sub techniques include “A message is communication data sent from sender to receiver.” AWS p. 2. “A subscriber is the message recipient. Subscribers have to register (or subscribe) to topics of interest.” AWS p. 2. “Pub/sub messaging instantly pushes asynchronous event notifications when messages are published to the message topic.” AWS pp. 2-3. “Pub/sub messaging provides significant advantages to developers who build applications that rely on real-time events.” AWS p. 5. Applying this technique to machine learning models is addressed in the motivation to combine in claim 2.
4. The method of claim 2, comprising: routing the production output data to the event streaming platform; and publishing the production output data to a second topic. (This reads on using pub/sub techniques to transmit data output from models to another computing component (i.e. for evaluating the models.) Setting up a given “topic” for a given type of data to be transmitted is how pub/sub works, as explained in AWS. In other words, given the model output data to be analyzed, this claim reads on merely using standard pub/sub operations to transmit the data. AWS teaches: “A message is communication data sent from sender to receiver.” AWS p. 2. “A subscriber is the message recipient. Subscribers have to register (or subscribe) to topics of interest.” AWS p. 2. “Pub/sub messaging instantly pushes asynchronous event notifications when messages are published to the message topic.” AWS pp. 2-3. “In some cases, publishers can also be subscribers.” AWS pp. 4. “Pub/sub messaging provides significant advantages to developers who build applications that rely on real-time events.” AWS p. 5. See also AWS p. 3 illustrating the overall system. Applying this technique to machine learning models is addressed in the motivation to combine in claim 2.)
5. The method of claim 4, comprising: routing the offline output data to the event streaming platform; and publishing the offline output data to a third topic. (See rejection of claim 4. Given output of multiple models for analysis, these claims merely recite conventional ways of transmitting data using pub/sub techniques.)
6. The method of claim 5, comprising: subscribing, by the model monitoring engine, to the second topic and the third topic. (See rejection of claim 4. Given output of multiple models for analysis by some other component, these claims merely recite conventional ways of transmitting data using pub/sub techniques.)
For rejections of claims 12-16, see rejections of claim 2-6.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Christopher Hilton Tranquillin and Sayal (AB-BPM: Performance-Driven Instance Routing for Business Process Improvement; 2017)
7. The method of claim 1, wherein each of the plurality of production machine learning models receives a different percentage of the input data relative to each other production machine learning model of the plurality of production machine learning models. (The previously cited art does not expressly teach production models receiving different percentages of input data.
Sayal teaches “we develop a technique called AB-BPM that offers AB testing for process versions with immediate feedback at runtime. We implemented this technique in such a way that two versions (A and B) are operational in parallel and any new process instance is routed to one of them. The routing decision is made at runtime on the basis of the achieved results for the registered performance metrics of each version. AB-BPM provides for ultimate convergence towards the best performing version, no matter if it is the old or the new version.” Sayal Abstract. “AB testing in DevOps compares two versions of a deployed product (e.g., a Web page) by observing users’ responses to versions A/B, and determines which one performs better [8]. We implemented this technique in such a way that two versions (A and B) of a process are operational in parallel and any new process instance is routed to one of them. Through a series of experiments and observations, we have developed an instance routing algorithm, LTAvgR, which is adapted to the context of executing business processes. The routing decision is guided by the observed results for registered performance metrics of each version at runtime. The technique has been evaluated extensively on both synthetic and real-life data. The results showed that AB-BPM provides for ultimate convergence towards the best performing version.” Sayal P. 114. “From the above analysis, we derive the following three requirements: . . . R3 Enable rapid adjustments on process model level: the benefits of a solution should be suited to process models and their specific characteristics. . . . One DevOps practice is live testing, where new versions of the software are tested in production with actual users of the system. The most popular form of live testing is AB testing, where two versions (A and B) are deployed side by side and both receive a share of the production workload while being monitored closely. The monitoring data is then used to draw conclusions about the effectiveness of one version over the other, for instance in the form of increased revenue from higher click-through rates.” Sayal P. 115. “In our work, the monitoring is performed by the instance router by observing a Process Performance Indicator (PPI) like satisfaction ratings obtained from end users. Based on the chosen PPI, the instance router dynamically adjusts the request distribution rates” Sayal P. 116. “We designed the
architecture such that the two versions of the process model are deployed side by AB-BPM: Performance-Driven Instance Routing 119 side in the same execution engine. The instance router distributes the instance creation requests as per its internal logic. . . . However, the multi-armed bandit algorithms can identify the superior version during the experimentation and alter the allocation of requests to different versions. When a version is clearly superior to the other, most of the requests are sent to the superior version.” Sayal PP. 118-119.
It would have been obvious to one of ordinary skill in the art before the effective filing date to apply the teaching of Sayal to modify the prior art such that production models receive different percentages of data, because this is part of a technique that more quickly allocates more data to the better model, which ultimately leads to a higher proportion of more desirable outcomes from the better model.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Christopher Hilton, Tranquillin, Sayal, and Vittal Deploy shadow ML models in Amazon SageMaker; 2021
8. The method of claim 7, wherein each of plurality of shadow models receives 100% of the input data. (The previously cited art does not expressly teach the sending shadow models (i.e. non-production models) 100% of input data.
Vittal teaches “Shadow deployment consists of releasing version B alongside version A, fork version A’s incoming requests, and send them to version B without impacting production traffic. This is particularly useful to test production load on a new feature and measure model performance on a new version without impacting current live traffic. A rollout of the application is triggered when stability and performance meet the requirements. Shadow deployment has the following advantages: You can evaluate model performance without impacting production traffic or workload. There is no impact on the customer or production workload behavior. You can explore synchronous as well as asynchronous approaches. You can check to see if the stability and performance of the application meets your requirements, which reduce risk.” Vittal P. 1.
It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the teaching of Vittal because using all the data may result in better trained models before deployment.
For rejection of claim 17, see rejections of claims 7 and 8.
Response to Arguments
Applicant's arguments filed 07/10/2026 have been fully considered but they are not persuasive.
Rejections under § 112
No specific arguments are put forth.
Rejections under § 103
Applicant asserts “[t]he specification explains the claimed distinction” between “the claimed model monitoring engine performing analytics on production and offline model outputs in a comparative model-serving environment to determine promotion and demotion paths for machine learning models based on model performance” and the concepts in the prior art. Rem. 11. Per the Remarks, the description in the Specification of a “model monitoring engine [that] evaluates output of models, assigns or aggregates decision scores, and uses scores or analytics to determine upgrade paths, downgrade paths, retraining paths, decommissioning, and promotion/demotion of models” results in a claim scope that does not read on the art of record. Rem. 11. As best understood, Applicant takes the position that the office action is deficient because it fails to articulate a reason that one of ordinary skill in the art would implement this unclaimed subject matter.1 Arguments which clearly articulate a distinction between the claimed subject matter and the prior art may be more convincing.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL M KNIGHT whose telephone number is (571) 272-8646. The examiner can normally be reached Monday - Friday 9-5 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached on (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
PAUL M. KNIGHTPrimary ExaminerArt Unit 2148
/PAUL M KNIGHT/
Primary Examiner, Art Unit 2148
1 “The cited combination does not provide an articulated reason why a person of ordinary skill would have modified . . . to arrive at this particular model-monitoring architecture and analytics-driven promotion/demotion workflow.” Rem. 11, citing Spec. ¶¶ 31-33, 53-56, 61-63.
Prosecution Insights