DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is made final.
This action is in response to the claims filed May 22, 2026.
Claims 1, 11, and 16 have been amended. Claims 2, 12, and 17 have been cancelled.
Response to Amendment
The amendments filed May 22, 2026 have been entered. Claims 1, 3-11, 13-16, and 18-20 remain pending in the case and have been rejected.
Response to Arguments
Regarding the claim objection response
Applicant’s amendments have been considered and the objection is withdrawn.
Regarding the 101 arguments
Applicant argues
The Rejection of Claims Under 35 U.S.C. & 101
Claims 1-20 are rejected under 35 U.S.C. § 101 as being directed to a judicial exception (i.e., law of nature, a natural phenomenon, or an abstract idea), without significantly more. (Office Action, page 3). This rejection is respectfully traversed.
The United States Patent and Trademark Office (USPTO) recently updated the Manual of Patent Examining Procedure (MPEP) to reflect the precedential decision issued in the Appeals Review Panel decision in Ex Parte Desjardins. Specifically, Ex Parte Desjardins explained the following: Enfish ranks among the Federal Circuit's leading cases on the eligibility of technological improvements. In particular, Enfish recognized that "[much of the advancement made in computer technology consists of improvements to software that, by their very nature, may not be defined by particular physical features but rather by logical structures and processes." 822 F.3d at 1339. Moreover, because "[software can make non-abstract improvements to computer technology, just as hardware improvements can," the Federal Circuit held that the eligibility determinations should turn on whether "the claims are directed to an improvement to computer functionality versus being directed to an abstract idea." Id. at 1336. (Desjardins, page 8).
In Ex Parte Desjardins, the Appeals Review Panel (ARP) overall credited benefits including reduced storage, reduced system complexity and streamlining, and preservation of performance attributes associated with earlier tasks during subsequent computational tasks as technological improvements that were disclosed in the patent application specification.
Specifically, the ARP upheld the Step 2A Prong One finding that the claims recited an abstract idea (i.e., mathematical concept). In Step 2A Prong Two, the ARP then determined that the specification identified improvements as to how the machine learning model itself operates, including training a machine learning model to learn new tasks while protecting knowledge about previous tasks to overcome the problem of "catastrophic forgetting" encountered in continual learning systems. (MPEP § 2106.04(d), subsection III, updated in view of Ex Parte Desjardins).
Now, in the present case, Applicant has designed an AI architecture and related method in which multiple models trained to serve different use cases share common training and configuration infrastructure. In this manner, developers can quickly test new features without needing to train a new model scratch, reducing system complexity, compute costs, and streamlining new feature rollouts. This technological improvement, present in the amended claims, is also reflected in at least paragraphs 3, 92, and 93 of the originally filed specification. Paragraphs 92-93 are reproduced below for convenience.
[0092] For illustration purposes, a comparison is presented of the effort required to implement a new model by a developer. In conventional model development, without the functionality of AIMM 906, the developer would have had to configure and train a new model from scratch and then integrate the new model with the functionality already provided by the experience. If another developer wanted to use the same model, the other developer will have to also build and train the new model and then integrate the model with their own experience 1106. Further, comparing the performance of different models would require having to set up multiple experiments with the multiple models, and the models could not be run simultaneously, so the result data may not be as precise because each model had a different running environment.
[0093] With the AIMM 906, a common training, configuration, and serving infrastructure is available for multiple developers. Further, the models can be shared between experiences (e.g., the same model can be applied to several experiences that send emails), and different models can be ramped to different sub- segments of the same experiences (e.g., country-specific models for generating job alerts). New integrations are not needed to apply models, and the models do not need to be manually re-trained for each different segment; the AIMM 906 will automatically train the models and generate estimates for each segment. Also, multiple models can be ramped to the same audience, and the relative model performance can be compared. The model performance can also be compared to static variants.
Reducing system complexity and streamlining have been found by the Appeals Review Panel (ARP) (refer above) to offer a technological improvement under Step 2A Prong Two. For at least the foregoing reasons, the claims contain patent-eligible subject matter.
Examiner Response
Argues that the amended claims provide a technical logical improvement through common training and configuration infrastructure that reduces system complexity, a compute cost, and streamlines model deployment. This argument has been considered but is not persuasive. Although the specification describes these asserted benefits, the claims recite the common infrastructure, concurrent model execution, performance evaluation, and model selection only at a functional level Without reciting a particular technological manner by which the asserted reductions in system complexity, compute costs, or streamlined deployment are achieved. Rather, the claims broadly recite the desired functions of configuring, executing, evaluating, and selecting ML models. Accordingly, the claims do not reflect a particular technological solution that integrates the judicial exception into a practical application under Step 2A, Prong Two.
Applicant argues
Notwithstanding the above, the USPTO's current guidance for determining whether a claim is directed to patent-eligible subject matter is further set forth in the Ninth Edition, Revision 10.2019 (revised June 2020) of the Manual of Patent Examination Procedure (MPEP), and particularly Sections 2103 through 2106.07(c). This guidance was recently expanded in the Reminders on Evaluating Subject Matter Eligibility of claims under 35 U.S.C. 101 (the "August Memo") issued on August 4, 2025. Notably, the August Memo establishes a concrete threshold for making eligibility rejections. Examiners must now demonstrate that it is "more likely than not"-that is, a greater than 50% probability-that a claim is ineligible. This burden has not been met, for at least the following reasons.
Step 2A, Prong 1
In Prong One examiners evaluate whether the claims at issue recite a judicial exception, i.e., whether a law of nature, natural phenomenon, or abstract idea is set forth or described in the claim. MPEP 2106.04.II.A.1.
A. THE EXAMINER'S ALLEGED ABSTRACT CONCEPT TO WHICH THE CLAIMS ARE ALLEGEDLY DIRECTED DOES NOT FALL WITHIN THE MENTAL PROCESSES GROUPING
The Examiner alleges that the claims cover mental processes (referring, e.g., to the originally claimed "configuring" and "selecting" elements, see Office Action pages 3-4).
The courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. MPEP 2106.04(a)(2).III. Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions. Id.
Claims do not recite a mental process when they do not contain limitations that can practically be performed in the human mind, for instance when the human mind is not equipped to perform the claim limitations. MPEP 2106.04(a)(2).III.A (emphasis added). See SRI Int'l, Inc. v. Cisco Systems, Inc., 930 F.3d 1295, 1304 (Fed. Cir. 2019) (declining to identify the claimed collection and analysis of network data as abstract because "the human mind is not equipped to detect suspicious activity by using network monitors and analyzing network packets as recited by the claims"); CyberSource, 654 F.3d at 1376, 99 USPQ2d at 1699 (distinguishing Research Corp. Techs. v. Microsoft Corp., 627 F.3d 859, 97 USPQ2d 1274 (Fed. Cir. 2010), and SiRF Tech., Inc. v. Int'l Trade Comm'n, 601 F.3d 1319, 94 USPQ2d 1607 (Fed. Cir. 2010), as directed to inventions that "could not, as a practical matter, be performed entirely in a human's mind"). Id.
By this Response, the independent claims are amended to recite, in part, "configuring one or more ML models from the plurality of ML models for the experiment based on the parameter values entered on the first UI, wherein the modeling manager comprises a common training and configuration infrastructure for configuring the plurality of ML models" and "initializing the experiment, the initializing comprising using the modeling manager to concurrently execute two or more models targeted at different sets of users and evaluating a performance of each of the concurrently executed models".
Applicant respectfully notes that it is not practical for a person to mentally concurrently execute a plurality of models in the manner and context currently claimed. Moreover, even if a person were able to somehow execute the models concurrently, by the time a person had mentally completed the required evaluations, the period to functionally respond (that is, to serve the request to the correct model, evaluated concurrently, and then send the response to the experience module) would be long over. In other words, even if theoretically possible, it is not practical to perform the recited steps mentally in the context currently claimed.
B. CONCLUSION
Therefore, because the claims do not fall within any of the enumerated groupings of abstract ideas, it is reasonable to find that the claims do not recite an abstract idea. Accordingly, it is respectfully requested that the Examiner reconsiders and withdraws the rejection of the claims under 35 U.S.C. § 101.
Examiner Response
Applicant argues that the claims do not recite to mental process because certain limitations involving execution of ML models cannot practically be performed in the human mind. This argument has been considered but is not persuasive. the rejection does not characterize the execution of the ML models itself as a mental process. rather, the claims recite evaluating the performance of alternatives and selecting an alternative based on the evaluated performance, which constitute evaluations and judgments that can practically be performed in the human mind. The presence of additional limitations that cannot practically be performed in the human mind does not prevent the claim from reciting a mental process under Step 2A Prong One; those additional elements are considered under Step 2A, Prong Two.
Applicant argues
Step 2A, Prong 2
In Prong Two, examiners evaluate whether the claim as a whole integrates the exception into a practical application of that exception. MPEP 2106.04.II.A.2.
Applicant respectfully submits that the claims, as amended, satisfy Step 2A, Prong 2, for at least the reasons already discussed with respect to the Appeals Review Panel decision in Ex Parte Desjardins.
Accordingly, it is respectfully requested that the Examiner reconsiders and withdraws the rejection of the claims under 35 U.S.C. § 101.
Step 2B
Step 2B asks: Does the claim recite additional elements that amount to significantly more than the judicial exception? MPEP 2106.05.II. Examiners should answer this question by first identifying whether there are any additional elements (features/limitations/steps) recited in the claim beyond the judicial exception(s), and then evaluating those additional elements individually and in combination to determine whether they contribute an inventive concept (i.e., amount to significantly more than the judicial exception(s)). Id.
1. ISSUES WITH SUBSTANCE
The Supreme Court has identified a number of considerations as relevant to the
evaluation of whether the claimed additional elements amount to an inventive concept. MPEP 2106.05. Limitations that the courts have found to qualify as "significantly more" when recited in a claim with a judicial exception are described in MPEP 2106.05(a)-(e). See id. Limitations that the courts have found not to be enough to qualify as "significantly more" when recited in a claim with a judicial exception are described in MPEP 2105.05(f), 2105.05(d), 2106.05(g), and 2106.05(h). See id. The list of considerations here is not intended to be exclusive or limiting. Id. Additional elements can often be analyzed based on more than one type of consideration and the type of consideration is of no import to the eligibility analysis. Id. Additional discussion of these considerations, and how they were applied in particular judicial decisions, is provided in in MPEP § 2106.05(a) through (h). Id.
A. THE CLAIMS AT ISSUE ADD A SPECIFIC LIMITATION OTHER THAN WHAT IS WELL-UNDERSTOOD, ROUTINE, OR CONVENTIONAL
Limitations that the courts have found to qualify as "significantly more" when recited in a claim with a judicial exception include: "v. Adding a specific limitation other than what is well- understood, routine, conventional activity in the field, or adding unconventional steps that confine the claim to a particular useful application, e.g., a non-conventional and non-generic arrangement of various computer components for filtering Internet content, as discussed in BASCOM Global Internet v. AT&T Mobility LLC, 827 F.3d 1341, 1350-51, 119 USPQ2d 1236, 1243 (Fed. Cir. 2016) (see MPEP § 2106.05(d))." MPEP 2106.05.A.
In short, it is not well-understood, routine, or conventional to configured a modeling manager to implement a plurality of different models using a common training and configuration infrastructure, nor to using the modeling manager to concurrently execute two or more models targeted at different sets of users and evaluating a performance of each of the concurrently executed models.
B. CONCLUSION
Therefore, because the claims at issue recite limitations that the courts have found enough to qualify as "significantly more" when recited with a judicial exception, the claims at issue include additional elements that amount to an inventive concept. Accordingly, it is respectfully requested that the Examiner reconsiders and withdraws the rejection of the claims under 35 U.S.C. § 101.
Examiner response
Applicant argues that the additional limitations concerning the common training configuration infrastructure and concurrent execution of multiple ML models are not well understood, routine, or conventional and therefore amount to significantly more than judicial exception. This argument has been considered but is not persuasive. As discussed in the rejection, the additional elements come individually and in combination, perform conventional machine learning model configuration, execution, experimentation, and evaluation functions at a high level of generality and do not provide an inventive concept beyond the judicial exception. Accordingly, the additional elements do not amount to significantly more under Step 2B.
Regarding the 103 arguments
Applicant argues
Respectfully, Bowers does not disclose, or even vaguely suggest, that a modeling manager can be configured with common training and configuration infrastructures for configuring a plurality of ML models. Instead, Bowers specifically discloses that a workflow is generated for each experience, which requires, at minimum, building and testing the respective models (refer, e.g., to Col. 1, lines 30-42 and Col. 7, lines 59-65, reproduced below for convenience).
''A typical machine learning workflow may include building a model from a sample dataset (referred to as a ''training set''), evaluating the model against one or more additional sample datasets (referred to as a ''validation set'' and/or a ''test set'') to decide whether to keep the model and to benchmark how good the model is, and using the model in ''production'' to make predictions or decisions against live input data captured by an application service. The training set, the validation set, and/or the test set can respectively include pairs of input datasets and expected output datasets that correspond to the respective input datasets." (Col. 1, lines 30-42).
…
A workflow of an experiment can include preprocessing of an input dataset, training a machine learning model, validating the machine learning model, processing a test dataset through the machine learning model to compute test results, post-processing the test results for analysis, or any combination thereof. In some examples, the workflow can include post-processing the input dataset for analysis. Post-processing for analysis can include computing statistical measures, computing comparative measures (e.g., between the test results and expected results), computing an evaluative measure (e.g., based on an evaluation algorithm), or any combination thereof. (Col. 7, lines 59-65).
In other words, Bowers is generally describing the conventional model serving architecture described in paragraph 92 of Applicant's specification, and over which the present Application offers an improvement.
Examiner Response
The rejection does not rely solely on the fact that an individual workflow may include training and testing a machine learning model. Rather, Bowers teaches a machine learning system, corresponding to the claimed modeling manager, that includes shared workflow authoring and workflow execution components through which workflows used to train, test, and evaluate ML models are authored, configured, and executed. Bowers further teaches that processing operators are reusable across workflows, that workflows may be repeatedly executed in different experiments, and that an experiment may include multiple workflows. Thus, although respective ML models may be associated with the respective workflows, the workflows are created and executed through the same machine learning system and shared components. Accordingly, Bowers suggests a common training and configuration infrastructure for configuring a plurality of ML models.
Regarding the rest of the arguments
Applicant’s arguments with respect to claim(s) 1, 11, and 16 have been considered but are moot because the amendments necessitated the new ground of rejection. Applicant’s arguments are directed to the rejection set forth in the prior Office Action, whereas the present rejection has been modified to account for the amened claim language and further relies on the new ground of rejection for the amended language.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
To determine if a claim is directed to patent ineligible subject matter, the Court has guided the Office to apply the Alice/Mayo test, which requires:
Step 1: Determining if the claim falls within a statutory category.
Step 2A: Determining if the claim is directed to a patent ineligible judicial exception consisting of a law of nature, a natural phenomenon, or abstract idea; and Step 2A is a two prong inquiry. MPEP 2106.04(II)(A). Under the first prong, examiners evaluate whether a law of nature, natural phenomenon, or abstract idea is set forth or described in the claim. Abstract ideas include mathematical concepts, certain methods of organizing human activity, and mental processes. MPEP 2104.04(a)(2). The second prong is an inquiry into whether the claim integrates a judicial exception into a practical application. MPEP 2106.04(d).
Step 2B: If the claim is directed to a judicial exception, determining if the claim recites limitations or elements that amount to significantly more than the judicial exception. (See MPEP 2106).
Claims 1-20 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-10 are directed to a method (a process), Claims 11-15 are directed to a system comprising one or more processors (a machine), and Claims 16-20 are directed to a non-transitory machine-readable storage medium (a manufacture). Therefore, Claims 1-20 are directed to a process, machine or manufacture or composition of matter.
Regarding claim 1
Step 2A Prong 1
Claim 1 recites the following mental processes, that in each case under the broadest reasonable interpretation, covers performance of the limitation in the mind (including observation, evaluation, judgement, opinion) or with the aid of pencil and paper but for recitation of generic computer components (e.g., “modeling manager”, “experience module”, “machine-learning models”, and “user interface”) [see MPEP 2106.04(a)(2)(III)].
“configuring … for the experiment based on the parameter values entered on the first UI” (e.g., a human can enter values into a spreadsheet)
“selecting, … one of the configured ML models for providing a response to the request based on the evaluated performance” (e.g., a human can select a completed process based on desired results, evaluating information and selecting an alternative based on the evaluated results)
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of “modeling manager”, “experience module”, “machine-learning models”, and “user interface” which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)). The Examiner notes that this is used throughout the claim limitations, and is rejected thusly for each claim which recites the same language.
Regarding the “receiving, by a modeling manager, a schema from an experience module, the experience module implementing one or more features of an online service, the schema being a data structure that defines variables for an experiment, the modeling manager managing a plurality of machine-learning (ML) models” this additional element is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of inputting data for use in the claimed process (see MPEP 2106.05(g)). The examiner notes that “the experience module implementing one or more features of an online service”, “the schema being a data structure that defines variables for an experiment”, and “the modeling manager managing a plurality of machine-learning (ML) models” is merely defining where the data is derived from, defining the data structure that is being received for the process, and stating there are multiple machine learning models which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)).
Regarding the “providing, by the modeling manager, a first user interface (UI) based on the schema for entering parameter values for the experiment” limitation, which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)). The Examiner notes that this is used to collect data for the machine learning process, and could be interpreted as data gathering (MPEP 2106.05(g)).
Regarding the “wherein the modeling manager comprises a common training and configuration infrastructure for configuring the plurality of ML models” which is recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)). The limitation merely requires that the recited configuring of the plurality of ML models be performed using a common training and configuration infrastructure, without reciting a particular technological mechanism, by which the infrastructure performs the configuring. Thus, the limitation amounts to using generic computer infrastructure as a tool to perform the recited functionality.
Regarding the “initializing the experiment, the initializing comprising using the modeling manager to concurrently execute two or more models targeted at different sets of users and evaluating a performance of each of the concurrently executed models” limitation, which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)). The limitation is using the modeling manager to concurrently execute tow or more models targeted at different sets of users foes not integrate the execution into a practical application because it recites, at a high functional level, using the modeling manger and ML models as computer tools for carrying out the recited experimental evaluation, without reciting a particular technological mechanism by which the concurrent execution or targeting is accomplished.
Regarding the “during the experiment, receiving, by the modeling manager, a request from the experience module for data associated with the experiment”, and “getting the response from the selected ML model based on input provided to the ML model based on the request”, these additional elements are recited at a high level of generality and amount to extra-solution activity of inquiring data during a process and outputting data, i.e. post-solution activity of data gathering for use in the claimed process (see MPEP 2106.05(g)).
Regarding the “getting the response from the selected ML model based on input provided to the ML model based on the request” limitation, this additional element is recited at a high level of generality and amounts to extra-solution activity of transmitting machine learning results, i.e. post-solution activity of data outputting (see MPEP 2106.05(g)).
Regarding the “sending, by the modeling manager, the response to the experience module”, and “providing a second UI for presenting results of the experiment” limitations, which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of a “modeling manager”, “experience module”, “machine-learning models”, and “user interface” which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Regarding the “receiving, by a modeling manager, a schema from an experience module, the experience module implementing one or more features of an online service, the schema being a data structure that defines variables for an experiment, the modeling manager managing a plurality of machine-learning (ML) models” this additional element is recited at a high level of generality and amounts to extra-solution pre-solution activity of inputting data. Regarding the “during the experiment, receiving, by the modeling manager, a request from the experience module for data associated with the experiment”, and “getting the response from the selected ML model based on input provided to the ML model based on the request”, these additional elements are recited at a high level of generality and amount to extra-solution post-solution activity of data gathering. Regarding the “getting the response from the selected ML model based on input provided to the ML model based on the request” this additional element is recited at a high level of generality and amounts to extra-solution post-solution activity of data outputting. The courts have found limitations directed to obtaining information electronically, recited at a high-level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding the “providing, by the modeling manager, a first user interface (UI) based on the schema for entering parameter values for the experiment” limitation, which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)).
Regarding the “wherein the modeling manager comprises a common training and configuration infrastructure for configuring the plurality of ML models” which is recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)). The limitation merely requires that the recited configuring of the plurality of ML models be performed using a common training and configuration infrastructure, without reciting a particular technological mechanism, by which the infrastructure performs the configuring. Thus, the limitation amounts to using generic computer infrastructure as a tool to perform the recited functionality.
Regarding the “initializing the experiment, the initializing comprising using the modeling manager to concurrently execute two or more models targeted at different sets of users and evaluating a performance of each of the concurrently executed models” limitation, which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)).
Regarding the “sending, by the modeling manager, the response to the experience”, and “providing a second UI for presenting results of the experiment” limitations, which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional element individually or in combination does not amount to significantly more than the judicial exception.
Regarding claim 2 (Canceled)
Regarding claim 3
Step 2A Prong 1
Claim 3 does not recite an abstract idea, but is directed to the abstract idea identified in its parents claim(s).
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of “assigning a percentage of requests served by each of the models during the experiment” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of a “assigning a percentage of requests served by each of the models during the experiment” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).In particular it is merely describing how the data is labeled for use in the claimed process.
Accordingly, at Step 2B, the additional element individually or in combination does not amount to significantly more than the judicial exception.
Regarding claim 4
Step 2A Prong 1
Claim 4 does not recite an abstract idea, but is directed to the abstract idea identified in its parents claim(s).
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of “wherein the experiment is for defining text for a notification to be sent to a user, wherein each of the configured ML models provides the text for the notification based on user identification (ID) and segment ID” which is recited at a high-level of generality such that they amount to no more than generally linking the use of abstract idea to a particular technological environment or field of use using a generic computer component (See MPEP 2106.05(h)). In particular it is merely describing how the data is labeled for use in the claimed process.
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of a “wherein the experiment is for defining text for a notification to be sent to a user, wherein each of the configured ML models provides the text for the notification based on user identification (ID) and segment ID” which is recited at a high-level of generality such that they amount to no more than generally linking the use of abstract idea to a particular technological environment or field of use using a generic computer component (See MPEP 2106.05(h)). In particular it is merely describing how the data is labeled for use in the claimed process.
Accordingly, at Step 2B, the additional element individually or in combination does not amount to significantly more than the judicial exception.
Regarding claim 5
Step 2A Prong 1
Claim 4 does not recite an abstract idea, but is directed to the abstract idea identified in its parents claim(s).
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of “wherein the experiment is for providing multiple options for text on a webpage, wherein the schema defines a control value and one or more variants as the multiple options for the text on the webpage” which is recited at a high-level of generality such that they amount to no more than generally linking the use of abstract idea to a particular technological environment or field of use using a generic computer component (See MPEP 2106.05(h)). In particular it is merely describing how the data is labeled for use in the claimed process.
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of a “wherein the experiment is for providing multiple options for text on a webpage, wherein the schema defines a control value and one or more variants as the multiple options for the text on the webpage” which is recited at a high-level of generality such that they amount to no more than generally linking the use of abstract idea to a particular technological environment or field of use using a generic computer component (See MPEP 2106.05(h)). In particular it is merely describing how the data is labeled for use in the claimed process.
Accordingly, at Step 2B, the additional element individually or in combination does not amount to significantly more than the judicial exception.
Regarding claim 6
Step 2A Prong 1
Claim 6 does not recite an abstract idea, but is directed to the abstract idea identified in its parents claim(s).
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of “wherein the first UI is provided as a browser extension that provides a toolbar presented with a user feed webpage” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of a “wherein the first UI is provided as a browser extension that provides a toolbar presented with a user feed webpage” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional element individually or in combination does not amount to significantly more than the judicial exception.
Regarding claim 7
Step 2A Prong 1
Claim 7 does not recite an abstract idea, but is directed to the abstract idea identified in its parents claim(s).
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of “notifying an experiment tracking system of a configuration for the experiment, wherein the second UI is provided by the experiment tracking system” which is recited at a high-level of generality such that it amounts to extra-solution activity of reporting/logging/sending data to a system and presenting results, i.e. post-solution activity of data outputting for use in the claimed process (see MPEP 2106.05(g)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of a “notifying an experiment tracking system of a configuration for the experiment, wherein the second UI is provided by the experiment tracking system” limitation, the additional element is recited at a high-level of generality and amounts to extra-solution activity of post-solution activity of data outputting. The courts have found limitations directed to obtaining information electronically, recited at a high-level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Accordingly, at Step 2B, the additional element individually or in combination does not amount to significantly more than the judicial exception.
Regarding claim 8
Step 2A Prong 1
Claim 8 does not recite an abstract idea, but is directed to the abstract idea identified in its parents claim(s).
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of “wherein the request comprises a user identifier (ID) of a user associated with a communication being sent to the user” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of a “wherein the request comprises a user identifier (ID) of a user associated with a communication being sent to the user” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional element individually or in combination does not amount to significantly more than the judicial exception.
Regarding claim 9
Step 2A Prong 1
Claim 9 does not recite an abstract idea, but is directed to the abstract idea identified in its parents claim(s).
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of “wherein the request comprises a segment (ID) for a segment of users” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of a “wherein the request comprises a segment (ID) for a segment of users” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional element individually or in combination does not amount to significantly more than the judicial exception.
Regarding claim 10
Step 2A Prong 1
Claim 10 does not recite an abstract idea, but is directed to the abstract idea identified in its parents claim(s).
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of “wherein the modeling manager manages training of the plurality of ML models, wherein the plurality of ML models is available to a plurality of experience modules” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of a “wherein the modeling manager manages training of the plurality of ML models, wherein the plurality of ML models is available to a plurality of experience modules” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional element individually or in combination does not amount to significantly more than the judicial exception.
Regarding claims 11-15, and 16-20
Claims 11-15, and 16-20 recites a system comprising one or more processors, and a non-transitory machine-readable storage medium , respectively. The addition of generic computer components executing instructions are insufficient to render the claims subject matter eligible for the same reasons as described above. Specifically:
Claim 11 corresponds to claim 1, with the added recitation of generic computer components to execute instructions to perform the same abstract method steps of claim 1.
Claim 13 corresponds to claim 3, with the added recitation of generic computer components to execute instructions to perform the same abstract method steps of claim 3.
Claim 14 corresponds to claim 4, with the added recitation of generic computer components to execute instructions to perform the same abstract method steps of claim 4.
Claim 15 corresponds to claim 5, with the added recitation of generic computer components to execute instructions to perform the same abstract method steps of claim 5.
Claim 16 corresponds to claim 1, with the added recitation of generic computer components to execute instructions to perform the same abstract method steps of claim 1.
Claim 18 corresponds to claim 3, with the added recitation of generic computer components to execute instructions to perform the same abstract method steps of claim 3.
Claim 19 corresponds to claim 4, with the added recitation of generic computer components to execute instructions to perform the same abstract method steps of claim 4.
Claim 20 corresponds to claim 5, with the added recitation of generic computer components to execute instructions to perform the same abstract method steps of claim 5.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-5, and 7-11, 13-16, and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bowers et al. (US 10417577 B2, referred to as Bowers), in view of Kohavi et al. ( “Online Controlled Experiments at Large Scale”, referred to as Kohavi), in view of Shariat et al. ("Online model evaluation in a large-scale computational advertising platform." , referred to as Shariat).
Regarding claim 1, Bowers teaches, a computer-implemented method comprising:
receiving, by a modeling manager, a schema from an experience module, the experience module implementing one or more features of an online service, the schema being a data structure that defines variables for an experiment, the modeling manager managing a plurality of machine-learning (ML) models (Col. 2, lines 65-67, cont. Col. 3, lines 1-37: Describes an input schema and an output schema for workflow/experiment.; Col. 4, lines 4-35: Describes an application service system providing application services via API/Web server/mobile service server, processing client requests in real time.; Col.4, lines 63-67 cont. Col.5, lines 1-14: Describes an experiment management engine that defines experiments/workflows and parameters. These correspond to receiving/using a schema for an experiment in an online-service context as the application service system (experience module) provides online application services via API/web/mobile servers and handles real-time client requests (live traffic). The machine learning system supports experiments defined as workflows, where workflows are configured to process input datasets consistent with an input schema and generate outputs consistent with an output schema. The schema is a data structure defining the variables/fields of experiment inputs (and outputs). The workflows operate on “input data” and “output data form the machine learning models”, which manages/uses the ML model(s) within the experiment framework.);
Although Bowers teaches receiving, by a modeling manager, a schema…, the experience module implementing one or more features of an online service, the schema being a data structure that defines variables for an experiment, the modeling manager managing a plurality of machine-learning (ML) models
Kohavi teaches receiving... from an experience module(Page 1173 Section 4.1: Describes an online experimentation system for an online service in which experiments are driven by configuration settings. “All systems in Bing are driven from configuration and an experiment is implemented as a change to the default configuration”, and further describes “A configuration API and tool enables experimenters to easily create the setting defining an experiment”. The setting defining an experiment corresponds to a schema/data structure defining variables for an experiment. The online service components providing user-facing features to implement multiple features online and the experiment management system consumes the configuration setting receives the schema for execution/management of the experiment.)
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined Bowers machine learning experiment management framework with Kohavi’s configuration driven online experimentation. Doing so would have enabled the system to create a scalable, modular definition of experiment variables for online-service features and to support efficient creation and execution of experiments using standardized configuration interfaces.
Bowers in view of Kohavi, further teaches, providing, by the modeling manager, a first user interface (UI) based on the schema for entering parameter values for the experiment (Col. 10, lines 64-67 cont. Col. 11, lines 1-39 and FIG. 2: Describes a “definition interface” where the system presents a website to a user to enter parameters for an experiment. Which corresponds to giving a user a user interface for the current schema.);
configuring one or more ML models from the plurality of ML models for the experiment based on the parameter values entered on the first UI (Bowers Col. 3, lines 19-37: Describes a workflow that includes data processing operators which perform training and evaluation of models.; Col. 10, lines 49-63: Describes that the system manages a repository, where it stores “one or more previously executed or created or currently running experiments”.; Col. 12 lines 35-67 cont. Col. 13, lines 1-4: Describes that the execution scheduler executes workflow runs defined by parameters. These correspond to parameter values being entered from a UI, which are used to configure workflow execution which includes machine learning models.) wherein the modeling manager comprises a common training and configuration infrastructure for configuring the plurality of ML models (Bowers Col. 2, lines23-64: Describes a machine learning system, corresponding to the modeling manager, that includes an experiment management engine, workflow authoring tool, and workflow execution engine for designing and executing machine learning processes, including training and testing machine learning models.; Col. 3, lines 38-58: Describes reusable data processing operator types that may be used across different workflows.; Col. 4 lines 36-63: Describes that a workflow may be executed in different experiments and that an experiment may include multiple workflows.; Col. 6, lines 12-50: Describes that the machine learning system provides a platform for running ml experiments and includes the experiment management engine, workflow authoring tool, and Workflow execution engine. These components form one shared platform used to create and execute multiple ml experiments.; Col. 7, lines 59 cont. Col 8, lines 1-4: Describes that a workflow may train, validate, test, and evaluate an ML model. This corresponds to the same machine learning system provides common training and configuration infrastructure through which multiple ML workflows and their associated ML models are configured and trained.);
initializing the experiment (Bowers Col. 12, lines 35-67 cont. Col. 13, lines 1-4: Describes scheduling the workflow run, and starting the execution engine to run those workflows.)
Although Bowers in view of Kohavi, teaches, initializing the experiment, they do not teach the initializing comprising using the modeling manager to concurrently execute two or more models targeted at different sets of users and evaluating a performance of each of the concurrently executed models.
Shariat teaches the initializing comprising using the modeling manager to concurrently execute two or more models targeted at different sets of users and evaluating a performance of each of the concurrently executed models ( Pages 3-5, Section III: Describes conducting an online A/B experiment in which a baseline model A and a treatment model B are evaluated using live user traffic. It splits the live traffic based on user identifiers by pre-allocating a subset of users to treatment model B, such that requests associated with those users are process by model B, while requests associated with the remaining users are processed by baseline model A. The models, A and B, operate on respective portions of the live user traffic during the same experiment, corresponding to concurrently executing two models targeted at different sets of sf users. It evaluates the respective performance of models A and B using an overall evaluation criterion, including separate return on investments for each model.);
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined Bowers in view of Kohavi’s machine learning experiment management framework with Shariat’s online experimentation framework. Doing so would have enabled the system to reliably evaluate live performance of different configured machine learning models and determine whether an experimental model should replace the baseline model.
Bowers in view of Kohavi, in view of Shariat further teaches during the experiment, receiving, by the modeling manager, a request from the experience module for data associated with the experiment (Kohavi Page 1173 Section 4.1: Describes “As a request is received from a browser, Bing’s frontend servers assign each request to multiple flights” and “Each layer in the system logs information, including the request’s flight assignments, to system logs that are then processed and used for offline analysis”. These show that the online service receives request, the assignment system routes those request and then the experiment system processes those requests. It further details that “All systems in Bing are driven from configuration and an experiment is implemented as a change to the default configuration”, which shows runtime interaction between the service and the experiment system.);
selecting, by the modeling manager, one of the configured ML models for providing a response to the request (Bowers Col. 7 lines 15-49 and Col. 12, lines 47-67 cont. Col. 13, lines 1-28: Describes that workflows include data processing operators implementing machine learning functionality and that the workflow execution engine executes workflow runs. The scheduler determines which configured workflow run is executed and workflows include machine learning operators. This corresponds to selecting one of the configured machine learning models for execution in response to a request. ) based on the evaluated performance (Shariat Pages 3-5, Figure 2 and Section III: Describes evaluating the relative performance of a baseline model A and an experimental model B during an online experiment. It increases the portion of incoming traffic processed by experimental model B as evidence is collected that model B performs better than baseline model A. The evaluation pipeline determines whether to accept the experimental model based on the evaluated performance and, when accepted, increases traffic to model B until model B becomes dominant and replaces the baseline model. Corresponding to selecting a model for processing subsequent requests based on the evaluated performance of the models.);
getting the response from the selected ML model based on input provided to the ML model based on the request (Bowers Col. 2 lines 65-67 cont. Col. 3, lines 1-37: Describes that workflows process one or more outputs and that machine learning models generate output data. This corresponds to getting a response from a selected machine learning model based on input provided to that model.);
sending, by the modeling manager, the response to the experience module (Kohavi Page 1173 Section 4.1: Describes an online service architecture in which frontend servers receive browser requests and route those requests through an experimentation system. Because the experimentation system modifies system behavior based on experiment configuration, the processed result is returned to the frontend for delivery to the user.); and
providing a second UI for presenting results of the experiment(Bowers Col. 3, lines 59-67 cont. Col. 4 lines 1-3 and FIG. 7B-F, Col. 15, lines 8-58: Describes generating and presenting experiment results via automatically generated visualizations displayed through a user interfaces. These visualizations are presented through a user distinct from the parameter definition interface. This corresponds to providing a second UI for presenting results of the experiment.).
Regarding claim 2 (cancelled)
Regarding claim 3, Bowers in view of Kohavi, in view of Shariat teaches, the method as recited in claim 1.
Kohavi further teaches, wherein configuring the one or more models comprises:
assigning a percentage of requests served by each of the models during the experiment (Page 1168-1169, Sections 1 and 1.1, and Page 1173, Section 4.1: Describes configuring an experiment by allocating traffic among variants. In controlled experiments, “users are randomly split between the variants” and provides “an experiment utilizing 20% of eligible users (10% control, 10% treatment)” which assigns percentages to the respective variants. Kohavi further describes request-level routing, where “As a request is received from a browser, Bing’s frontend servers assign each request to multiple flights”, where a flight is a variant to which a user/.request is exposed. The variants include backend components such as “relevance rankers” (model-type components) that are experimented with, configuring models/variants by assigning a percentage of requests/traffic served by each model during the experiment.).
Regarding claim 4, Bowers in view of Kohavi, in view of Shariat teaches, the method as recited in claim 1.
Bowers in view of Kohavi, further teaches, wherein the experiment is for defining text for a notification to be sent to a user (Kohavi Page 1169, Section 1.1: Describes running online controlled experiments where users are split between variants that provide different user-facing content, including textual changes such as “improving search result captions” and different ads layouts between control and treatment. This corresponds to an experiment used to define/modify text presented to users. Kohavi does not expressly teach a notification, rather it teaches defining/altering user-facing text content generally.), wherein each of the configured ML models provides the text for the notification (Bowser, Col. 2, lines 48-67 cont. Col. 13, lines 1-18:Describes that workflows within an experiment “utilize one or more machine learning models”, including “post-processing of output data from the machine learning models”, and that experiments/workflows process input datasets into outputs. This corresponds to configured machine learning models producing outputs during an experiment. Where the experiment output is text content, each configured machine learning model provides the text output for the user facing content) based on user identification (ID) and segment ID (Kohavi Page 1173, Section 4.1, and Pages 1174-1175, Section 5.1: Describes that experiment assignment is performed consistently using a hash of an :”anonymous user id” and further describes analyzing impacts on “specific user segments”. This corresponds to using user identification in assignment and using segmentation in the experimentation context, where segmentation is used for analysis/handling of cohorts.).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined Bower’s experiment management engine, machine learning infrastructure, with Kohavi’s online controlled experimentation. Doing so would enable the systems workflow/models to generate experiment outputs that are served to users under controlled traffic assignments, improving the ability to run, manage and analyze large-scale experiments in an online service.
Regarding claim 5, Bowers in view of Kohavi, in view of Shariat teaches, the method as recited in claim 1.
Bowers in view of Kohavi, further teaches, wherein the experiment is for providing multiple options for text on a webpage (Kohavi, Page 1169 Section 1.1: Describes running online controlled experiments in which different variants present different user-visible content on webpages, including textual modifications such as improving search result captions. Users are randomly split between variants that provide different webpage layouts or textual content.), wherein the schema defines a control value and one or more variants as the multiple options for the text on the webpage (Kohavi, Pages 1168-1169 Section 1, and 1.1: Describes controlled experiments that include a control and one or more treatment variants. ; Bowers, Col. 3, lines 19-37: Describes defining experiment parameters via structured schemas associated with workflows, where input dataset are processed consistent with an input schema. ).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined Bower’s schema-based framework, with Kohavi’s webpage experiments. Doing so would enable the system to control value and one or more variants as multiple options for webpage text.
Regarding claim 7, Bowers in view of Kohavi, in view of Shariat teaches, the method as recited in claim 1.
Bowers in view of Kohavi, further teaches, wherein initializing the experiment comprises:
notifying an experiment tracking system of a configuration for the experiment (Bowers, Col. 4, lines 63-67 cont. Col. 5 lines 1-40 and Col. 6, lines 28-50:Describes an experiment management engine that manages experiments and workflows within a machine learning system. A definition interface enables a user to define parameters associated with a new workflow run, and that experiment information is stored in repositories accessible by the experiment management interface. When a workflow run is created and initiated, the experiment configuration/definition is communicated to and managed by the experiment management engine.), wherein the second UI is provided by the experiment tracking system (Kohavi, Page 1173, Section 4.1: Describes an experiment management system (Control Tower) and an offline analysis pipeline that generates scorecards summarizing experiment results. The scorecards constitute a results presentation interface(a second UI) provided by the experiment tracking/management/analysis system. ).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined Bower’s experiment management engine, with Kohavi’s experiment tracking and results reporting. Doing so would enable the system to initialize an experiment by registering/notifying an experiment tracking system of the experiment’s configuration and provide results interface via the experiment tracking system to present experiment outcomes.
Regarding claim 8, Bowers in view of Kohavi, in view of Shariat teaches, the method as recited in claim 1.
Bowers in view of Kohavi, further teaches, wherein the request comprises a user identifier (ID) of a user associated with a communication being sent to the user (Kohavi Page 1169, Section 1.1, and Page 1173, Section 4.1: Descries that in a controlled online experiments, a pseudo-random hash of an “anonymous user id” is used to ensure consistent assignment of users to experimental variants. Requests are received form browsers in the online system. A requested is processed during an experiment includes a user identifier used for assignment to experimental variants. Online controlled experiments involve processing browser requests and delivering user-facing content based on experimental assignment. Because the system processes requests associated with a particular user identifier and delivers corresponding content to that user, the request comprises a user ID associated with a communication (the delivered content)sent to the user.).
Regarding claim 9, Bowers in view of Kohavi, in view of Shariat teaches, the method as recited in claim 1.
Bowers in view of Kohavi, further teaches, wherein the request comprises a segment (ID) for a segment of users (Kohavi Page 1173, Section 4.1, and Pages 1174-1175, Section 5.1: Describes conducting online controlled experiments that analyze and evaluate impacts on specific user segments, disclosing segmentation of users within the experimentation framework. It processes request associated with users and assigns users to experimental variants. Because segmentation is used tin the experimentation system, this corresponds to requests being processed within the experiment, which include information identifying the user’s segment in order to enable the segment-based assignment and analysis.).
Regarding claim 10, Bowers in view of Kohavi, in view of Shariat teaches, the method as recited in claim 1.
Bowers in view of Kohavi, further teaches, wherein the modeling manager manages training of the plurality of ML models (Bowers, Col. 2, lines 65-67 cont. Col. 3, lines 1-18, and Col.6, lines 28-58: Describes that workflows within the machine learning system create, modify, evaluate, validate, and/or utilize one or more machine learning models, and that the experiment management engine manages such workflows. Because creating and modifying machine learning models includes training operations, this correspond to the modeling manager managing training of a plurality of machine learning models.), wherein the plurality of ML models is available to a plurality of experience modules (Bowers, Col. 5, lines 37-67 cont. Col. 6, lines 1-27: Describes an application service system including multiple application services (application services 102A and 102B) that interact with a centralized machine learning system. The machine learning system includes workflows that create and utilize multiple machine learning models, and because the machine learning system operates within and supports multiple application services, the plurality of machine learning models is available to a plurality of experience modules (application services).).
Regarding claims 11, and 13-15, which recites substantially the same limitations as claims 1, and 3-5 and further recites a system comprising: a memory comprising instructions; and one or more computer processors (Bowers Col. 13, lines 52-67 cont. Col. 16, lines 1-15: Describes that the system can comprise of generic computer hardware including processors and software to execute the instructions of their methods.) to perform the method steps of claims 1-5, respectively, and are rejected for the same reasons as described above.
Regarding claims 16, and 18-20, which recites substantially the same limitations as claims 1, and 3-5 and further recites a non-transitory machine-readable storage medium including instructions (Bowers Col. 13, lines 52-67 cont. Col. 16, lines 1-15: Describes that the system can comprise of generic computer hardware including “volatile memory may be considered “non-transitory” in the sense that it is not a transitory signal” and software to execute the instructions of their methods.) to perform the method steps of claims 1-5, respectively, and are rejected for the same reasons as described above.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bowers et al. (US 10417577 B2, referred to as Bowers), in view of Kohavi et al. ( “Online Controlled Experiments at Large Scale”, referred to as Kohavi),in view of Shariat et al. ("Online model evaluation in a large-scale computational advertising platform." , referred to as Shariat), in view of Mozilla Developer Network ("Browser actions", referred to as MDN).
Regarding claim 6, Bowers in view of Kohavi, in view of Shariat teaches, the method as recited in claim 1.
Bowers in view of Kohavi, in view of Shariat, further teaches, wherein the first UI is provided as a browser extension that provides a toolbar presented with a user feed webpage (Bowers, Col. 2, lines 48-67 cont. Col. 3, lines 1-18: Describes managing experiments and workflows via a user interface executed within an online system environment.; Kohavi, Page 1173, Section 4.1: Describes online controlled experimentation systems operating within web-based frontend architectures. Providing the first user interface as a browser extension that represents a toolbar over a user feed webpage.).
Although Bowers in view of Kohavi, in view of Shariat teaches a first UI is provided ...that provides a toolbar presented with a user feed webpage. They do not teach, as a browser extension.
MDN teaches, as a browser extension ( Describes “A browser action is a button you can add to the browser toolbar. Users can click the button to interact with your extension” the browser action is a toolbar button presented in the browser UI while webpages are displayed, the toolbar is presented with a user feed webpage when the user navigates to such a webpage in the browser)
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined Bower’s configuration interface, with MDN’s browser extension toolbar. Doing so would have enabled the system to improve accessibility and usability by allowing the user to invoke the UI while viewing a target webpage without modifying the underlaying webpage or requiring a sperate application window.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONALD T RODEN whose telephone number is (571)272-6441. The examiner can normally be reached Mon-Thur 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at (571) 272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.T.R./Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights