Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicant
The following is a Non-Final Office action. In response to Examiner’s Final Rejection of 12/29/2025, Applicant, on 3/30/2026, amended claims 1, 4, 14, 17 and 20; cancelled claims 5 and 18; added claims 21-22 . Claims 1-4, 6-17, and 19-22 are pending in this application and have been rejected below.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this
application is eligible for continued examination under 37 CFR 1.114, and the fee set
forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on
3/30/2026 has been entered.
Response to Arguments
Applicant’s arguments filed March 30, 2026 have been fully considered but they are not persuasive and/or are moot in view of the revised rejections. Applicant’s arguments will be addressed herein below in the order in which they appear in the response filed March 30, 2026.
On Pgs. 7-11, regarding the 35 U.S.C. § 103 rejection, Applicant states Crabtree reference to be available as prior art as of an earlier parent application's filing date, the subject matter relied upon must be supported by the disclosure of that parent application (see, e.g., MPEP § 2154.01(b)) requiring "that a prior-filed application to which a priority or benefit claim is made must describe the subject matter from the U.S. patent document relied upon in a rejection."). In response, new ground(s) of rejection is made necessitated by amendment see MPEP 706.07a where Crabtree in view of Achin is now applied for Claims 1, 14 and 20. Regarding the 35 U.S.C. § 103 rejection, Applicant’s arguments with respect to claims has been considered but are moot in view of the new grounds of rejection.
On Pgs. 12-14, regarding the 35 U.S.C. § 101 rejection, the rejection has been withdrawn
Reasons for Eligibility under 35 USC 101
The reasons for withdrawal of the rejection of claims 1-4, 6-17, and 19-22 under 35 U.S.C. 101 can be found below:
Based on current claim language and USPTO Guidance, as a whole,
the claim limitations that are indicative of integration into a practical application
when recited in a claim with a judicial exception include: at 2A, Prong 2 – improving
computing technology; solution rooted in computing technology - based on
provide information for configuring simulators on the different systems, each simulator representing at least the machine learning model of a given one of the different systems;
perform multiple iterations of the simulation for the plurality of policies, wherein, for each iteration, the central simulation system: (i) predicts a state of the target population, (ii) provides the state of the target population to the simulators, and (iii) collects one or more metrics based on results of the simulators;
and select and send one of the policies to at least one of the different systems based on the collected metrics, wherein the at least one system updates its corresponding machine learning model based at least in part on the selected policy, and wherein, during the simulation, at least one of: (i) source code corresponding to at least one of the machine learning models is not shared with the central simulation system, and (ii) a dataset used to train at least one of the machine learning models is not shared with the central simulation system.
Thus, the limitations are applying or using the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular
technological environment.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention
pertains. Patentability shall not be negated by the manner in which the invention was made.
1-4, 6-17, and 19-22 are rejected under 35 U.S.C. 103 as being unpatentable over Crabtree et al., US Publication No. 20210209505A1, [hereinafter Crabtree], in view of Achin et al., US Publication No. 20220199266A1, [hereinafter Achin].
Regarding Claim 1,
Crabtree teaches
A computer-implemented method, the method comprising: obtaining, by a central simulation system, a plurality of policies to be used for performing a simulation involving multiple machine learning models interacting with a target population, wherein the machine learning models are implemented on different systems (Crabtree Par. 29- According to one embodiment, a distributed system for accurate and detailed modeling of systems with large and complex datasets using a distributed simulation engine employing artificial intelligence/machine learning algorithms is disclosed. The system further uses results of information analytics to optimize the making of decisions and allow for alternate action pathways to be simulated using the latest data and machine-mediated prediction algorithms. Par. 55- Simulation engine 1007 may initialize a simulation world which is a JSON schema defining a model with entities, behaviors, and related schema objects. A simulation policy may comprise a JSON schema for a policy. A policy may be associated to an account and may be required to define a simulation for execution and further sets the initialize file to use, simulation times, and other parameters. A policy defines how the simulation instance for a model should run, it may be implemented as a step in a pipeline workflow integrating a simulation execution according to a preferred embodiment.”; Par.56;60; 62)
providing information for configuring simulators on the different systems, each simulator representing at least the machine learning model of a given one of the different systems; (Crabtree Par. 56- According to one embodiment, simulation engine 1007 may use models of discrete event-based systems and models of continuous event-based systems. Simulation models reproduce behavioral traits of the discrete and continuous system elements. Each instance of a model is called an actor or agent. Each simulation instance involves actors who interact within an environmental scenario which is implemented as a simulation case. Environmental scenarios are managed via cells capable of representing physical or spatial state. Technical components enable the underlying simulator to operate on the models to provide useful evaluations of behavior or performance in the context of simulation cases.”)
performing multiple iterations of the simulation for the plurality of policies, wherein, for each iteration, the central simulation system: (i) predicts a state of the target population, (ii) provides the state of the target population to the simulators, and (iii) collects one or more metrics based on results of the simulators; (Crabtree Par. 40;46; Par. 58-“ Simulation engine 1007 in combination with a model engine 1006 may use a blended modeling approach by integrating both analytical methods and system model extraction for simulations. Past observations from one or more analytical methods (e.g., regression testing, neural networks, Bayesian learning, support vector machines, etc.) coupled with retrospective outcome data (which may abide by the rule that the outcome is valid only if past conditions match future conditions) allow the system to extract models and their subcomponents. Model engine 1006 may further explore alternative models via web-scraping, natural language processing, neural networks, and other informative or rules-based computing methods. According to one embodiment, model engine 1006 may establish the various models and subcomponents by using an Advanced Cyber-Decision Platform (ACDP). The numerous system models extracted provide simulation engine 1007 with the means to iteratively learn about casual relationships thus leading to a set of probable future outcomes. Additionally, simulation engine 1007 recommends continuously competing combinations of approaches and informs/assists users in decision making.;)
and selecting and sending one of the policies to at least one of the different systems based on the collected metrics, wherein the at least one system updates its corresponding machine learning model based at least in part on the selected policy…; (Crabtree Par. 67- It is sometimes useful to place individual instances of a particular model (actor) type into one or more groups to measure not only how each individual behaves in the simulation but the overall results obtained from the group. The embodiment allows this by supporting actor populations to be specified 414. Successful completion of a real-world reliable simulation may be greatly augmented by the ability to compare the results of a partial simulation run with as much of the limited real-world data as is available or possibly even incorporating portions of that real-world data into the constituent parts of the simulation such as the world configuration 412, selection of actor model characteristics 413 and types and the generation of appropriate actor populations 414. The decision operating system offers a mechanism for use of this potential advantage by allowing the simulation constituents listed to be attached as run parameters to the automated planning service module, bringing the inferential statistic and Monte Carlo heuristic algorithms into support in directing the intelligent inclusion and use of data from other modules of the decision operating system for improvement of the course taken by the simulation when run to completion 419.”)
wherein the method is carried out by at least one computing device (Crabtree Par. 87- “In some aspects, systems may be implemented on a standalone computing system. Referring now to FIG. 12, there is shown a block diagram depicting a typical exemplary architecture of one or more aspects or components thereof on a standalone computing system. Computing device 20 includes processors 21 that may run software that carry out one or more functions or applications of aspects, such as for example a client application 24.”) .
Crabtree teaches model analysis and the feature is expounded upon by Achin:
and wherein, during the simulation, at least one of: (i) source code corresponding to at least one of the machine learning models is not shared with the central simulation system, and (ii) a dataset used to train at least one of the machine learning models is not shared with the central simulation system; (Achin Par. 185- Method 400 may include a step in which the predictive model selected for the prediction problem is tuned. In some cases, deployment engine 240 provides the source code that implements the predictive model to the user, thereby enabling the user to tune the predictive model. However, disclosing a predictive model's source code may be undesirable in some cases (e.g., in cases where the predictive modeling technique or predictive model contains proprietary capabilities or information). To permit a user to tune a predictive model without exposing the model's source code, deployment engine 240 may construct human-readable rules for tuning the model's parameters based on a representation (e.g., a mathematical representation) of the predictive model, and provide the human-readable rules to the user. The user can then use the human-readable rules to tune the model's parameters without accessing the model's source code.”)
Crabtree and Achin are directed to machine learning analysis. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have improve upon data analysis of Crabtree, as taught by Achin, by utilizing additional predictive modelling steps, with a reasonable expectation of success of arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make the modification to the teachings of Crabtree with the motivation of improving predictive modelling (Achin Par. 6).
Regarding Claim 2 and Claim 15, Crabtree in view of Achin teach The computer-implemented method of claim 1,… and The computer program product of claim 14,…
wherein the multiple machine learning models each perform a common machine learning task (Crabtree Par. 29-“ According to one embodiment, a distributed system for accurate and detailed modeling of systems with large and complex datasets using a distributed simulation engine employing artificial intelligence/machine learning algorithms is disclosed. The system further uses results of information analytics to optimize the making of decisions and allow for alternate action pathways to be simulated using the latest data and machine-mediated prediction algorithms. Specifically, portions of the system are applied to the areas reliably predicting the outcomes of differential decision paths and prediction of risk to value for each set of decision choices through simulation of the progression of each decision pathway using the most current sensor data, specific programmed decision defining parameters and environment data available and then presenting that data in a format most useful to the authors of the simulation.”; Par. 58- Simulation engine 1007 in combination with a model engine 1006 may use a blended modeling approach by integrating both analytical methods and system model extraction for simulations. Past observations from one or more analytical methods (e.g., regression testing, neural networks, Bayesian learning, support vector machines, etc.) coupled with retrospective outcome data (which may abide by the rule that the outcome is valid only if past conditions match future conditions) allow the system to extract models and their subcomponents. Model engine 1006 may further explore alternative models via web-scraping, natural language processing, neural networks, and other informative or rules-based computing methods.)
Regarding Claim 3 and Claim 16, Crabtree in view of Achin teach The computer-implemented method of claim 2,… and The computer program product of claim 15,…
wherein each of the plurality of policies comprises one or more constraints on the common machine learning task (Crabtree Par.67-“ For any simulation, a reliably useful end result integrally depends on the proper configuration of the factors that will affect the actors of the simulation, forming the milieu in which they will perform their programmed actions, here designated the “world” 412. That the constraints and influences exercised by the configured factors of the world match those of the real-world under expected conditions of the simulation must be certain. Of equal importance is the proper configuration of the actors 413 modeling the real-world items within the simulation, again, reliable data concerning the behavior of each actor type and variants within actor types must closely match those of the real-world items under programmed simulation conditions. It is also important to carefully consider the selection of actor types and numbers of each type to be included in a simulation as this factor may change the outcome considerably and lead to conclusions contrary to reality if real-world proportions are far afield of those in the computer simulation. It is sometimes useful to place individual instances of a particular model (actor) type into one or more groups to measure not only how each individual behaves in the simulation but the overall results obtained from the group. The embodiment allows this by supporting actor populations to be specified 414.; Par. 75”)
Regarding Claim 4 and Claim 17, Crabtree in view of Achin teach The computer-implemented method of claim 1,… and The computer program product of claim 14,…
Crabtree teaches model analysis and the feature is expounded upon by Achin:
wherein at least one of the machine learning models is trained based at least in part on a dataset that is specific to a given one of the different systems, and wherein at least one of: (i) source code corresponding to the at least one machine learning model and (ii) the dataset that is specific to the given one of the different systems is not shared with other ones of the different systems during the simulation. (Achin Par. 12-“ A method of modeling at least one infectious disease, can include receiving, from one or more data sources, data including values associated with an occurrence of the infectious disease during a first time period, generating, using one or more models trained by a machine learning system taking as input the data from one or more of the data sources, one or more predictions from the received data for the occurrence; Par. 185”)
Crabtree and Achin are directed to machine learning analysis. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have improve upon data analysis of Crabtree, as taught by Achin, by utilizing additional predictive modelling steps, with a reasonable expectation of success of arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make the modification to the teachings of Crabtree with the motivation of improving predictive modelling (Achin Par. 6).
Regarding Claim 5 and Claim 18,- Cancelled
Regarding Claim 6 and Claim 19, Crabtree in view of Achin teach The computer-implemented method of claim 1,… and The computer program product of claim 14,…
Crabtree teaches model analysis and the feature is expounded upon by Achin:
wherein the collected metrics comprise one or more performance metrics associated with the machine learning models. (Achin Par. 221-“ The user interface 220 provides tools for monitoring and/or guiding the search of the predictive modeling space. These tools may provide insight into a prediction problem's dataset (e.g., by highlighting problematic variables in the dataset, identifying relationships between variables in the dataset, etc.), and/or insight into the results of the search. In some embodiments, data analysts may use the interface to guide the search, e.g., by specifying the metrics to be used to evaluate and compare modeling solutions, by specifying the criteria for recognizing a suitable modeling solution, etc. Thus, the user interface may be used by analysts to improve their own productivity, and/or to improve the performance of the exploration engine 210. In some embodiments, user interface 220 presents the results of the search in real-time, and permits users to guide the search (e.g., to adjust the scope of the search or the allocation of resources among the evaluations of different modeling solutions) in real-time. In some embodiments, user interface 220 provides tools for coordinating the efforts of multiple data analysts working on the same prediction problem and/or related prediction problems.”)
Crabtree and Achin are directed to machine learning analysis. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have improve upon data analysis of Crabtree, as taught by Achin, by utilizing additional predictive modelling steps, with a reasonable expectation of success of arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make the modification to the teachings of Crabtree with the motivation of improving predictive modelling (Achin Par. 6).
Regarding Claim 7, Crabtree in view of Achin teach The computer-implemented method of claim 6, comprising:…
Crabtree teaches model analysis and the feature is expounded upon by Achin:
maintaining, by the central simulation system, one or more fairness metrics for the simulation. (Achin Par. 142 A modeling technique may provide a focal point for developers and analysts to conceptualize an entire predictive modeling procedure, with all the steps expected based on the best practices in the field. In some embodiments, modeling techniques encapsulate best practices from statistical learning disciplines. Moreover, the modeling tool 300 can provide guidance in the development of high-quality techniques by, for example, providing a checklist of steps for the developer to consider and comparing the task graphs for new techniques to those of existing techniques to, for example, detect missing tasks, detect additional steps, and/or detect anomalous flows among steps.)
Crabtree and Achin are directed to machine learning analysis. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have improve upon data analysis of Crabtree, as taught by Achin, by utilizing additional predictive modelling steps, with a reasonable expectation of success of arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make the modification to the teachings of Crabtree with the motivation of improving predictive modelling (Achin Par. 6).
Regarding Claim 8, Crabtree in view of Achin teach The computer-implemented method of claim 7, comprising:…
Crabtree teaches model analysis and the feature is expounded upon by Achin:
aggregating the one or more fairness metrics and the one or more performance metrics to select the policy. (Achin Par. 118- A template may encode, for machine execution, pre-processing steps, model-fitting steps, and/or post-processing steps suitable for use with the template's predictive modeling algorithm(s). Examples of pre-processing steps include, without limitation, imputing missing values, feature engineering (e.g., one-hot encoding, splines, text mining, etc.), feature selection (e.g., dropping uninformative features, dropping highly correlated features, replacing original features by top principal components, etc.). Examples of model-fitting steps include, without limitation, algorithm selection, parameter estimation, hyper-parameter tuning, scoring, diagnostics, etc. Examples of post-processing steps include, without limitation, calibration of predictions, censoring, blending, etc.; Par. 142-143)
Crabtree and Achin are directed to machine learning analysis. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have improve upon data analysis of Crabtree, as taught by Achin, by utilizing additional predictive modelling steps, with a reasonable expectation of success of arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make the modification to the teachings of Crabtree with the motivation of improving predictive modelling (Achin Par. 6).
Regarding Claim 9,
The computer-implemented method of claim 7, comprising: obtaining real world data corresponding to a particular time of the simulation; and validating at least one of: the one or more collected metrics and the one or more fairness metrics based at least in part on the real world data. (Crabtree Par. 67-“For any simulation, a reliably useful end result integrally depends on the proper configuration of the factors that will affect the actors of the simulation, forming the milieu in which they will perform their programmed actions, here designated the “world” 412. That the constraints and influences exercised by the configured factors of the world match those of the real-world under expected conditions of the simulation must be certain. Of equal importance is the proper configuration of the actors 413 modeling the real-world items within the simulation, again, reliable data concerning the behavior of each actor type and variants within actor types must closely match those of the real-world items under programmed simulation conditions. It is also important to carefully consider the selection of actor types and numbers of each type to be included in a simulation as this factor may change the outcome considerably and lead to conclusions contrary to reality if real-world proportions are far afield of those in the computer simulation. It is sometimes useful to place individual instances of a particular model (actor) type into one or more groups to measure not only how each individual behaves in the simulation but the overall results obtained from the group. The embodiment allows this by supporting actor populations to be specified 414. Successful completion of a real-world reliable simulation may be greatly augmented by the ability to compare the results of a partial simulation run with as much of the limited real-world data as is available or possibly even incorporating portions of that real-world data into the constituent parts of the simulation such as the world configuration 412, selection of actor model characteristics 413 and types and the generation of appropriate actor populations 414.”)
Regarding Claim 10, Crabtree in view of Achin teach The computer-implemented method of claim 1, comprising:…
Crabtree teaches model analysis and the feature is expounded upon by Achin:
providing information to configure at least one other simulator of a new machine learning model; and adding the other simulator to the simulation after at least one of the iterations, wherein the adding comprises adjusting one or more parameters of one or more of the policies based on historical data associated with the new machine learning model. (Achin Par. 342; 356; Par. 415; Par. 420-“ FIG. 15F illustrates an example user interface to generate an experimental model including human subjects associated with a geographical location, further to the example model of FIG. 15E. On the Scenario Weights page, present implementations can run, by way of example, 1,000 simulations to create hyper-optimized vaccine trial strategies. Simulations can be distributed across, for example, nine different scenarios. In some implementations, the sum of all scenario weights must be 1. In the example above, if a user only wants to see results for Scenario E: Gradual Reopening (Midpoint), they can set the value to “1”. Once set, they only see the scenario weights and simulation outcomes for Scenario E. In some implementations, two types of risk are defined. These risk types can be risk of developing symptoms and risk of infection. Many factors can impact risk, including age, race, profession, and location. The Risk Type page provides solutions to account for these factors.”)
Crabtree and Achin are directed to machine learning analysis. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have improve upon data analysis of Crabtree, as taught by Achin, by utilizing additional predictive modelling steps, with a reasonable expectation of success of arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make the modification to the teachings of Crabtree with the motivation of improving predictive modelling (Achin Par. 6).
Regarding Claim 11, Crabtree in view of Achin teach The computer-implemented method of claim 1, comprising:…
Crabtree teaches model analysis and the feature is expounded upon by Achin:
wherein the simulation simulates a time period of at least one year (Achin Par. 342- Present implementations can then generate raw predictions and related output including but not limited to infection, mortality, recovery, and like predictions associated with particular geographic locations, geographic or political regions, time series windows covering at least one of days, weeks, months, or years.”).
Crabtree and Achin are directed to machine learning analysis. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have improve upon data analysis of Crabtree, as taught by Achin, by utilizing additional predictive modelling steps, with a reasonable expectation of success of arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make the modification to the teachings of Crabtree with the motivation of improving predictive modelling (Achin Par. 6).
Regarding Claim 12,
The computer-implemented method of claim 1, wherein the different systems are associated with different entities, and wherein the simulators execute at least within different private cloud environments of the different entities. (Crabtree Par.78 –“ at least some of the features or functionalities of the various aspects disclosed herein may be implemented in one or more virtualized computing environments (e.g., network computing clouds, virtual machines hosted on one or more physical computing machines, or other appropriate virtual environments).Par. 89)
Regarding Claim 13,
The computer-implemented method of claim 1, wherein software is provided as a service in a cloud environment for implementing the central simulation system. (Crabtree Par. 50-“ Modeling a simulation platform 1000 according to one aspect, allows Web clients 1009 to use the system 1000 remotely and thus includes typical components needed for such a system: a webserver 1002, a load-balancer 1003, an API gateway 1004, and a content cache 1005. In some embodiments, third party software may be used to handle all said tasks such as NGINX™ or other like applications. Further embodiments use bespoke or purpose-written applications by the implementors of the system 1000 and may only be accessible from LANs or intranets. Other combinations or embodiments may be appreciated by persons with ordinary skill in the art of computer networking and computer system implementations.; Par. 87; Par. 89;)
Regarding Claim 14,
Crabtree teaches
A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to: obtain, by a central simulation system, a plurality of policies to be used for performing a simulation involving multiple machine learning models interacting with a target population, wherein the machine learning models are implemented on different systems (Crabtree Par. 29- According to one embodiment, a distributed system for accurate and detailed modeling of systems with large and complex datasets using a distributed simulation engine employing artificial intelligence/machine learning algorithms is disclosed. The system further uses results of information analytics to optimize the making of decisions and allow for alternate action pathways to be simulated using the latest data and machine-mediated prediction algorithms. Par. 55- Simulation engine 1007 may initialize a simulation world which is a JSON schema defining a model with entities, behaviors, and related schema objects. A simulation policy may comprise a JSON schema for a policy. A policy may be associated to an account and may be required to define a simulation for execution and further sets the initialize file to use, simulation times, and other parameters. A policy defines how the simulation instance for a model should run, it may be implemented as a step in a pipeline workflow integrating a simulation execution according to a preferred embodiment.”; Par.56;60; 62; Par. 86)
provide information for configuring simulators on the different systems, each simulator representing at least the machine learning model of a given one of the different systems; (Crabtree Par. 56- According to one embodiment, simulation engine 1007 may use models of discrete event-based systems and models of continuous event-based systems. Simulation models reproduce behavioral traits of the discrete and continuous system elements. Each instance of a model is called an actor or agent. Each simulation instance involves actors who interact within an environmental scenario which is implemented as a simulation case. Environmental scenarios are managed via cells capable of representing physical or spatial state. Technical components enable the underlying simulator to operate on the models to provide useful evaluations of behavior or performance in the context of simulation cases.”)
perform multiple iterations of the simulation for the plurality of policies, wherein, for each iteration, the central simulation system: (i) predicts a state of the target population, (ii) provides the state of the target population to the simulators, and (iii) collects one or more metrics based on results of the simulators; (Crabtree Par. 40;46; Par. 58-“ Simulation engine 1007 in combination with a model engine 1006 may use a blended modeling approach by integrating both analytical methods and system model extraction for simulations. Past observations from one or more analytical methods (e.g., regression testing, neural networks, Bayesian learning, support vector machines, etc.) coupled with retrospective outcome data (which may abide by the rule that the outcome is valid only if past conditions match future conditions) allow the system to extract models and their subcomponents. Model engine 1006 may further explore alternative models via web-scraping, natural language processing, neural networks, and other informative or rules-based computing methods. According to one embodiment, model engine 1006 may establish the various models and subcomponents by using an Advanced Cyber-Decision Platform (ACDP). The numerous system models extracted provide simulation engine 1007 with the means to iteratively learn about casual relationships thus leading to a set of probable future outcomes. Additionally, simulation engine 1007 recommends continuously competing combinations of approaches and informs/assists users in decision making.;)
and select and send one of the policies to at least one of the different systems based on the collected metrics, wherein the at least one system updates its corresponding machine learning model based at least in part on the selected policy,…; (Crabtree Par. 67- It is sometimes useful to place individual instances of a particular model (actor) type into one or more groups to measure not only how each individual behaves in the simulation but the overall results obtained from the group. The embodiment allows this by supporting actor populations to be specified 414. Successful completion of a real-world reliable simulation may be greatly augmented by the ability to compare the results of a partial simulation run with as much of the limited real-world data as is available or possibly even incorporating portions of that real-world data into the constituent parts of the simulation such as the world configuration 412, selection of actor model characteristics 413 and types and the generation of appropriate actor populations 414. The decision operating system offers a mechanism for use of this potential advantage by allowing the simulation constituents listed to be attached as run parameters to the automated planning service module, bringing the inferential statistic and Monte Carlo heuristic algorithms into support in directing the intelligent inclusion and use of data from other modules of the decision operating system for improvement of the course taken by the simulation when run to completion 419.”)
Crabtree teaches model analysis and the feature is expounded upon by Achin:
and wherein, during the simulation, at least one of: (i) source code corresponding to at least one of the machine learning models is not shared with the central simulation system, and (ii) a dataset used to train at least one of the machine learning models is not shared with the central simulation system; (Achin Par. 185- Method 400 may include a step in which the predictive model selected for the prediction problem is tuned. In some cases, deployment engine 240 provides the source code that implements the predictive model to the user, thereby enabling the user to tune the predictive model. However, disclosing a predictive model's source code may be undesirable in some cases (e.g., in cases where the predictive modeling technique or predictive model contains proprietary capabilities or information). To permit a user to tune a predictive model without exposing the model's source code, deployment engine 240 may construct human-readable rules for tuning the model's parameters based on a representation (e.g., a mathematical representation) of the predictive model, and provide the human-readable rules to the user. The user can then use the human-readable rules to tune the model's parameters without accessing the model's source code.”)
Crabtree and Achin are directed to machine learning analysis. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have improve upon data analysis of Crabtree, as taught by Achin, by utilizing additional predictive modelling steps, with a reasonable expectation of success of arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make the modification to the teachings of Crabtree with the motivation of improving predictive modelling (Achin Par. 6).
Regarding Claim 20,
Crabtree teaches
A computer-implemented method, the method comprising: obtaining, by a central simulation system, a plurality of policies to be used for performing a simulation involving multiple machine learning models interacting with a target population, wherein the machine learning models are implemented on different systems (Crabtree Par. 29- According to one embodiment, a distributed system for accurate and detailed modeling of systems with large and complex datasets using a distributed simulation engine employing artificial intelligence/machine learning algorithms is disclosed. The system further uses results of information analytics to optimize the making of decisions and allow for alternate action pathways to be simulated using the latest data and machine-mediated prediction algorithms. Par. 55- Simulation engine 1007 may initialize a simulation world which is a JSON schema defining a model with entities, behaviors, and related schema objects. A simulation policy may comprise a JSON schema for a policy. A policy may be associated to an account and may be required to define a simulation for execution and further sets the initialize file to use, simulation times, and other parameters. A policy defines how the simulation instance for a model should run, it may be implemented as a step in a pipeline workflow integrating a simulation execution according to a preferred embodiment.”; Par.56;60; 62)
providing information for configuring simulators on the different systems, each simulator representing at least the machine learning model of a given one of the different systems; (Crabtree Par. 56- According to one embodiment, simulation engine 1007 may use models of discrete event-based systems and models of continuous event-based systems. Simulation models reproduce behavioral traits of the discrete and continuous system elements. Each instance of a model is called an actor or agent. Each simulation instance involves actors who interact within an environmental scenario which is implemented as a simulation case. Environmental scenarios are managed via cells capable of representing physical or spatial state. Technical components enable the underlying simulator to operate on the models to provide useful evaluations of behavior or performance in the context of simulation cases.”)
performing multiple iterations of the simulation for the plurality of policies, wherein, for each iteration, the central simulation system: (i) predicts a state of the target population, (ii) provides the state of the target population to the simulators, and (iii) collects one or more metrics based on results of the simulators; (Crabtree Par. 40;46; Par. 58-“ Simulation engine 1007 in combination with a model engine 1006 may use a blended modeling approach by integrating both analytical methods and system model extraction for simulations. Past observations from one or more analytical methods (e.g., regression testing, neural networks, Bayesian learning, support vector machines, etc.) coupled with retrospective outcome data (which may abide by the rule that the outcome is valid only if past conditions match future conditions) allow the system to extract models and their subcomponents. Model engine 1006 may further explore alternative models via web-scraping, natural language processing, neural networks, and other informative or rules-based computing methods. According to one embodiment, model engine 1006 may establish the various models and subcomponents by using an Advanced Cyber-Decision Platform (ACDP). The numerous system models extracted provide simulation engine 1007 with the means to iteratively learn about casual relationships thus leading to a set of probable future outcomes. Additionally, simulation engine 1007 recommends continuously competing combinations of approaches and informs/assists users in decision making.;)
and selecting and sending one of the policies to at least one of the different systems based on the collected metrics, wherein the at least one system updates its corresponding machine learning model based at least in part on the selected policy…; (Crabtree Par. 67- It is sometimes useful to place individual instances of a particular model (actor) type into one or more groups to measure not only how each individual behaves in the simulation but the overall results obtained from the group. The embodiment allows this by supporting actor populations to be specified 414. Successful completion of a real-world reliable simulation may be greatly augmented by the ability to compare the results of a partial simulation run with as much of the limited real-world data as is available or possibly even incorporating portions of that real-world data into the constituent parts of the simulation such as the world configuration 412, selection of actor model characteristics 413 and types and the generation of appropriate actor populations 414. The decision operating system offers a mechanism for use of this potential advantage by allowing the simulation constituents listed to be attached as run parameters to the automated planning service module, bringing the inferential statistic and Monte Carlo heuristic algorithms into support in directing the intelligent inclusion and use of data from other modules of the decision operating system for improvement of the course taken by the simulation when run to completion 419.”)
wherein the method is carried out by at least one computing device (Crabtree Par. 87- “In some aspects, systems may be implemented on a standalone computing system. Referring now to FIG. 12, there is shown a block diagram depicting a typical exemplary architecture of one or more aspects or components thereof on a standalone computing system. Computing device 20 includes processors 21 that may run software that carry out one or more functions or applications of aspects, such as for example a client application 24.”) .
Crabtree teaches model analysis and the feature is expounded upon by Achin:
and wherein, during the simulation, at least one of: (i) source code corresponding to at least one of the machine learning models is not shared with the central simulation system, and (ii) a dataset used to train at least one of the machine learning models is not shared with the central simulation system; (Achin Par. 185- Method 400 may include a step in which the predictive model selected for the prediction problem is tuned. In some cases, deployment engine 240 provides the source code that implements the predictive model to the user, thereby enabling the user to tune the predictive model. However, disclosing a predictive model's source code may be undesirable in some cases (e.g., in cases where the predictive modeling technique or predictive model contains proprietary capabilities or information). To permit a user to tune a predictive model without exposing the model's source code, deployment engine 240 may construct human-readable rules for tuning the model's parameters based on a representation (e.g., a mathematical representation) of the predictive model, and provide the human-readable rules to the user. The user can then use the human-readable rules to tune the model's parameters without accessing the model's source code.”)
Crabtree and Achin are directed to machine learning analysis. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have improve upon data analysis of Crabtree, as taught by Achin, by utilizing additional predictive modelling steps, with a reasonable expectation of success of arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make the modification to the teachings of Crabtree with the motivation of improving predictive modelling (Achin Par. 6).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure WO 2019/099478 Al to Crabtree et al.- Abstract A system and methods for operating an outcome modeling engine that incorporates a wide range of input data from various sources including (but not limited to) scientific advances in data analytics, agent-based modeling, discrete event simulation, and the mathematics of entropy to aid in making better decisions about real-world socio-technical systems. ; US20180165587A1 to Crabtree et al.- A system and methods for operating an outcome modeling engine that incorporates a wide range of input data from various sources including (but not limited to) scientific advances in data analytics, agent-based modeling, discrete event simulation, and the mathematics of entropy to aid in making better decisions about real-world socio-technical systems.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Chesiree Walton, whose telephone number is (571) 272-5219. The examiner can normally be reached from Monday to Friday between 8 AM and 5 PM. If any attempt to reach the examiner by telephone is unsuccessful, the examiner’s supervisor, Patricia Munson, can be reached at (571) 270-5396. The fax telephone numbers for this group are either (571) 273-8300 or (703) 872-9326 (for official communications including After Final communications labeled “Box AF”).
Another resource that is available to applicants is the Patent Application Information Retrieval (PAIR). Information regarding the status of an application can be obtained from the (PAIR) system. Status information for published applications may be obtained from either Private PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, please feel free to contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Applicants are invited to contact the Office to schedule an in-person interview to discuss and resolve the issues set forth in this Office Action. Although an interview is not required, the Office believes that an interview can be of use to resolve any issues related to a patent application in an efficient and prompt manner.
Sincerely,
/CHESIREE A WALTON/ Examiner, Art Unit 3624
Prosecution Insights