TECHNIQUES FOR SUPPORTING MODEL REGISTRIES AND MODEL ENSEMBLE DETERMINATION

Application No. 18/649,156

• Filed: Apr 29, 2024
• Examiner: ROTARU, OCTAVIAN
• Assignee: Red Hat Inc.
• OA Round: 1 (Non-Final)

Office Action

Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.

DETAILED ACTION
The following NON-FINAL Office action is in response to application 18649156 filed 04/29/2024.
Status of Claims
Claims 1-20 are currently pending and have been rejected as follows.
IDS
The information disclosure statement filed on 04/29/2024 complies with the provisions of 37 CFR 1.97, 1.98 and MPEP § 609 and is considered by the Examiner.
Objection
Claim 11 is independent and objected for redundantly reciting the informality among others:
A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to: […] ;
confirm, by the processing device, the metadata includes sample predictions generated based on benchmark data associated with the task corresponding to the AI model;
- instead of -
A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to: […] ;
confirm
Clarification and/or correctio is/are required.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(B) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.

The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.

Claims 2-5,8-18,20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claims 2,11,20 are dependent and recite among others:
identifying / identify a plurality of AI models stored in the AI model registry, the plurality of AI models associated with the task, and the plurality of AI models including the AI model; and
analyzing / analyze the sample predictions for each of the plurality of models associated with the task to determine a suggested ensemble of models for performing the task, wherein
the suggested ensemble of tasks includes one or more of the plurality of models.
Claims 2,11,20 are rendered vague and indefinite because there is insufficient antecedent basis for “the suggested ensemble of tasks” [bolded emphasis added].
Claims 2,11,20 are recommended to be amended to each recite among others:
identifying / identify a plurality of AI models stored in the AI model registry, the plurality of AI models associated with the task, and the plurality of AI models including the AI model; and
analyzing / analyze the sample predictions for each of the plurality of models associated with the task to determine a suggested ensemble of models for performing the task, wherein the suggested ensemble of the models for performing the task includes one or more of the plurality of models.
Claims 3-4, 12-13 are dependent and rejected based on rejected parent claims 2,11.
Claims 5,14 each recite among others: “wherein the AI model registry includes a set of groupings with each grouping in the set of groupings associated with one of a plurality of tasks” “further comprising storing” / … “store” “the AI model and the sample predictions in the grouping associated with the task”.
Claims 5,14 are rendered vague and indefinite because it is unclear to which of antecedently recited “groupings” [plural] does “the grouping” [singular] “associated with the task” belongs to.
Claims 5,14 are recommended to be amended to each recite among others: wherein the AI model registry includes a set of groupings with each grouping in the set of groupings associated with one of a plurality of tasks… further comprising storing / … store / the AI model and the sample predictions in a grouping of the set of groupings associated with the task.
Claims 8,17 are dependent and each recite among others: “storing” / “store” “the second AI model and the second sample predictions in the AI model registry in response to confirming the third uploaded data includes metadata includes the second sample predictions generated based on the second benchmark data associated with the second task”.
Claims 8,17 are rendered vague and indefinite because it is unclear if “metadata” as subsequently recited in said dependent Claims 8,17 relate back to “metadata” as antecedently recited at parent independent Claims 1,10.
Claims 8,17 are recommended to be amended to each recite, as an example only: “storing” / “store” “the second AI model and the second sample predictions in the AI model registry in response to confirming the third uploaded data ”.
Claims 9,18 are dependent and rejected based on rejected parent claims 8,17.
Claim 10 is independent and recites, among others: “A system comprising: a memory; and a processing device, operatively coupled to the memory, to”: […]
“confirm, by a processing device, the metadata includes sample predictions generated based on benchmark data associated with the task corresponding to the AI model”
Claim 10 is rendered vague and indefinite because it is unclear if “a processing device” as subsequently recited in said claim relates back to “a processing device” as antecedently recited at the preamble of said independent Claim 10. Since “a processing device” appears to implement each and all limitations at independent claim 10, the Examiner recommends, as an example only, that Applicant ament independent Claim 10 to recite, among others: “A system comprising: a memory; and a processing device, operatively coupled to the memory, to”: […]
“confirm
Claims 11-18 are dependent and rejected based on rejected parent independent Claim 10.
Clarifications and/or corrections are required.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.

Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea, here abstract idea) without significantly more. The claim(s) recite(s) describe or set forth the abstract idea. Examiner first points to MPEP 2106.04(a): “…examiners should identify at least one abstract idea grouping, but preferably identify all groupings to the extent possible…”.
Bases on this, Examiner finds that here, when tested under the broadest reasonable interpretation as per MPEP 2111, the claims recite, describe or at least set forth the abstract mathematical relationships expressed in words of MPEP 2106.04(a)(2) I A, as evidenced here by “metadata includes sample predictions generated based on benchmark data” at independent Claims 1,10,19 and similarly dependent Claims 8,17, used for the collecting and analyzing of information1 as part of the evaluation (here “determining”, “confirming”, “identifying”, “analyzing” limitations) and judgment (here “suggested” “models” at Claims 2-3, 11,12, 20) which according to of MPEP 2106.04(a)(2) III ¶2 fall within the broad abstract grouping of computer-aided mental processes as further elucidated by MPEP 2106.04(a)(2) III C #1,#2,#3.
- Here, such collecting is set forth as: “receiving uploaded data comprising an artificial intelligence (AI) model and metadata associated with the AI model”; (independent Claims 1,10,19)
“receiving second uploaded data comprising a second AI model” (dependent Claims 6,15),
“receiving, from the client device, third uploaded data”; (dependent Claims 8,17).
- Also here, such analysis or evaluation is set forth as: “determining a task corresponding to the AI model”; “confirming” “the metadata includes sample predictions generated based on benchmark data associated with the task corresponding to the AI model” (independent Claims 1,10,20), “identifying a plurality of AI models stored in the AI model registry, the plurality of AI models associated with the task, and the plurality of AI models including the AI model; and analyzing the sample predictions for each of the plurality of models associated with the task to determine a suggested ensemble of models for performing the task, wherein the suggested ensemble of tasks includes one or more of the plurality of models” (dependent Claims 2,11,20), “analyzing the sample predictions and additional metadata associated with each of the plurality of models” “to determine the suggested ensemble of models for performing the task” (dependent Claims 3,12), “determining a second task corresponding to the second AI model; and determining the second uploaded data fails to include second sample predictions generated based on second benchmark data associated with the second task corresponding to the second AI model” (dependent Claims 6,15), “generating the second sample predictions based on the second benchmark data associated with the second task corresponding to the second AI model” (dependent Claims 7,16); “confirming the third uploaded data includes the second sample predictions generated based on the second benchmark data associated with the second task”; (dependent Claims 8,17)
Importantly, here, given the breadth of the claims as tested based on broadest reasonable interpretation of MPEP 2111, the recitation of “artificial intelligence (AI) model” as repeatedly and preponderantly recited throughout Claims 1-8,10-17,19-20 can be argued to represent a computer environment or tool upon which the abstract processes, as identified above are being performed. Yet, MPEP 2106.04(a)(2) III C #2, #3 is clear that recitation of such computer environment or tool upon which the abstract processes are performed, does not preclude said claims to still recite, describe or set forth the abstract exception. This degree of computerization, will be further and more granularly, scrutinized, granularly at the subsequent steps below. For now, for the purpose of Step 2A prong one, it is clear that, given the preponderance of legal evidence as demonstrated above, the claims’ character as a whole is undeniably abstract. Step 2A prong one.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This judicial exception is not integrated into a practical application because per Step 2A prong two, the individual, or combination, of the additional, computer-based elements are/is found, per MPEP 2106.05(f), to merely apply the above abstract idea and/or narrow the abstract idea to a field of use or technological environment per MPEP 2106.05(h). Here Examiner identified the computer aids above as tools, computer environments etc. to aid performing the abstract processes as recognized above. Now, even when more granularly testing the aforementioned computerization as representative of additional, computer-based elements, the Examiner finds that its underlining “memory” and “processing device” (Claims 1,10-12,14-18,19) and “client device” (Claims 8-9,17-18), merely apply, the already identified abstract concepts and its respective algorithm(s) on a computer2, along with other tasks to receive, store and transmit data3, which according to MPEP 2106.05(f)(2) represent mere invocation of computer components or machinery which does not integrate the abstract exception into a practical application.
Such algorithms executed on computer are preponderantly recited here with respect to “metadata includes sample predictions generated based on benchmark data associated with the task corresponding to the AI model” (Claims 1,10,19 and similarly dependent Claims 6-8, 15-17).
Such data storing is recited here as “storing the AI model and the sample predictions in an AI model registry” (independent Claims 1,10,19), “storing the AI model and the sample predictions in the grouping associated with the task” (dependent Claims 5,14), “storing the second AI model and the second sample predictions in the AI model registry” (dependent Claims 7,8, 16,17).
Such data transmission to a computer is recited here as: “providing the second benchmark data associated with the second task to the client device” (dependent Claims 9,18).
Also here, recitations of “AI models” of Claims 1-2,5-8,10-11,14-17,19,20) can be argued to constitute, along with recitations of “at least one of compute resource requirements for each of the plurality of models, a video random access memory (VRAM) requirement for each of the plurality of models, or compute time for each of the plurality of models” at dependent Claims 4,13, a technological environment or field of use upon which the abstract concepts identified above are narrowed to. This finding is important because MPEP 2106.05(h) states that such narrowing of the abstract exception such as the abstract combination of collecting and analyzing of information to certain results of the collection and analysis related to a technological environment or field of use (MPEP 2106.05(h)(vi), does not integrate the abstract exception into a practical application.
Based on such preponderance of legal evidence, the Examiner submits that, none of the above additional elements, when tested per MPEP 2106.05(f), and/or (h), integrate the abstract idea into a practical application. Step 2A prong two.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because as shown above, the additional computer-based elements merely apply the already recited abstract idea [MPEP 2106.05(f)] and/ or narrow it to a field of use or technological environment [MPEP 2106.05(h)]. Specifically, Examiner points to MPEP 2106.05 (d) II and carries over the finings tested per MPEP 2106.05 (f) and (h), and submits that here, the additional computer-based elements also do not provide significantly more. Examiner submits that the above tests show the applying of the abstract idea [MPEP 2106.05 (f)] and narrowing the abstract idea to a field of use or technological environment [MPEP 2106.05 (h)], suffice in showing that the additional computer-based elements also do not provide significantly more without having to rely on the conventionality test [MPEP 2106.05(d)].
Yet, assuming arguendo, further evidence would be required to demonstrate conventionality of the additional, computer-based elements, Examiner would further point to MPEP 2106.05(d) II demonstrating conventionality of the additional computer-based elements as follows: receiving or transmitting data 4 , electronic recordkeeping5, arranging a hierarchy of groups and sorting the information6, performing repetitive calculations7, the latter recited here with respect to respective “first”, “second” and “third” “uploaded data” of respective “first” and “second” “AI model”, and determin[ation] of the respective “first” and “second” “uploaded data” fail[ing] to include respective “first” and “second” “sample predictions” as recited throughout Claims 1,6-8,10,15-17. If necessary, Examiner would also point to PTAB Appeal 2025-003304 citing Recentive Analytics, Inc. v. Fox Corp., 134 F.4th 1205,1212 (Fed. Cir. 2025): The requirements that the machine learning model be ‘iteratively trained’ or dynamically adjusted based on real time changes do not represent a technological improvement” at least because they are “incident to the very nature of machine learning”. Additionally, or alternatively, the Examiner submits in the arguendo, that, MPEP 2106.05(d) I 2. would also demonstrate or corroborate the conventionality given the high level of generality of the claimed additional computer elements when read in light of:
- Original Specification ¶ [0018], reciting at high level of generality: “Generally, the AI model registry 108 may operate to store a collection of AI models (e.g., AI models 122) in conjunction with model metadata (e.g., model metadata 124) in the repository 120. In one embodiment, the AI model registry 108 may include a Red HatTM OpenShiftTM AI model registry. The client interface 116 may enable client devices client device 106 to interact with the AI model registry 108, such as via the network 104. In some embodiments, client interface 116 may provide a graphical user interface (GUI) for interacting with client device 106. The repository manager 118 may be responsible for generating, updating, maintaining, storing data to, and retrieving data from the repository 120. The repository 120 may include a data store for storing models and corresponding metadata”.
- Original Specification ¶ [0051] reciting at high level: “The example computing device 600 may include a processing device 602 (e.g., a general purpose processor, a PLD, etc.), a main memory 604 (e.g., synchronous dynamic random access memory (DRAM), read-only memory (ROM)), a static memory 606 (e.g., flash memory and a data storage device 618), which may communicate with each other via a bus 630”.
- Original Specification ¶ [0052] reciting at high level: “Processing device 602 may be provided by one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. In an illustrative example, processing device 602 may include a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets.Processing device 602 may also include one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 602 may execute the operations described herein, in accordance with one or more aspects of the present disclosure, for performing the operations and steps discussed herein”.
- Original Specification ¶ [0054] reciting at high level of generality: “Data storage device 618 may include a machine-readable storage medium 628 on which may be stored one or more sets of instructions 625 that may include instructions for a component (e.g., one or more components of AI model registry 108 and/or one or more components of AI model registry extension 110) for carrying out the operations described herein, in accordance with one or more aspects of the present disclosure. Instructions 625 may also reside, completely or at least partially, within main memory 604 and/or within processing device 602 during execution thereof by computing device 600, main memory 604 and processing device 602 also constituting computer-readable media. The instructions 625 may further be transmitted or received over a network 620 via network interface device 608”.
- Original Specification ¶ [0057] reciting at high level: “Examples described herein also relate to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computing device selectively programmed by a computer program stored in the computing device. Such a computer program may be stored in a computer-readable non-transitory storage medium”.
- Original Specification ¶ [0058] “The methods and illustrative examples described herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear as set forth in the description above”.
In conclusion, Claims 1-20 although directed to statutory categories (“method” or process at Claims 1-9, “system” or machine at Claims 10-18, “non-transitory medium” or computer product or article of manufacture Claims 19-20) they still recite or set forth the abstract idea (Step 2A prong one), with their additional, computer-based elements not integrating the abstract idea into a practical application (Step 2A prong two) or providing significantly more than what was already found to be the abstract idea itself (Step 2B). Therefore, Claims 1-20 are patent ineligible.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Claim Rejections - 35 USC § 102
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.

Claims 1,6-10 and 15-19 are rejected under 35 U.S.C. 102(a)(1) based upon a public use or sale or other public availability of the invention as disclosed by:
Blomberg et al, US 20200410296 A1 hereinafter Blomberg
Claims 1,10,19 Blomberg teaches: “A method comprising:” / “A system comprising: a memory; and a processing device, operatively coupled to the memory, to:” / “A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to” (Blomberg ¶ [0087]- ¶ [0089]) :
- “receiving uploaded data comprising an artificial intelligence (AI) model and metadata associated with the AI model”; (Blomberg ¶ [0029] 2nd sentence: user devices 104 provide access to model validation system 112 to select from among and run (i.e. execute) available models to validate results of the models. For example, at ¶ [0059] model analyzer 612 receives inputs 628 including, client data and variables (historical and updated raw data) and executes code corresponding to respective stage 624. For example, each of model analyzers 612 validate and/or verify the stability of models automatically (e.g. periodically and/or in response to updates to client data), in response to user requests, continuously, etc. For example, ¶ [0060] 3rd-4th sentences: model stage 624-1 receives inputs 628 and subsequent stages 624-1 through 624-n receive output 632 of preceding stages 624. In other words, each output 632 corresponds to a result of a single step of the algorithm executed by respective stage 624 of model analyzer 612. ¶ [0062] 4th sentence: metadata includes results and scoring for each of stages 624, respective variables used in calculations in each of stages 624, statistics, etc. The metadata provided to and stored in data hub 616 is selectively accessible via user interface module 620. ¶ [0069] 1st sentence: model validation system 212 provides model data including the respective outputs 632 and corresponding metadata from each of stages 624 of model analyzers 612 to the data hub)

- “determining a task corresponding to the AI model” (Blomberg ¶ [0078] 1st sentence:
identifying model stages and variables of a selected model wherein ¶ [0058] 2nd-3rd sentences explains that model stages 624 correspond to steps of algorithm executed by model validation software 612 to validate individual models in Figs.2-5. Respective stages 624 correspond to, algorithm execution stage, output storing stage of model validation, client behavior stage etc.)

- “confirming, by a processing device, the metadata includes sample predictions generated based on benchmark data associated with the task corresponding to the AI model”; (Blomberg ¶ [0063] 1st sentence: Each time a model is validated, model validation system 212 consecutively execute each of model stages 624 and output outputs 632 and corresponding metadata from stage 624-1 through stage 624-n. ¶ [0062] 3rd sentence: metadata includes results and scoring for each of stages 624, respective variables used in calculations in each of stages 624, statistics etc. For example, ¶ [0064] 2nd-3rd sentences: model validation system 212 determine that particular output 632 indicates whether the model meet a validation threshold. In response, the model validation system 212…generate metadata for the output 632 indicating whether the model meet a validation threshold) “and”

- “storing the AI model and the sample predictions in an AI model registry in response to confirming the metadata includes the sample predictions generated based on the benchmark data associated with the task corresponding to the AI model” (Blomberg Fig.7 step 712: Validate model stage using corresponding output using model validation system->Stage Failed 716?->No->724: Store data corresponding to executed output of the executed model stage -> All stages of model complete?->Yes 320: generate and store model data in data hub using model validation system)

PNG
media_image1.png
1062
729
media_image1.png
Greyscale

Blomberg Fig.7 in support of rejection arguments
Claims 6,15. Blomberg teaches all the limitations in claims 1,10 above. Further,
Blomberg teaches further comprising:
- “receiving second uploaded data comprising a second AI model”
(Blomberg ¶ [0029] 2nd sentence: user devices 104 provide access to the model validation system 112 to select from among and run (i.e. execute) available models to validate results of the models. ¶ [0008] 2nd sentence: model data includes a second set of model data. ¶ [0029] 3rd-5th sentences: selected models are executed to determine whether respective predicted likelihoods (i.e. rates) for a behavior event using the models are greater than a natural rate of the behavior event. A ratio of the predicted likelihood to the natural rate may be referred to a lift of the model (e.g. a target response divided by the average response). Models having a lift above a desired threshold may be retained and implemented (i.e. as production models) while models having a lift below the desired threshold may be discarded and/or adjusted. ¶ [0043] 2nd sentence: each model is executed to determine the lift of the model relative to the natural rate of the behavior event. For example, at ¶ [0044] the model validation system 212 automatically (e.g., periodically, in response to updates to the client data stored in the data stack 216, etc.) execute model validation software corresponding to various cross-validation techniques as described above. Similarly, iterative Fig.4 & ¶ [0049], Fig.6B & ¶ [0059], ¶ [0070], and iterative Fig. 7 and ¶ [0074]);
- “determining a second task corresponding to the second AI model”
(Blomberg ¶ [0058] Fig.6B shows example one of model analyzers 612 including model stages 624-1,624-2…624-n, referred to collectively as model stages 624. For example, the model stages 624 correspond to respective steps of an algorithm executed by the model analyzer 612 (e.g. model validation software) to validate individual models as described above in Figs.2-5. Respective stages 624 may correspond to, for example, an algorithm execution stage, an output storing stage (e.g. stage for storing outputs of the model validation system 212), a client behavior stage etc. For example, model stages 624 are executed using a software package or container (e.g. Docker container). Each of the stages 624 has a different set of code (e.g., a code base stored and accessible in the data hub 616) to be executed by the model analyzer 612); “and”
- “determining the second uploaded data fails to include second sample predictions generated based on second benchmark data associated with the second task corresponding to the second AI model” (Blomberg Fig.7 and ¶ [0065] 1st-3rd sentences: model validation system 212 re-execute selected ones of stages 624 in response to user request and/or automatically (i.e., upon detecting a failure in one of the stages 624). In other words, if one of stages 624 fails, the model validation system 212 does not need to re-execute the entire model analyzer 612 and instead execute the failed one of the stages 624 or selected ones of the stages 624. For example, in response to a notification that a particular stage 624 failed, a user can transmit a request to the model validation system 212 to re-execute the failed stage 624. Specifically as depicted at
Blomberg Fig.7 step 728:continue to next stage?->Yes->708->712->716:Stage Failed ?-> Yes. For example, per ¶ [0061] 5th sentence: each output 632 may have expected range of values and if the output 632 is not within the expected range, the model validation system 212 determines that the model failed at the corresponding stage 624. Similarly, ¶ [0064] 2nd, 4th sentences: model validation system 212 may determine that a particular output 632 indicates that the model has failed to meet a validation threshold. variable input did not provide a result, such as a distribution shift, within a desirable range; as another example, the input variable indicated another anomaly, such as a calculated population stability index exceeding predetermined threshold)

Claims 7,16 Blomberg teaches all the limitations in claims 6,15 above. Further,
Blomberg teaches
- “generating the second sample predictions based on the second benchmark data associated with the second task corresponding to the second AI model”;
(Blomberg ¶ [0029] 3rd-5th sentences: selected models are executed to determine whether respective predicted likelihoods (i.e. rates) for a behavior event using the models are greater than natural rate of behavior event. A ratio of predicted likelihood to the natural rate is referred to a lift of the model (e.g. target response divided by average response). Models having lift above desired threshold may be retained and implemented (i.e., as production models) while models having a lift below the desired threshold may be discarded and/or adjusted. Fig.5, Fig.7 step 716->etc. etc., Blomberg ¶ [0050] method 500 for validating and verifying models for predicting client behavior. At 504, method 500 validates developed models to determine the accuracy of respective models previously developed and stored (e.g., in the data stack 216) by users to determine which model to select as a production model as described above in Fig.3. For example, each model may be validated using various cross-validation techniques to determine the lift of the model relative to the natural rate of the behavior event. At 508, the method 500 selects the developed model having the greatest lift to be implemented as the production model.
Blomberg ¶ [0051] At 512, the method 500 verifies the stability of the selected production model. For example, method 500 determines whether the actual performance of the production model (i.e., an actual lift of the model) achieves a desired lift in accordance with new client data that is acquired subsequent to the selection of the model as the production model as described above in Fig.3. If true, method 500 continues to 516. If false, the method 500 continues to 520. At 516, the method 500 continues to use the verified model as the production model.
Blomberg ¶ [0052] At 520, the method 500 selectively validates the developed models (including both the selected model and non-selected models) in accordance with the new client data. The method 500 may also verify the stability of the selected production model. In various implementations, method 500 verifies stability of the model automatically in response to updates to client data. As described above with respect to 320, the method 500 may optionally select a different model based on the stability of the production model. ¶ [0053] Control then continues to 508 to select developed model as the production model. In other words, the method 500 may continue to compare the performance of all developed models to select the model having the greatest accuracy (e.g., the greatest lift based on incoming, updated client data) and

- “storing the second AI model and the second sample predictions in the AI model registry” (Blomberg Fig.7 step 320 and ¶ [0029] 5th sentence: models having lift above desired threshold may be retained and implemented (i.e., as production models).

Claims 8,17 Blomberg teaches all the limitations in claims 6,15 above. Further,
Blomberg teaches “further comprising”:
- “requesting, from a client device, the second sample predictions generated based on the second benchmark data associated with the second task in response to determining the second uploaded data fails to include the second sample predictions generated based on the second benchmark data”; (Blomberg Fig.7 iterative steps extracted below and ¶ [0065] 1st-3rd sentences: model validation system 212 re-execute selected ones of stages 624 in response to user request and/or automatically (i.e., upon detecting a failure in one of the stages 624). In other words, if one of stages 624 fails, the model validation system 212 does not need to re-execute the entire model analyzer 612 and instead can execute the failed one of stages 624 or selected ones of the stages 624 in response to a user request. For example, in response to a particular stage 624 failed, a user transmit a request to the model validation system 212 to re-execute the failed stage 624.
Blomberg Fig.7 step 728 Continue to next stage ?->Yes->708->712->716: Stage Failed ? -> Yes. ¶ [0061] 5th sentence: each output 632 may have an expected range of values and if the output 632 is not within the expected range, the model validation system 212 determines that the model failed at the corresponding stage 624. Similarly, ¶ [0064] 2nd, 4th sentences: model validation system 212 may determine that a particular output 632 indicates that the model has failed to meet a validation threshold. variable input did not provide a result, such as a distribution shift, within a desirable range; as another example, the input variable indicated another anomaly, such as a calculated population stability index exceeding predetermined threshold)

- “receiving, from the client device, third uploaded data”;
(Blomberg ¶ [0065] 1st-3rd sentences: model validation system 212 re-execute selected ones of stages 624 in response to user request and/or automatically (i.e., upon detecting a failure in one of the stages 624). In other words, if one of the stages 624 fails, the model validation system 212 does not need to re-execute the entire model analyzer 612 and instead can execute the failed one of the stages 624 or selected ones of the stages 624 in response to a user request. For example, in response to a notification that a particular stage 624 failed, a user can transmit a request to the model validation system 212 to re-execute the failed stage 624. For example,
Blomberg Fig. 7 step 728: Continue to next stage ?->Yes-> 708->712->716: Stage Failed ?->Yes->720->728: Continue to next stage->Yes ->708: Execute model stage to generate an output using model validation system)

- “confirming the third uploaded data includes the second sample predictions generated based on the second benchmark data associated with the second task”;
(Blomberg Fig. 7 step 728: Continue to next stage ?->Yes-> 708->712->716: Stage Failed ? -> Yes-> 720->728: Continue to next stage->Yes -> 708: Execute model stage to generate an output using model validation system-> 712: Validate model stage using corresponding output using model validation system -> Stage Failed 716 ? -> No) and

- “storing the second AI model and the second sample predictions in the AI model registry in response to confirming the third uploaded data includes metadata includes the second sample predictions generated based on the second benchmark data associated with the second task” (Blomberg Fig. 7 step 712: Validate model stage using corresponding output using model validation system -> Stage Failed 716? -> No -> 724: Store data corresponding to the executed output of the executed model stage -> All stages of model complete? -> Yes 320: generate and store model data in data hub using model validation system)

PNG
media_image2.png
1013
695
media_image2.png
Greyscale

Blomberg Fig.7 in support of rejection arguments

Claims 9,18 Blomberg teaches all the limitations in claims 8,17 above. Further,
Blomberg teaches: “providing the second benchmark data associated with the second task to the client device” (Blomberg ¶ [0026] 3rd sentence: the model management system implements a user interface that provides access to the models and validation data. ¶ [0061] 2nd sentence noting such validation data comprises validation threshold for each stage 624. Similarly, ¶ [0041] 3rd - 5th sentences: development module 248 may provide outputs results of the variable selection algorithms to the user device 204. For example, the output results may include a report of the selected variables. Similarly, ¶ [0048] last sentence: development module 248 generates a report of a selected subset of variables and outputs the report to the user device 204).
Rejections under 35 § U.S.C. 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.

Claims 2-4, 11-13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over:
Blomberg as applied to claims 2,11, and in view of
Rawat et al, US 20220198222 A1 hereinafter Rawat. As per,
Claims 2,11,20 Blomberg teaches all the limitations in claims 1,10,19 above. Furthermore,
Blomberg teaches “further comprising”:
- “identifying a plurality of AI models stored in the AI model registry, the plurality of AI models associated with the task, and the plurality of AI models including the AI model”;
(Blomberg ¶ [0026] 2nd sentence, ¶ [0029] 2nd sentence: user devices 104 provide access to model validation system 112 to select from among and run (i.e. execute) available models to validate results of the model. ¶ [0030] 1st-2nd sentence data stack 116 stores data including the models. data stored in data stack 116 may be accessed and retrieved by the model development system 108, model validation system 112, and user devices 104 to develop, validate, and run the models. ¶ [0044] 1st sentence: model validation system 212 automatically validate the developed models (including current selected production model and non-selected models). Additional details at ¶¶ [0029], [0042] 1st-2nd sentences, [0043], [0046] 2nd sentence, [0047], [0050]-[0054])
* Further, while *
Blomberg Fig.3 and ¶ [0043] 2nd sentence recites: each model may be validated using cross-validation techniques. ¶ [0044] model validation system 212 automatically validate the developed models (including a current selected production model and non-selected models). For example, model validation system 212 automatically (e.g., periodically, in response to updates to client data stored in the data stack 216, etc.) execute model validation software corresponding to various cross-validation techniques as described above. In other examples, the model validation system 212 validate all or selected ones of the developed models in response to inputs received at the user device 204. ¶ [0046] 2nd sentence: the model validation system 212 continue to automatically validate other (i.e., non-selected) developed models using the newly-acquired client data. In some examples, as client data is acquired, the client data corresponding to the variables used by the selected model is provided to model validation system 212 in real-time for continuous verification of the selected model. ¶ [0047] 1st-2nd sentences: model validation system 212 optionally select a different model based on the stability of the production model. For example, the model validation system 212 may select a different model in response to the lift of the selected model decreasing below a threshold a predetermined number of times, in response to an average lift of the selected model over a given period decreasing below a threshold, a lift of one of the non-selected models increasing above the lift of the selected model, etc.)
* Nevertheless *
Blomberg falls short to exactly recite, to clearly anticipate:
- “analyzing the sample predictions for each of the plurality of models associated with the task to determine a suggested ensemble of models for performing the task, wherein the suggested ensemble of tasks includes one or more of the plurality of models” as claimed.
* However *
Rawat in analogous machine learning model pipeline combinations teaches/suggests:
- “analyzing the sample predictions for each of the plurality of models associated with the task to determine a suggested ensemble of models for performing the task, wherein the suggested ensemble of tasks includes one or more of the plurality of models”
(Rawat ¶ [0006] recognizes the benefits of an ensemble generation technique that learns from previous experiences. The present invention recognizes the benefits of using preferred pipelines, considering several ensemble strategies, using an ensemble evaluation metamodel trained on a history of ensemble performance, recommending pipeline enhancements, and considering the original and enhanced pipelines when selecting preferred ensembles. The present invention also recognizes the benefits of using customized combinations of models and tailored model parameters to improve AI performance.
Rawat ¶ [0035] 1st-3rd sentences: at Fig.3, pipeline optimizer 118 forwards optimized pipelines to ensemble sampler 120 for further processing. It is noted that performance of certain pipeline combinations provides better performance than any single (even optimized) pipeline for many datasets. According to aspects of the invention, the ensemble generator 120 suggests various combinations of optimized pipelines from among the pipelines received from the pipeline optimizer 118 as a way of providing performance even better than any single optimized pipeline.
Rawat ¶ [0036] last sentence: metamodel 114 is trained on this info to make ensemble performance recommendations and rank expected ensemble performance on wide dataset types. Rawat ¶ [0038] 4th-5th sentences: With access to historical perspective from HEPD 116, the 114 metamodel improves ensemble recommendation efficiency by focusing training efforts in ETEM 124 on pipeline combinations predicted to provide exceptional performance with the provided dataset 106. According to aspects of the invention, metamodel 114 predict ensemble performance for various datasets and also suggest updates to suggested ensembles and also suggest new ensemble pipelines based on info stored from previous ensemble performance over many different datasets. ¶ [0039] 1st sentence: With continued reference to Fig.3, the system of the invention iteratively evaluates (e.g in ETEM 1124), suggested ensemble candidates, updates the HEPD 116 and providing incremental metamodel 114 training with each round of evaluation).
It would have been obvious to one skilled in the art, before the effective filling date of the claimed invention, to have modified Blomberg’s “method” / “system” above to have included Rawat teachings or suggestions in order to have efficiently chose from among possible pipeline combinations (i.e. ensembles), by having provided a complementary optimization process that would have improved ensemble selection efficiency by applying historic performance data from historic ensemble performance database for datasets similar to the currently provided dataset
(Rawat ¶ [0036] 3rd sentence in view of MPEP 2143 C, D and/or G). With access to historical perspective from historic ensemble performance database, the Rawat’s metamodel would have improved ensemble recommendation efficiency by focusing training efforts in ensemble training and evaluation module on pipeline combinations predicted to provide exceptional performance with the provided dataset (Rawat ¶ [0038] 4th sentence in view of MPEP 2143 C, D and/or G). The predictability of such modification would have been corroborated by the broad level of skill of one of ordinary skills in the art as articulated by Blomberg ¶ [0090] in view of Rawat ¶ [0082].
Further, the claimed invention is merely a combination of old elements in a similar machine learning field of endeavor. In such combination each element merely would have performed same analytical function as separately. Thus, one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Blomberg in view of Rawat, the to be combined elements would have fitted together, like puzzle pieces in a logical, complementary, technologically feasible and/or economically desirable manner. Thus, it would have been reasoned that the combination results would have been predictable (MPEP 2143 A).

Claims 3,12 Blomberg/Rawat teaches all the limitations in claims 2,11 above. Furthermore,
Blomberg does not explicitly recite:
- “analyzing the sample predictions and additional metadata associated with each of the plurality of models stored in the model registry to determine the suggested ensemble of models for performing the task” as explicitly claimed. However,
Rawat in analogous machine learning model pipeline combinations teaches/suggests:
- “analyzing the sample predictions and additional metadata associated with each of the plurality of models stored in the model registry to determine the suggested ensemble of models for performing the task” (Rawat ¶ [0006] recognizes the benefits of an ensemble generation technique that learns from previous experiences. The present invention recognizes the benefits of using preferred pipelines, considering several ensemble strategies, using an ensemble evaluation metamodel trained on a history of ensemble performance, recommending pipeline enhancements, and considering the original and enhanced pipelines when selecting preferred ensembles. The present invention also recognizes the benefits of using customized combinations of models and tailored model parameters to improve AI performance. ¶ [0036] last sentence: The metamodel 114 is trained on this info to make ensemble performance recommendations, and in turn, to rank expected ensemble performance for a wide array of dataset types. ¶ [0038] 4th-5th sentences: With access to historical perspective from HEPD 116, the 114 metamodel improves ensemble recommendation efficiency by focusing training efforts in the ETEM 124 on pipeline combinations predicted to provide exceptional performance with the provided dataset 106. According to aspects of the invention, metamodel 114 predict ensemble performance for various datasets and also suggest updates to suggested ensembles and also suggest new ensemble pipelines based on information stored from previous ensemble performance over many different datasets. ¶ [0039] 1st sentence: With continued reference to FIG. 3, the system of the present invention iteratively evaluates (e.g in ETEM 1124), suggested ensemble candidates, updates the HEPD 116 and providing incremental metamodel 114 training with each round of evaluation).
It would have been obvious to one skilled in the art, before the effective filling date of the claimed invention, to have modified Blomberg’s “method” / “system” above to have included Rawat teachings or suggestions in order to have efficiently chose from among possible pipeline combinations (i.e. ensembles), by having provided a complementary optimization process that would have improved ensemble selection efficiency by applying historic performance data from historic ensemble performance database for datasets similar to the currently provided dataset
(Rawat ¶ [0036] 3rd sentence in view of MPEP 2143 C, D and/or G). With access to historical perspective from historic ensemble performance database, the Rawat’s metamodel would have improved ensemble recommendation efficiency by focusing training efforts in ensemble training and evaluation module on pipeline combinations predicted to provide exceptional performance with the provided dataset (Rawat ¶ [0038] 4th sentence in view of MPEP 2143 C, D and/or G). The predictability of such modification would have been corroborated by the broad level of skill of one of ordinary skills in the art as articulated by Blomberg ¶ [0090] in view of Rawat ¶ [0082].
Further, the claimed invention is merely a combination of old elements in a similar machine learning field of endeavor. In such combination each element merely would have performed same analytical function as separately. Thus, one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Blomberg in view of Rawat, the to be combined elements would have fitted together, like puzzle pieces in a logical, complementary, technologically feasible and/or economically desirable manner. Thus, it would have been reasoned that the combination results would have been predictable (MPEP 2143 A).

Claims 4,13 Blomberg / Rawat teaches all the limitations in claims 3,12 above. Furthermore,
Blomberg teaches: “wherein the additional metadata includes at least one of compute resource requirements for each of the plurality of models” (Blomberg ¶ [0062] 3rd sentence: metadata includes results and scoring for each of stages 624. For example, ¶ [0045], ¶ [0047] model validation system 212 select a different model in response to the lift of the selected model decreasing below a threshold a predetermined number of times, in response to an average lift of selected model over a given period decreasing below a threshold, a lift of one of the non-selected models increasing above the lift of the selected model, etc. In this manner, the model validation system 212 selects the model having the most accurate prediction of the client behavior event as additional client data is acquired. Similarly, ¶ [0051] At 512, the method 500 verifies the stability of the selected production model. For example, the method 500 determines whether the actual performance of the production model (i.e., an actual lift of the model) achieves a desired lift in accordance with new client data that is acquired subsequent to the selection of the model as the production model as described above in FIG. 3. If true, the method 500 continues to 516. If false, the method 500 continues to 520. At 516, the method 500 continues to use the verified model as the production model), “a video random access memory (VRAM) requirement for each of the plurality of models, or compute time for each of the plurality of models”.
Rawat also teaches or at least suggests: “wherein the additional metadata includes at least one of compute resource requirements for each of the plurality of models” (Rawat mid-[0017] HEPD 116 reference content includes historic performance metadata, including performance metrics for plurality of historic ensembles applied previously to a corresponding plurality of historic datasets, along with feature metadata for the historic ensembles and historic datasets associated with HEPD. ¶ [0039] 3rd sentence: two preferred stop conditions include passing of a predetermine amount of evaluation resources (e.g. certain amount of computation time, etc.) having been spent and arrival of an ensemble performance) “a video random access memory (VRAM) requirement for each of the plurality of models, or compute time for each of the plurality of models”.
Rationales to have modified/combined Blomberg and/or Rawat are above and reincorporated.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Claims 5,14 are rejected under 35 U.S.C. 103 as being unpatentable over:
Blomberg as applied to claims 1,10, and in view of
Dubey et al, US 20170161336 A1 hereinafter Dubey. As per,
Claims 5,14 Blomberg teaches all the limitations in claims 1,10 above.
Blomberg does not exactly recite to clearly anticipate, as claimed:
- “wherein the AI model registry includes a set of groupings with each grouping in the set of groupings associated with one of a plurality of tasks, and the method further comprising” / “and the processing device is further to” “storing” / “store” “the AI model and the sample predictions in the grouping associated with the task”.
Dubey however in analogous adaptive ensemble teaches or suggests:
- “wherein the AI model registry includes a set of groupings with each grouping in the set of groupings associated with one of a plurality of tasks, and the method further comprising” / “and the processing device is further to” “storing” / “store” “the AI model and the sample predictions in the grouping associated with the task”.
(Dubey ¶ [0032] 6th sentence: model repository includes model registry 206 that stores information about model stacks [or groups] 204 a-204 n, as well as store 208 for data and/or metadata about the model training, baselining, and model refinement. For example, at
Dubey ¶ [0008] and claim 28: for each different processing group, starting with processing group associated with lowest level of information uncertainty and moving upwardly: identify corresponding model stacks [or groups] from model registry to be executed on respective processing group, the model registry storing a plurality of different classification model stacks [or groups], each classification model stack including at least one classification model and at least one different and independent confidence model; execute each identified model stack on the respective processing group to arrive at a classification result and confidence level [or prediction] for each data entry in the respective processing group using the classification and confidence models in the respective model stack; adaptively ensemble results from the execution of each identified model stack, using the classification results & confidence levels, along with processing group specific ensembler, to group the data entries in the processing group into one of first and second result type groups, the first result type group corresponding to a confirmed classification and second result type group corresponding to an unconfirmed classification; and move each data entry in the first result type group to a final result set; and for the second result type group.
Dubey ¶ [0064] 5th sentence: As noted above, the results of model selection, training, and baselining, may be stored within the model registry in store 208.
Dubey ¶ [0093] 2nd sentence: A sample distribution of classification performance with sparse (i.e., single source) training set is provided in Fig.6. ¶ [0096] 1st-6th sentence: BGLM operates on a problem space that combines the VSM clustering output with the reference universe profile. The Fig.7 schematic diagram illustrates the operation of this model stack. As shown in Fig.7, the VSM clustering classification 702 output is fed into a confidence model 704. The confidence model is, in essence, the BGLM being used to provide a classification confidence readout on top of the VSM clustering output. A decision as to whether to trust the output is then made (e.g., based on the classification confidence). If the output is to be trusted as determined in block 706, then it is moved to the final result set as indicated in block 708. ¶ [0097] 1st sentence: BGLM classifier basically provides the value of following: p(class(Bij)=Ck:x1…xn) which is the probability of the jth line item of processing bucket Bi, being a part of class Ck (i.e., the correct class assignment) given the respective values of predictors x1…xn. Similarly, ¶ [0110] 4th sentence: Fig.8, shows a sample illustration of a decision boundary that separates the zone of competence (the lighter dots at the bottom right of the illustration) from rest of the problem space).
It would have been obvious to one skilled in the art, before the effective filling date of the claimed invention, to have modified Blomberg’s “method” / “system” to have included Dubey’s teachings or suggestions in order to have further improved the integrity and consistency of data imported from Big data and/or other data sources (Dubey ¶ [0002] 1st sentence in view of MPEP 2143 C, de and/or G) such as the complementary data sources taught by Blomberg. Predictability of such modification would have been corroborated by the broad level of skill of one of ordinary skills in the art as articulated by Blomberg ¶ [0090] in view of Dubey ¶ [0005] 2nd sentence.
Further, the claimed invention could have also been viewed as a mere combination of old elements in a similar machine learning field of endeavor. In such combination each element merely would have performed same analytical and storing function as it did separately. Thus, one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Blomberg in view of Dubey, the to be combined elements would have fitted together, like pieces of a puzzle in a logical, complementary, technologically feasible and/or economically desirable manner. Thus, it would have been reasoned that the combination results would have been predictable (MPEP 2143 A).

Conclusion
Following art is made of record and considered pertinent to Applicant’s disclosure:
- WO 2017100072 A1 teaching Automatically classifying and enriching imported data records to ensure data integrity and consistency
- Caruana et al, Ensemble selection from libraries of models, InProceedings of 21 international conference on Machine learning, p18, Jul 4, 2004
- Giacinto et al, Dynamic classifier selection based on multiple classifier behaviour, Pattern Recognition, 34 9, pp1879-1881, Sep 1, 2001
- US 20210264321 A1 teaching machine learning model registry, with emphasis on:
¶ [0025] 2nd-4th sentences: cron job identifies and trains machine learning models 123 that were recently received by machine learning model registry 125 or previously stored in machine learning model registry 125. The cron job executes to train machine learning models 123 based on training data or actual data that has been updated. For example, the cron job executes to train existing or machine learning models 123 that are new based on training data or actual data that was recently received (e.g., updated or new training data, updated or new actual data, etc.).
¶ [0027] 2nd-8th sentences: For example, the cron job identifies an access module 121 to validate whether the access module 121 properly interoperates with the machine learning model 123 being validated. In one embodiment, the cron job identifies the access module 121 by searching the machine learning model registry 125 to identify an access module 121 that is registered as deployable, registered as associated with the machine learning model 123, and registered as having the same number of features. If the access module 121 is found, the cron job replays past prediction requests to the identified access module 121 that in turn, communicates the requests to the machine learning model 123 being validated that, in turn, generates the predictions. The cron job may compare the distributions of the predictions that are generated by the machine learning model 123 being validated with the distributions of the predictions of the machine learning model 123 being utilized as a benchmark. For example, the cron job may submit the same requests to the respective machine learning models 123 and compare the distributions. If the cron job identifies the predictions as having similar distribution, then the cron job identifies the machine learning model 123 being validated as having passed validation and registers it as deployable. For example, the cron job identifies distributions as being similar based on a means squared error test. The scheduler module 135 (e.g., cron job) communicates the results of the validation including the interoperability test and the prediction test to the registry server machine 132 that, in turn, stores the results (e.g., metadata) in the appropriate elements of machine training information 138 and deployment information 144 in the machine learning model registry 125.
¶ [0032] At operation “B” and responsive to committing the access module 121, the deploy module 146 executes an acceptance test of the access module 121. The acceptance test may include a replay request test and an interoperability test. For example, the deploy module 146 may replay previous requests to the access module 121 and monitor its prediction. If the access module 121 fails to generate a prediction or generates a prediction that is not within a predetermined range (e.g., acceptable home prices), then the access module 121 fails the acceptance test. The results of the acceptance test may be stored in an element of the access module information 142 (corresponding to the access module 121) in the machine learning model registry 125. For example, the deploy module 146 may register the access module 121 as deployable in the access module information 142.
¶ [0034] At operation “D,” the deploy module 146 communicates the results of the acceptance test to the registry server machine 132 that, in turn, stores the results (e.g., metadata) in the appropriate element of access module information 142 and the deployment information 144 in the machine learning model registry 125. If the deploy module 146 identifies the access module 121 as passing the replay request test, then the deploy module causes the access module 121 to be registered as deployable in the appropriate element of access module information 142 in the machine learning model registry 125. In addition, if the deploy module identifies the access module 121 as passing the interoperability test, then the deploy module causes the access module 121 to be registered as deployable in the deployment information 144 in the machine learning model registry 125 and as interoperating with the identified machine learning models 123 in the deployment information 144 in the machine learning model registry 125.
¶ [0056] The error information 454 displays the type of error causing the validation to fail. For example, the error information displays a “COEFFICIENT SHIFT” indicating one of the coefficients of the machine learning model 123 candidate was not included in a predetermined range. The row information 458 may display additional information regarding the error. For example, the row information 458 displays dissimilar square footage coefficients (e.g., “SQ FT”). The machine learning model 123 candidate entitled “Home Valuation—Linear Regression—Version 25 Model” failed validation because the square footage coefficient (e.g., 100) for the machine learning model 123 candidate is identified as being dissimilar from a square footage coefficient (e.g., 80) for the benchmark machine learning model 123. According to an embodiment, a dissimilar square footage coefficient may be defined as a candidate square footage coefficient that is identified as not being included in a predetermined range of square footage coefficients defined by a lower limit=(benchmark square footage coefficient−X) (e.g., 70=80−10) and an upper limit=benchmark square footage coefficient+X (e.g., 90=80+10).
- US 20210173993 A1 Fig. 8 below noting at step 309: Does re-validated ML model meet predetermined criteria?->No->step 303: First fail for ML model?->No->step 311: adding additional training data -> step 214: re-validating the again-failed ML model. ¶ [0063] If the re-validated first-failed ML model does not meet the predetermined criteria (309) the process proceeds back to step 303. ¶ [0064] If the ML model has failed more than once (303), training validation and testing ML module 6 adds (311) additional training data to the training data set of the again-failed ML model from new sets of ERD examples to generate an updated training data set and optionally stores the updated training data set. Training, validation and testing ML module 6 re-trains (312) the again-failed ML model using the updated training data set, stores (313) the re-trained again-failed ML model; and re-validates (314) the again-failed ML model and optionally stores the re-validated again-failed ML model. If the re-validated again-failed ML model does not meet the predetermined criteria (309) the process of steps 303 and 311-314 is repeated (i.e., additional training data is added at each pass through reference numeral 311) until the predetermined criteria are met in step 309. If the re-validated again-failed ML model meets the predetermined criteria (309) the validation process ends and training, validation and testing ML module 6 proceeds to testing (310) of the re-validated again-failed ML model.

PNG
media_image3.png
966
746
media_image3.png
Greyscale

US 20210173993 A1 Fig. 8
- US 20220309391 A1 Fig.2 and ¶ [0095] Various criteria can be applied for providing a list of recommended predictive models for presentment on model recommendation area 3221. According to one use case, developer system 110 can perform the following process at action decision block 1006 to determine a recommend set of K predictive models for deployment on enterprise system 140A: (a) sort the M models based on performance under test; (b) generate a list K of top K performing models; (c) identify failure conditions of models on the list K; (d) for every failure condition identified in (c), confirm that there are at least two models of the list K without the failure condition and (e) if necessary to satisfy the criterion of (d) replace model(s) of the list K, with next highest scoring models of the models M until the criterion of (c) is satisfied; and (f) label the finalized list K (after (e) is performed) as the finalized list K for specifying within presented prompting data. At block 1006, developer system 110 can return an action decision to determine a recommend list K of K models for deployment in an ensemble model deployment scenario in which multiple models can be deployed, some having areas of less than optimal performance but which are balanced by the providing of supplemental models that are stronger in the weaker areas
- US 9665628 B1 column 22 lines 54-56: (a) identify one or more corresponding model stacks from a model registry to be executed on the respective processing group, the model registry storing a plurality of different model stacks, each model stack including at least one classification model and at least one different and independent confidence model;
- US 20190166024 A1 teaching anomaly analysis, emphasis on Fig.3 and associated text
- US 20240185116 A1 teaching Systems and methods for bagging ensemble classifiers for imbalanced big data with emphasis on ¶ [0034] last sentence: if a the ensemble of base models fails the validation, then the training system 220 may retrain each of the machine learning base models by training the model using more or less iterations, including more data in the training data chunks (e.g., creating larger chunks that include some overlap in majority cases between the chunks), tuning hyperparameters, increasing/decreasing regularization on the model, and/or providing the model with some constraints to make it less flexible.
- US 20220237521 A1 teaching Method, device, and computer program product for updating machine learning model
- US 20210383281 A1 teaching at its abstract: Obtaining a first evaluation result representing performance of a first machine learning model having learned using first learning data, the first evaluation result being calculated using first validation data; obtaining a second evaluation result representing performance of a second machine learning model having learned using second learning data, the second evaluation result being calculated using second validation data; and calculating, based on the first evaluation result and the second evaluation result, a comprehensive evaluation result representing performance of a single machine learning model including the first machine learning model and the second machine learning model, the performance of the single machine learning model being predicted when the single machine learning model is applied to unevaluated, unknown data relevant to a prescribed event.
- US 20200394451 A1 teaching Selecting artificial intelligence model based on input data ¶ 0088] the same panda image (e.g., see reference numeral 130) may be input to the first, second, and third AI models. Although the first AI model may have the lowest general accuracy for recognizing an object, the first AI model may correctly output a result indicating a ‘panda.’ On the other hand, although the second AI model and the third AI model may have higher general accuracies for recognizing an object, the second AI model may incorrectly output a result indicating a ‘dog,’ and the third AI model may incorrectly output a result indicating a ‘person.’
¶ [0091] 2nd sentence: automatically selecting the best AI model having a highest accuracy for performing object recognition may, in fact, more accurately recognize an object of an image than an AI model having a highest general accuracy for recognizing any given input object.
¶ [0137] For example, in some cases, certain input data may be correctly recognized and classified by the first AI model M1 but may be incorrectly recognized by the second AI model M2. ¶ [0117] It is assumed that the AI model 410 receives the image 305 including a dog as an object. In this case, the AI model 410 may output a classification result indicating a ‘dog’ when the object included in the input image 305 is correctly recognized and classified, or output a classification result indicating a ‘cat’ when the object included in the input image 305 is incorrectly recognized and classified. ¶ [0118] Hereinafter, when the AI model 410 outputs an incorrect result, the input data is called ‘misclassified data.’ For example, when the AI model 410 receives the image 305 and outputs a classification result indicating a ‘cat,’ the image 305 may be included in the ‘misclassified data.’
¶ [0189] Specifically, when the first data is input to the selected AI model, the selected AI model may analyze the first data and perform neural network calculations for outputting a corresponding classification result. For example, when the selected AI model is an AI model for performing the classification operation illustrated in FIG. 4, the selected AI model may recognize and classify an object included in the input first data and output a classification result (e.g., a result indicating a dog or a cat).
¶ [0190] As described above, an embodiment of the disclosure may increase accuracy of an output result by generating the output result by selecting an AI model having the lowest probability of misclassifying data to be tested (i.e., the ‘first data’), namely an AI model having a highest probability of correctly classifying the input data.
- US 20190251476 A1 teaching Reducing redundancy and model decay with embeddings
¶ [0031] last sentence: Process 400 then proceeds to act 418, where the trained embedding(s) and their corresponding benchmarking results are stored in the feature registry for use by one or more machine learning models, as illustrated in Fig.3 and described briefly above
- US 20210350280 A1 teaching Model modification and deployment emphasis on Fig.5

PNG
media_image4.png
1030
806
media_image4.png
Greyscale

US 20210350280 A1 Fig. 5
- US 20260004204 A1 emphasis on Fig.13 and associated text

PNG
media_image5.png
1154
818
media_image5.png
Greyscale

US 20260004204 A1 emphasis on Fig.13

- US 20200226321 A1 recting at ¶ [0063] The best performing machine learning model by overall accuracy in the Holdout dataset was the SVM model (81.2%). When stratifying by the same “High”, “Medium”, and “Low” CP metrics, one finds a 93.1% accuracy for the “High” (CP>=1.6) group encompassing 58.0% of the dataset. At the more stringent confidence (CP>=2.0), accuracy again increases (94.7%) with a decrease in dataset coverage to 48.0%. Accuracy within the top 3 was 97.0% (CP>=1.6) and 96.3% (CP>=2.0) for the Holdout dataset. If the Train/Test and Holdout dataset were merged and a random 58,510 cases were held out, the overall accuracy increases to 84.5%, 94.0% for “High”, and 95.3% for the more stringent levels, covering 62.0% and 49.4% of cases, respectively.
- US 20230385835 A1 ¶ [0053] survival model can be used in conjunction with survival analysis in, e.g. fraud modeling to enable unbiased learning from recent data that may be missing data tags (e.g., fraud confirmation tags) due to an incomplete maturity window. Survival analysis is useful for analyzing datasets that may include censored data having a waiting time until a terminal event. For example a transaction dataset that spans a timeframe less than a fraud maturity window, will have some transactions tagged as fraudulent. There will be other transactions, however, that will later be tagged as fraudulent but that are not so tagged in the dataset due to the dataset being collected over a shorter time frame than the fraud maturation time frame. Thus, using survival analysis, a model can be trained on a dataset prior to all fraudulent transactions in a recent dataset being tagged as such. Using survival analysis, bias in a dataset collected over a timeframe that is shorter than fraud maturation can be removed from immature fraud tags by predicting the probability that a transaction will eventually be tagged as fraudulent.
- US 11676016 B2 teaching Selecting artificial intelligence model based on input data
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OCTAVIAN ROTARU whose telephone number is [phone]. The examiner can normally be reached on 571.270.7950 from 9AM to 6PM. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, PATRICIA H MUNSON, can be reached at telephone number [phone]. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.

/OCTAVIAN ROTARU/
Primary Examiner, Art Unit 3624 A
July 29th, 2026

1 Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016);
2 Alice Corp. Pty. Ltd. V. CLS Bank Int’l, 573 U.S. 208, 223, 110 USPQ2d 1976, 1983 (2014);
Versata Dev Group, Inc v SAP Am Inc 793 F.3d 1306,1334,115 USPQ2d 1681, 1701 (Fed Cir 2015)
3 Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); 
TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016)
Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015).
4 Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)  
5 Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 225, 110 USPQ2d 1984 (2014) (creating and maintaining "shadow accounts"); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 
6 Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1331, 115 USPQ2d 1681, 1699 (Fed. Cir. 2015).
7 Flook, 437 U.S. at 594, 198 USPQ2d at 199; Bancorp Services v. Sun Life, 687 F.3d 1266,1278,103 USPQ2d 1425,1433 (Fed. Cir. 2012)

Prosecution Timeline

• Apr 29, 2024: Application Filed
• Jul 31, 2026: Non-Final Rejection mailed — §101, §102, §103 (current)