Office Action Analysis: 18349738 — FRAMEWORK FOR EVALUATION OF COMPUTER-BASED MODELS

Office Action

§101 §103 §112
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1- 23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. 
Claims 7-8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA  35 U.S.C. 112, the applicant), regards as the invention.
Claim 21 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA  35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim(s) 1-20, 22-23 is/are rejected under 35 U.S.C. 103 as being unpatentable over US PGPUB No. US20170124487A1 by Szeto & Chan in further view of NPL: "A DISTRIBUTED PLATFORM FOR MODELEVALUATION AND BENCHMARKING AT SCALE" by Dakkak et al.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over US PGPUB No. US20170124487A1 by Szeto & Chan in view of NPL: "A DISTRIBUTED PLATFORM FOR MODELEVALUATION AND BENCHMARKING AT SCALE" by Dakkak et al. in further view of NPL: “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation” by Liu et al.
This action is made Non-Final.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.


The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1- 23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. 
The claimed invention is directed to an abstract idea falling within the enumerated abstract idea groupings of Mental Processes (MPEP § 2106.04(a)(2)(III)) and Certain Methods of Organizing Human Activity (MPEP § 2106.04(a)(2)(II)). Specifically, Claim 1 recites the concept of receiving user parameters (modeling objective, evaluation data set, evaluation library, and subset definitions), creating/storing an evaluation configuration, associating the configuration with a modeling objective, and evaluating models based on the specified parameters. Collecting data, organizing parameters into structural relationships, and evaluating performance against metrics are concepts that can practically be performed in the human mind or by a human using pen and paper. (See, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674)
The judicial exception is not integrated into a practical application because the claim lacks additional limitations that meaningfully restrict the abstract idea. The additional limitations beyond the abstract concept consist of:
"one or more hardware processors executing program instructions";
"one or more graphical user interfaces"; and
"creating, storing, and/or updating..." in computer memory.
These additional elements merely invoke generic, general-purpose computer hardware and standard user interface functions as tools to execute the abstract administrative and evaluation workflow. (See CyberSource v. Retail Decisions, 654 F.3d 1366, 1370, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011)( MPEP § 2106.05(f))( See, e.g., Versata Development Group v. SAP America, 793 F.3d 1306, 1335, 115 USPQ2d 1681, 1702 (Fed. Cir. 2015))( See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016)) The claim does not recite any specific improvement to computer functionality, specialized algorithm, or physical transformation. (2106.04(d)(1))( 2106.05(b))( 2106.05(c)) Limiting the application of the abstract evaluation steps to the technological environment of "computer-based models" constitutes generic field-of-use placement (MPEP § 2106.05(h)) and fails to integrate the exception into a practical application. (2106.05(h))( See e.g., Parker v. Flook, 437 U.S. 584, 588-90, 198 USPQ 193, 197-98 (1978))
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. (See MPEP § 2106)( MPEP § 2111)( MPEP § 2106.05(d)) Taken individually: Executing instructions on hardware processors and storing records in memory are well-understood, routine, and conventional computer activities in the art (MPEP § 2106.05(d)). (see DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258, 113 USPQ2d 1097, 1106 (Fed. Cir. 2014)) Receiving input choices via a graphical user interface is generic, routine data entry activity. (2106.05(g))( Performing model evaluations based on the stored configuration represents mere instructions to "apply" the abstract concept on a computer. (2106.05(f))( See MPEP § 2106.07(a))(See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016); Intellectual Ventures I v. Symantec, 838 F.3d 1307, 1327, 120 USPQ2d 1353, 1366 (Fed. Cir. 2016); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1417 (Fed. Cir. 2015))( see, e.g., McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 120 USPQ2d 1091 (Fed. Cir. 2016); Thales Visionix Inc. v. United States, 850 F.3d. 1343, 121 USPQ2d 1898 (Fed. Cir. 2017))( see e.g., Ultramercial, Inc. v. Hulu, LLC, 772 F.3d 709, 112 USPQ2d 1750 (Fed. Cir. 2014); Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 119 USPQ2d 1739 (Fed. Cir. 2016)) (see MPEP § 2106.05(a)) (see MPEP § 2106.05(b) and (c), respectively) (see MPEP § 2106.05(d)).
Taken as an ordered combination, the limitations merely arrange generic computer functions in their standard, logical sequence (input → store → associate → evaluate). This combination does not perform functions beyond generic computer operations and does not confer an inventive concept (MPEP § 2106.05(d)). Accordingly, Claim 1 is ineligible under 35 U.S.C. § 101. (See MPEP § 2106.03) (MPEP § 2106.04(a)(2))(2106.07(a))
Claims [ 1]:
Step 1: the claims are drawn to a method and system respectively, falling under one of the four statutory categories of invention.
Step 2A, Prong 1: This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim “recites” a judicial exception when the judicial exception is “set forth” or “described” in the claim. The limitations are bolded for abstract idea/judicial exception identification.
Claim 1
Mapping Under Step 2A Prong 1
A computer-implemented method for managing one or more models, the computer-implemented method comprising, by one or more hardware processors executing program instructions:

receiving, from a user and via one or more graphical user interfaces, one or more user inputs including at least:











a first user input requesting to add an evaluation configuration to a defined modeling objective, wherein the defined modeling objective is associated with one or more models;




















a second user input specifying at least a first evaluation data set for the evaluation configuration;






















a third user input specifying at least a first evaluation library for the evaluation configuration, wherein the first evaluation library defines at least a first one or more evaluation metrics; and





















a fourth user input specifying at least a first subset definition for the evaluation configuration, wherein the first subset definition identifies a first plurality of subsets of the first evaluation data set;




















in response to the one or more user inputs:
























creating, storing, and/or updating the evaluation configuration including at least indications of the first evaluation data set, the first evaluation library, and the first subset definition; and

associating the evaluation configuration with the defined modeling objective; and

evaluating, based on the evaluation configuration, the one or more models associated with the defined modeling objective.


See Prong 2




Mental Processes (MPEP § 2106.04(a)(2)(III)): This limitation recite receiving user inputs. Collecting parameters is a cognitive operation that can practically be performed in the human mind or with pen and paper (e.g., comparing model performance metrics against threshold values across dataset subsets). The recitation of computer execution does not negate the underlying mental nature of these evaluation steps. (See, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674)( See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699)

Certain Methods of Organizing Human Activity (MPEP § 2106.04(a)(2)(II)): The limitation recites collecting, organizing, and structuring administrative or organizational specifications for managing models (requesting evaluation configurations) Managing workflows, structuring evaluations, and following rules or instructions for organizing project data fall directly within the sub-grouping of managing behavior and organizational interactions (See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 (2012))
Mental Processes (MPEP § 2106.04(a)(2)(III)): This limitation recite collecting parameters. Collecting parameters is a cognitive operation that can practically be performed in the human mind or with pen and paper (e.g., comparing model performance metrics against threshold values across dataset subsets). The recitation of computer execution does not negate the underlying mental nature of these evaluation steps. (See, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674)( See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699)

Certain Methods of Organizing Human Activity (MPEP § 2106.04(a)(2)(II)): The limitation recites collecting, organizing, and structuring administrative or organizational specifications for managing models. (linking data sets) Managing workflows, structuring evaluations, and following rules or instructions for organizing project data fall directly within the sub-grouping of managing behavior and organizational interactions (See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 (2012))
Mental Processes (MPEP § 2106.04(a)(2)(III)): This limitation recite collecting parameters. Collecting parameters is a cognitive operation that can practically be performed in the human mind or with pen and paper (e.g., comparing model performance metrics against threshold values across dataset subsets). The recitation of computer execution does not negate the underlying mental nature of these evaluation steps. (See, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674)( See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699)

Certain Methods of Organizing Human Activity (MPEP § 2106.04(a)(2)(II)): The limitation recites collecting, organizing, and structuring administrative or organizational specifications for managing models. (linking evaluation libraries) Managing workflows, structuring evaluations, and following rules or instructions for organizing project data fall directly within the sub-grouping of managing behavior and organizational interactions (See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 (2012))
Mental Processes (MPEP § 2106.04(a)(2)(III)): This limitation recite collecting parameters. Collecting parameters is a cognitive operation that can practically be performed in the human mind or with pen and paper (e.g., comparing model performance metrics against threshold values across dataset subsets). The recitation of computer execution does not negate the underlying mental nature of these evaluation steps. (See, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674)( See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699)

Certain Methods of Organizing Human Activity (MPEP § 2106.04(a)(2)(II)): The limitation recites collecting, organizing, and structuring administrative or organizational specifications for managing models. (linking data subset definitions) Managing workflows, structuring evaluations, and following rules or instructions for organizing project data fall directly within the sub-grouping of managing behavior and organizational interactions (See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 (2012))
Mental Processes (MPEP § 2106.04(a)(2)(III)): This limitation recite collecting parameters. Collecting parameters is a cognitive operation that can practically be performed in the human mind or with pen and paper (e.g., comparing model performance metrics against threshold values across dataset subsets). The recitation of computer execution does not negate the underlying mental nature of these evaluation steps. (See, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674)( See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699)

Certain Methods of Organizing Human Activity (MPEP § 2106.04(a)(2)(II)): The limitation recites collecting, organizing, and structuring administrative or organizational specifications for managing models. (requesting evaluation configurations, linking data sets, evaluation libraries, and data subset definitions)Managing workflows, structuring evaluations, and following rules or instructions for organizing project data fall directly within the sub-grouping of managing behavior and organizational interactions (See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 (2012))
Mental Processes (MPEP § 2106.04(a)(2)(III)): This limitation recite collecting parameters. Collecting parameters is a cognitive operation that can practically be performed in the human mind or with pen and paper (e.g., comparing model performance metrics against threshold values across dataset subsets). The recitation of computer execution does not negate the underlying mental nature of these evaluation steps. (See, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674)( See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699)
See Prong 2




See Prong 2

Mental Processes (MPEP § 2106.04(a)(2)(III)): This limitation recite evaluating models based on metrics and data subsets. Evaluating performance against predefined metrics is a cognitive operation that can practically be performed in the human mind or with pen and paper (e.g., comparing model performance metrics against threshold values across dataset subsets). The recitation of computer execution does not negate the underlying mental nature of these evaluation steps. (See, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674)( See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699)

Step 2A, Prong 2: This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d). As per (1) the additional elements are identified as bold parts of the limitations in column 1 of the table below, and as per (2) the evaluation is shown in the mapping section of the table.
In accordance with this step, the judicial exception is not integrated into a practical application.  
Claim 1
Mapping Under Step 2A Prong 2
A computer-implemented method for managing one or more models, the computer-implemented method comprising, by one or more hardware processors executing program instructions:



















receiving, from a user and via one or more graphical user interfaces, one or more user inputs including at least:

a first user input requesting to add an evaluation configuration to a defined modeling objective, wherein the defined modeling objective is associated with one or more models;

a second user input specifying at least a first evaluation data set for the evaluation configuration;

a third user input specifying at least a first evaluation library for the evaluation configuration, wherein the first evaluation library defines at least a first one or more evaluation metrics; and

a fourth user input specifying at least a first subset definition for the evaluation configuration, wherein the first subset definition identifies a first plurality of subsets of the first evaluation data set;

in response to the one or more user inputs:

creating, storing, and/or updating the evaluation configuration including at least indications of the first evaluation data set, the first evaluation library, and the first subset definition; and







associating the evaluation configuration with the defined modeling objective; and










evaluating, based on the evaluation configuration, the one or more models associated with the defined modeling objective.

Additional Element: This limitation recites a general-purpose hardware processors and standard graphical user interfaces merely provides generic machinery as a tool to execute the abstract logic. (See CyberSource v. Retail Decisions, 654 F.3d 1366, 1370, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011))( See, e.g., Versata Development Group v. SAP America, 793 F.3d 1306, 1335, 115 USPQ2d 1681, 1702 (Fed. Cir. 2015)) (2106.05(f))( See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016)) Generic UI interactions for receiving parameters do not improve the technical operation of the computer screen or processor itself.
Field of Use / Insignificant Extra-Solution Activity (MPEP § 2106.05(g), (h)): Applying the management steps to the field of "computer-based models" merely links the abstract idea to a specific technological environment

See Prong 1



See Prong 1




See Prong 1



See Prong 1




See Prong 1





See Prong 1

Additional Element: Storing and updating configuration records in computer memory represent generic, well-understood data management operations specified at a high level of generality. (See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016))(see MPEP § 2106.05(g))

Additional Element: indexing configuration records in computer memory represent generic, well-understood data management operations specified at a high level of generality. (See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016))(see MPEP § 2106.05(g))

See Prong 1


Step 2B: This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP 2106.05. This step determines whether the additional elements amount to "significantly more" than the exception by providing an unconventional technological solution. (see MPEP § 2106.05(g) and see MPEP § 2106.05(h))
Hardware Processors & Memory Storage: Executing instructions, creating data structures, and storing records in memory are generic computer functions recognized by courts as well-understood, routine, and conventional. (see MPEP § 2106.05(g))( 2106.05(d))(
Graphical User Interfaces: Using a GUI to accept user selections and inputs is routine and conventional extra-solution activity. (See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016))( 2106.05(g))
Evaluating Models: Executing evaluation routines based on selected metrics and data subsets is the generic application of the abstract evaluation rule itself. (2106.05(f))( See MPEP § 2106.07(a))( See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016); Intellectual Ventures I v. Symantec, 838 F.3d 1307, 1327, 120 USPQ2d 1353, 1366 (Fed. Cir. 2016); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1417 (Fed. Cir. 2015))
Ordered Combination: The steps follow a conventional logical sequence: receive user selections via GUI → store configuration → link configuration to objective → execute evaluation. Arranging generic computer components to perform routine software data processing steps in their expected order does not yield an inventive combination (MPEP § 2106.05(d)).
Regarding Claim 2
The claim recites, “The computer-implemented method of claim 1, wherein the evaluating comprises at least: executing each of the one or more models associated with the defined modeling objective on the first evaluation data set; determining, for each of the one or more models, and based on the first evaluation library and outputs of each of the one or more models, overall values of the first one or more evaluation metrics; and determining, for each of the first plurality of subsets of the first evaluation data set, and based on the first evaluation library and outputs of each of the one or more models, subset values of the first one or more evaluation metrics.” 
Mathematical Concepts (MPEP § 2106.04(a)(2)): Mathematical calculations, statistical error calculations, and comparisons between predicted and expected numerical values (See SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163).
Mental Processes (MPEP § 2106.04(a)(2)): Comparing data points and observing performance results (See Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1354).
Certain Methods of Organizing Human Activity (MPEP § 2106.04(a)(2)): Basic data analysis and record verification workflows.
Regarding Claim 3 
The claim recites, “The computer-implemented method of claim 2, wherein the first evaluation data set includes first expected values, and wherein the overall values of the first one or more evaluation metrics and the subset values of the first one or more evaluation metrics are determined based, at least in part, on the first expected values.”
Mathematical Concepts (MPEP § 2106.04(a)(2)): Mathematical calculations, statistical error calculations, and comparisons between predicted and expected numerical values (See SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163).
Regarding Claim 4
The claim recites, “The computer-implemented method of claim 3, wherein: specifying the first evaluation library includes at least: identifying a first data field of the first expected values; and
identifying or defining a second data field for model output values.”
Mathematical Concepts (MPEP § 2106.04(a)(2)): Mathematical calculations, statistical error calculations, and comparisons between predicted and expected numerical values ( See SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163).
Regarding Claim 5
The claim recites, “The computer-implemented method of claim 2, further comprising, by the one or more hardware processors executing program instructions: causing display, via one or more graphical user interfaces, of results of the evaluating, wherein the results include at least, for each of the one or more models: the overall values of the first one or more evaluation metrics; and
for each of the first plurality of subsets, the subset values of the first one or more evaluation metrics.”
Additional Elements: displaying results on a GUI.
Generic Display (MPEP § 2106.04(d)): Recites displaying the computed overall and subset values on a GUI. Transmitting and displaying the results of abstract calculations on a conventional computer monitor is well-established insignificant extra-solution activity (See Electric Power Group, 830 F.3d at 1355).

Regarding Claim 6
The claim recites, “The computer-implemented method of claim 1, wherein the one or more models associated with the defined modeling objective comprises a plurality of models, and wherein the evaluating is performed for each of the plurality of models.”
 Additional Elements: Repeating an execution step for a plurality of models
Executing evaluations sequentially or in parallel over multiple models merely applies the abstract data analysis idea across a broader domain.
Regarding Claim 7
The claim recites, “The computer-implemented method of claim 1, wherein:
the one or more user inputs further include: a fifth user input specifying that newly submitted or updated models associated with the defined modeling objective are to be automatically evaluated upon submission of a new model or update of a models; and the computer-implemented method further comprises, by the one or more hardware processors executing program instructions: in response to association of a new model with the defined modeling objective and/or updating of a model of the one or more models associated with the defined modeling objective, automatically evaluating, based on the evaluation configuration, the new model and/or the model.”
Additional Elements: automatically triggering execution upon file/model upload.
Automating a manual task (triggering an evaluation upon receiving an updated file) using generic computer logic is not a technical improvement to computer functionality (Perfect Web Techs., Inc. v. InfoUSA, Inc., 587 F.3d 1324, 1328).
Regarding Claim 8
The claim recites, “The computer-implemented method of claim 1, wherein the computer-implemented method further comprises, by the one or more hardware processors executing program instructions:
causing display, via the one or more graphical user interfaces, of a listing of one or more selectable evaluation libraries, wherein specifying the first evaluation library includes at least: receiving user input selection of the first evaluation library from the of the one or more selectable evaluation libraries.”
Additional Elements: Displaying lists on GUIs.
Generic GUI & Search Functions: displaying menus is  standard database and web application features (Intellectual Ventures I LLC v. Erie Indemnity Co., 850 F.3d 1315, 1327).No Specific Technical Improvement to User Interfaces: Unlike Core Wireless (880 F.3d 1356) or Trading Technologies (675 Fed. App'x 1001), which claimed specific structural improvements to display efficiency or transaction entry accuracy the claim recite high-level generic GUI dropdowns (Interval Licensing LLC v. AOL, Inc., 896 F.3d 1335, 1344).
Regarding Claim 9
The claim recites, “The computer-implemented method of claim 8, wherein the computer-implemented method further comprises, by the one or more hardware processors executing program instructions:
determining, based on one or more evaluation library definitions, the one or more selectable evaluation libraries.”
Mental Processes (organizing and selecting information from a list) (CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1352).
Regarding Claim 10
The claim recites, “The computer-implemented method of claim 9, wherein the one or more evaluation library definitions each include at least: respective parsable descriptions of the evaluation libraries,
respective identifications of user definable data fields, and respective specifications for determining related evaluation metrics.”
Additional Elements: parsing structured text/code descriptions.
Generic GUI & Search Functions: Searching a database is  standard database and web application features (Intellectual Ventures I LLC v. Erie Indemnity Co., 850 F.3d 1315, 1327).
No Specific Technical Improvement to User Interfaces: Unlike Core Wireless (880 F.3d 1356) or Trading Technologies (675 Fed. App'x 1001), which claimed specific structural improvements to display efficiency or transaction entry accuracy, the claim recite high-level, generic GUI dropdowns (Interval Licensing LLC v. AOL, Inc., 896 F.3d 1335, 1344).
Regarding Claim 11
The claim recites, “The computer-implemented method of claim 10, wherein the listing of the one or more selectable evaluation libraries includes respective evaluation library descriptions determined based on the respective parsable descriptions of the respective library definitions.”
Additional Elements: Displaying lists on GUIs.
Generic GUI & Search Functions: displaying menus is  standard database and web application features (Intellectual Ventures I LLC v. Erie Indemnity Co., 850 F.3d 1315, 1327).No Specific Technical Improvement to User Interfaces: Unlike Core Wireless (880 F.3d 1356) or Trading Technologies (675 Fed. App'x 1001), which claimed specific structural improvements to display efficiency or transaction entry accuracy the claim recite high-level generic GUI dropdowns (Interval Licensing LLC v. AOL, Inc., 896 F.3d 1335, 1344).
Regarding Claim 12
The claim recites, “The computer-implemented method of claim 9, wherein the one or more evaluation library definitions are user definable as computer-readable code.”
Inventive Concept / "Significantly More" (MPEP § 2106.05)
Storage of code functions in libraries, user-defined script files represent conventional database and software application engineering practices.

Regarding Claim 13
The claim recites, “The computer-implemented method of claim 8, wherein the listing of the one or more selectable evaluation libraries includes core evaluation libraries and user-defined evaluation libraries.”
Certain Methods of Organizing Human Activity:  (managing data access and user permissions)
Regarding Claim 14
The claim recites, “The computer-implemented method of claim 8, wherein the listing of the one or more selectable evaluation libraries is searchable.”
Inventive Concept / "Significantly More" (MPEP § 2106.05)
text searching represent conventional database and software application engineering practices.
Regarding Claim 15
The claim recites, “The computer-implemented method of claim 8, wherein the listing of the one or more selectable evaluation libraries is determined based on permissions associated with the user.”
Inventive Concept / "Significantly More" (MPEP § 2106.05)
role-based access control represent conventional database and software application engineering practices.
Regarding Claim 16
The claim recites, “The computer-implemented method of claim 1, wherein: specifying the first subset definition includes at least: identifying at least a first data field for subsetting; and
identifying, for the first data field, at least one of: data buckets, data ranges, or a number of data buckets or ranges; and the computer-implemented method further comprises, by the one or more hardware processors executing program instructions: causing display, via the one or more graphical user interfaces, of one or more visualizations of the first evaluation data set based at least in part on the first data field for subsetting.”
Mathematical Concepts: (defining numerical ranges/buckets)
Regarding Claim 17
The claim recites, “The computer-implemented method of claim 16, wherein specifying the first subset definition further includes at least: identifying at least a second data field for subsetting; and
identifying, for the second data field, at least one of: data buckets, data ranges, or a number of data buckets or ranges.”
Mathematical Concepts: (defining numerical ranges/buckets)
Regarding Claim 18
The claim recites, “The computer-implemented method of claim 1, wherein at least one of:
the second user input includes specifying a second evaluation data set; the third user input includes specifying a second evaluation library, wherein the second evaluation library defines at least a second one or more evaluation metrics; or the fourth user input includes specifying a second subset definition, wherein the second subset definition identifies a second plurality of subsets of the first evaluation data set and second evaluation data set, if specified.”
Additional Elements: Repeating data analysis operations across a second dataset.
Comparative data display is generic information presentation. Stating that the comparison is made across two datasets rather than one does not alter the abstract nature of the underlying processing (TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 613).
Regarding Claim 19
The claim recites, “The computer-implemented method of claim 18, wherein the evaluating comprises at least: executing each of the one or more models associated with the defined modeling objective on the first evaluation data set and the second evaluation data set, if specified; determining, for each of the one or more models, and based on the first evaluation library and outputs of each of the one or more models, overall values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified; and determining, for each of the subsets of each of the evaluation data sets, and based on the first evaluation library and outputs of each of the one or more models, subset values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified.”
Mental Processes: (comparing two sets of numerical evaluation results) and Mathematical Concepts (Electric Power Group, 830 F.3d at 1353).
Regarding Claim 20
The claim recites, “The computer-implemented method of claim 19, wherein the computer-implemented method further comprises, by the one or more hardware processors executing program instructions: causing display, via one or more graphical user interfaces, of results of the evaluating, wherein the results include at least, for each of the one or more models: the overall values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified; and
for each of the subsets of each of the evaluation data sets, the subset values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified.”
Additional Elements: displaying comparative metrics side-by-side on a GUI dashboard.
Comparative data display is generic information presentation. Stating that the comparison is made across two datasets rather than one does not alter the abstract nature of the underlying processing (TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 613).
Regarding Claim 21
The claim recites, “The computer-implemented method of claim 1, wherein the computer-implemented method further comprises, by the one or more hardware processors executing program instructions:
communicating, to a large language model (“LLM”), one or more items of information associated with the defined modeling objective; receiving, from the LLM, an output indicative of at least one of:
the defined modeling objective, identify additional models that may be related to the defined modeling objective, the first evaluation data set, the first evaluation library, the first subset definitions, the one or more evaluation metrics, or an evaluation of evaluation metric values; and parsing the output and updating the one or more graphical user interfaces based on the output.”
Additional Elements: Sending text strings over an API network connection to an LLM, receiving a JSON/text response, parsing data strings, and populating GUI text boxes.
Interfacing with AI/LLM Models: Offloading text generation or parameter suggestion to a pre-trained LLM via generic API calls does not provide a technical solution to a technical problem in computer hardware/networks. It uses the LLM as a generic black-box tool to perform abstract consultative tasks (UpHealth Holdings, Inc.).
Parsing and Auto-Populating GUIs: Extracting values from a response stream and filling in GUI input fields is standard, generic front-end software execution (Secured Mail Solutions LLC v. Universal Wilde, Inc., 873 F.3d 905, 911).
Regarding Claim 22
The claim recites, “A system comprising:
one or more computer-readable storage mediums having program instructions embodied therewith; and one or more processors configured to execute the program instructions to cause the system to perform the computer-implemented method of claim 1.”
The claim recite generic computer hardware (processors, memory, computer-readable media) programmed to execute the abstract steps of Claim 1. Under Alice, nominal recitation of a "system" or "computer-readable medium" implementing an abstract method does not convert an ineligible process claim into an eligible machine or manufacture claim (Alice, 573 U.S. at 226; MPEP § 2106.05(f)).
Regarding Claim 23
The claim recites, “A computer program product comprising one or more computer-readable storage mediums having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform the computer-implemented method of claim 1.”
The claim recite generic computer hardware (processors, memory, computer-readable media) programmed to execute the abstract steps of Claim 1. Under Alice, nominal recitation of a "system" or "computer-readable medium" implementing an abstract method does not convert an ineligible process claim into an eligible machine or manufacture claim (Alice, 573 U.S. at 226; MPEP § 2106.05(f)).

Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b)  CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.


The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.


Claims 7-8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA  35 U.S.C. 112, the applicant), regards as the invention.
Claim 7 recites the limitation "upon submission of a new model or update of a models” The combination of the singular article "a" with the plural noun "models" creates an ungrammatical and internally contradictory limitation. It is unclear whether the claim requires an update to a single model or a plurality of models to trigger automatic evaluation. Under MPEP § 2173.05(a), claim language that is ambiguous, inconsistent, or grammatically conflicting fails to set forth the boundaries of the assertion.
Claim 8 recites "receiving user input selection of the first evaluation library from the of the one or more selectable evaluation libraries". The phrase "the of the" represents an incoherent grammatical construction due to an omitted word or typographical repetition. Under MPEP § 2173.05(a), claim language that is opaque, incoherent, or missing essential words renders the metes and bounds of the claim scope indefinite.

Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.

The following is a quotation of the first paragraph of pre-AIA  35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.

Claim 21 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA  35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. 
Claim 21 recites the functional limitations of "communicating, to a large language model (“LLM”), one or more items of information... receiving, from the LLM, an output... and parsing the output and updating the one or more graphical user interfaces based on the output".
Under MPEP § 2161.01(I) and MPEP § 2163, written description support for a computer-implemented functional claim limitation requires the specification to disclose an algorithm (in prose, flowcharts, or logic steps) that achieves the recited computer function. The specification of US 2024/0420258 A1 mentions in [0020] and that LLM outputs may be parsed or formatted, but provides no algorithm, routine, or step-by-step procedure explaining how unstructured natural language text generated by an LLM is parsed into structured parameters (such as subset ranges, metric identifiers, or modeling objectives) to programmatically update graphical user interfaces.


Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA  35 U.S.C. 102 and 103 (or as subject to pre-AIA  35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA  to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.  
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.

The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-20, 22-23 is/are rejected under 35 U.S.C. 103 as being unpatentable over US PGPUB No. US20170124487A1 by Szeto & Chan in further view of NPL: "A DISTRIBUTED PLATFORM FOR MODELEVALUATION AND BENCHMARKING AT SCALE" by Dakkak et al.
Regarding Claim 1
Szeto teaches A computer-implemented method for managing one or more models, the computer-implemented method comprising, by one or more hardware processors executing program instructions: ([Abstract]: “In accordance with disclosed embodiments, there are provided systems, methods, and apparatuses for implementing machine learning model training and deployment with a rollback mechanism within a computing environment...” [0436]: “In accordance with a particular embodiment there is non-transitory computer readable storage media having instructions stored thereupon that, when executed by a processor and memory of a machine learning platform, the instructions cause the machine learning platform to perform operations including...” [Claim 1]: “A method performed by a machine learning platform having at least a processor and a memory therein, wherein the method comprises:...” 
[FIG. 26]:

    PNG
    media_image1.png
    848
    720
    media_image1.png
    Greyscale

See [0454] for support.
The examiner interprets where A computer-implemented method for managing one or more models, the computer-implemented method comprising, by one or more hardware processors executing program instructions is shown in a computer-implemented platform (machine learning platform 2600) having a processor 2690 and a memory 2695 to execute a predictive engine generator 2685 for implementing machine learning model training and deployment.) receiving, from a user and via one or more graphical user interfaces, one or more user inputs including at least: ([0451-0453]: “In particular, there is depicted a configuration editor 2584 which provides an API or UI through which a developer having provided training data as input for training a machine learning model may enter the requisite values as mandated by the configuration, or select changeable values from a template or otherwise specify tuning parameters and other configuration information, such as algorithms and algorithm parameters for use in training the machine learning model for the creation of the new predictive engine variant. The model Configuration Editor 2584 provides means by which to enter such configuration information and may be provided by the source code of the model or the machine learning platform to allow for entry of the values for the model configuration information. Element 2581 provides a confirmation box to save the entered information provided by the developer or user as entered into the model configuration editor 2584.”
[FIG. 25]:

    PNG
    media_image2.png
    848
    810
    media_image2.png
    Greyscale

The examiner interprets where receiving, from a user and via one or more graphical user interfaces, one or more user inputs including at least… is shown in a Model Configuration Editor 2584 operating at a computing device 2501 (such as a PC, mobile, smartphone, or tablet) which provides a graphical user interface (GUI) through which a developer or user may enter or select configuration information, such as algorithms and algorithm parameters, and click a confirmation box 2581 to save the entered information.) a first user input requesting to add an evaluation configuration to a defined modeling objective,  wherein the defined modeling objective is associated with one or more models; ( [0361]: “More particularly, FIG. 22 depicts an exemplary machine learning deployment scenario using PredictionIO λ or the machine learning platform, implemented according to one embodiment. Tenant 1 owns a Git repository containing machine learning code that produces models. PredictionIO λ or the machine learning platform associates itself with the tenant and the Git Repository. Users of PredictionIO λ or the machine learning platform can run code from different commit to produce models. Advanced users can define multiple data sources and multiple experiments to be run in parallel. Such tools provided by PredictionIO λ or the machine learning platform increases productivity by allowing quick insights into any problems regarding experiment or deployment runs.”
[FIG. 22]:

    PNG
    media_image3.png
    848
    676
    media_image3.png
    Greyscale

 “According to the depicted embodiment, the machine learning platform 2600 includes the processor 2690 and the memory 2695 to execute instructions at the machine learning platform 2600; a database system 2650 to store a plurality of machine learning models 2643, each such model including configuration parameters 2636 defined by a developers, templates, or the machine learning platform. Further stored within database system 2650 are a plurality of transactions defining features and labels representing correct answers for the features from which the machine learning models 2643 may be trained based on, for example, the configuration parameters 2636 and other provided configuration information 2639.”
See [0057], [0220], [0363-0364],  [0387-0388], [0455] for support.
The examiner interprets where a first user input requesting to add an evaluation configuration to a defined modeling objective, wherein the defined modeling objective is associated with one or more models is shown in the database 2650 stores a plurality of machine learning models 2643. A user can initialize a project or tenant (representing a specific type of prediction problem, i.e., a modeling objective such as product recommendation or fraud detection). A user can add or remove "evaluations" (evaluation configurations) to/from the project, which are run sequentially or in parallel. A model parameter set (governing DataSource, Algorithm, Serving, and Evaluator components) specifies an engine variant (evaluation configuration). The user requests to add/run these evaluations on the models associated with the project.) a second user input specifying at least a first evaluation data set for the evaluation configuration; ([0288-0289]:”…

    PNG
    media_image4.png
    376
    996
    media_image4.png
    Greyscale

“Parameter set 814 states that variant 820 uses DataSource x1, and Algorithms 1 and 2. The values of algorithm parameter1 and algorithm parameter2 of Algorithm 1 are set to a1 and a2, while the value of the parameter Z of Algorithm 2 is set to 23.”
[FIG. 8]:

    PNG
    media_image5.png
    818
    554
    media_image5.png
    Greyscale

See [0365], [0431], [0433] for support.
The examiner interprets where a second user input specifying at least a first evaluation data set for the evaluation configuration is shown in the developer or operator specifying "additional selection of data sources for use as training data input", and parameter sets (e.g., set 814) specify the use of "DataSource X1". The user can define multiple data sources and multiple evaluations to run in parallel, which is received as configuration input representing an evaluation dataset.) a third user input specifying at least a first (See [FIG. 8 & 23], [0005]: “…A simple example is a recommendation engine for suggesting a certain number of products to a customer based on pricing, product availabilities, product similarities, current sales strategy, and other factors. Such recommendations may be personalized to a specific user by taking into account user purchase history, browsing history, geographical location, or other user preferences or settings. Exemplary software tools for building machine learning systems include Apache Spark MLlib, Apache Mahout™, scikit-learn, and R.”  
[0257]: “In addition, to evaluate the performance of the prediction process to compare different algorithms, algorithm parameter settings, as well as different engine variants, an Evaluator component 450 receives data from Serving component 440, and applies one or more metrics to compute evaluation result 455 as an output. An engine variant is a deployable instance of a predictive engine, specified by an engine parameter set. The engine parameter set includes parameters that control each component of a predictive engine. ...”  
[0260]: “In certain embodiments, Apache Spark can be used to power the Data, Algorithm, Serving, and Evaluator components. Apache Spark is a large-scale data processing engine. In this case, distributed algorithms and single-machine algorithms may both be supported by the PredictionIO or machine learning Serve.” 
See [0253], [FIG. 4], [FIG. 3A-3B], [0293], [FIG. 5A-5B], [FIG. 6A-6B], [FIG. 7A], [0431] for support.
The examiner interprets where a third user input specifying at least a first evaluation library for the evaluation configuration, wherein the first evaluation library defines at least a first one or more evaluation metrics is shown in the engine parameters specifying an Evaluator component 450 which uses metrics to evaluate the predictive process and using standard software libraries like "Apache Spark, MLlib, scikit-learn, or R" to power the Evaluator component, and using metrics (e.g., metrics 842 and 844).) a fourth user input specifying at least a first subset definition for the evaluation configuration, wherein the first subset definition identifies a first plurality of subsets of the first evaluation data set; ([0230]: “... For example, iterative replay of dynamic queries, corresponding predicted results, and subsequent actual user actions may provide to operators insights into the tuning of data sources, algorithms, algorithm parameters, as well as other system parameters that may affect the performance of the machine learning system. Prediction performances may be evaluated in terms of prediction scores and visualized through plots and diagrams. By segmenting available replay data, prediction performances of different engines or engine variants may be compared and studied conditionally for further engine parameter optimization.” 
[0233]: “Another feature of the disclosed embodiments is that it is capable of tracking multiple user actions, behaviors, or responses both immediately and over a delayed time frame. Sequences of user actions, such as mouse clicks followed by an online purchase, may be grouped and tracked under the same tracking tag or replay tag associated with a particular query. In addition, user actions may be tracked across different sessions, cohorts, according to different segmentation rules.”
 [0330]: “…. In some embodiments, this time period of interest may cover one or more specific dates, consecutive or non-consecutive, or a range of dates. The vertical axis 1320 with label 1325 refers to accumulated prediction scores. Recall that each query may have a timestamp indicating the time and/or date at which the query has been received or when a prediction has been made by a PredictionIO or machine learning platform in response to the query. Although not shown explicitly here, Replay Group 1 may have been obtained through a query segment filter that selects all queries with timestamps within January, 2015..”

[FIG. 13]:

    PNG
    media_image6.png
    830
    650
    media_image6.png
    Greyscale


See [0350-0353] & [FIG. 10, 19-20] for further support.
The examiner interprets where a fourth user input specifying at least a first subset definition for the evaluation configuration, wherein the first subset definition identifies a first plurality of subsets of the first evaluation data set is shown in the testing/evaluation data being segmented or filtered. Specifically,  teaches "By segmenting available replay data, prediction performances of different engines or engine variants may be compared and studied conditionally." [0233] teaches that "user actions may be tracked... according to different segmentation rules." [0330] teaches that "Replay Group 1 may have been obtained through a query segment filter that selects all queries with timestamps within January, 2015." [0350-0353] teach that an operator can select a period of time on the chart to display a table of query records that fall into this time period (subpopulation subsets). This represents receiving a user input specifying a segmentation rule or query segment filter (subset definition) which segments the evaluation records (the evaluation dataset) into specific Replay Groups/cohorts (subsets) for granular, conditional performance analysis.) in response to the one or more user inputs: ([0414]: “… Method 2300 may be performed by processing logic that may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device) to perform various operations such as receiving, specifying, generating, versioning, deploying, rolling back, downrevving, storing, maintaining, executing, creating, operating, restricting, returning, presenting, interfacing, communicating, transmitting, querying, processing, providing, determining, triggering, displaying, retrieving, updating, sending, etc., in pursuance of the systems and methods as described herein…”



[FIG. 23]:

    PNG
    media_image7.png
    830
    564
    media_image7.png
    Greyscale

See [FIG. 24] for support.
The examiner interprets where in response to the one or more user inputs is shown in performing processing actions (such as generating variants, saving configurations, executing evaluations) in response to receiving user inputs.) creating, storing, and/or updating the evaluation configuration including at least indications of the first evaluation data set, the first evaluation library, and the first subset definition; and (See [FIG. 26], [0455]: “.. database system 2650 to store a plurality of machine learning models 2643, each such model including configuration parameters 2636 defined by a developers, templates, or the machine learning platform. Further stored within database system 2650 are a plurality of transactions defining features and labels representing correct answers for the features from which the machine learning models 2643 may be trained based on, for example, the configuration parameters 2636 and other provided configuration information 2639.” [0459]: “…a receive interface 2626 to receive training data as input at the machine learning platform 2600, wherein the training data includes a multiple transactions 2637, each of the transactions specifying a plurality of features upon which to make a prediction and a label representing a correct answer for the plurality of features according to each respective transaction 2637; the receive interface 2626 to receive a specified model 2643 to be trained by the machine learning platform 2600 using the training data, wherein the model 2643 includes a plurality of algorithms and source code; a predictive engine generator 2685 to generate a new predictive engine variant 2667 by training the model to algorithmically arrive upon the label representing the correct answer as provided with the training data based on the plurality of features for each of the multiple transactions; …”
See [0221] & [FIG. 1B] for support.
 The examiner interprets where creating, storing, and/or updating the evaluation configuration including at least indications of the first evaluation data set, the first evaluation library, and the first subset definition is shown in storing the machine learning models, configuration parameters, transactions, and engine variants in database system 2650, and persistently storing a link to every generated variant with unique IDs.) associating the evaluation configuration with the defined modeling objective; and (See [FIG. 1B], [0221], [0363-0364], [0387-0388], [0459]: The examiner interprets where associating the evaluation configuration with the defined modeling objective is shown in associating the created engine variants (evaluation configurations) with specific projects or tenants (modeling objectives) and database system entries, representing the association of the configuration with the defined modeling objective.) evaluating, based on the evaluation configuration, the one or more models associated with the defined modeling objective. (See [FIG. 8], [0293],  [0229-0330]: “In addition to creating and deploying machine learning systems, described herein are methods and systems for evaluating and tuning machine learning systems in production. For instance, variants of predictive engines and algorithms are evaluated by an evaluator, using one or more metrics with test data…
 Evaluation results thus generated are used in automatic parameter set generation and selection for the machine learning system. Multiple instances of a predictive engine, or engine variants, may be evaluated at the same time and subsequently compared to determine a dynamic allocation of incoming traffic to the machine learning system…
 Prediction performances may be evaluated in terms of prediction scores and visualized through plots and diagrams. By segmenting available replay data, prediction performances of different engines or engine variants may be compared and studied conditionally for further engine parameter optimization.” The examiner interprets where evaluating, based on the evaluation configuration, the one or more models associated with the defined modeling objective is shown in the evaluator component 450 (or evaluator 840) evaluates the performances of the model variants based on the parameter sets (evaluation configuration), datasets, and segmentation rules (query segment filters), computing prediction scores and statistical summaries across the subsets/replay groups.)
	Szeto does not explicitly disclose that the user input specifies an "evaluation library" defining one or more evaluation metrics, nor does it explicitly show a unified modular assembly of the dataset, library, and subset definitions within a single configuration.
However, Dakkak teaches evaluation library ([Section 3.1]: “All models in MLModelScope are described using a model specification (called manifest). The manifest is a text file and includes the information needed to run a model. The manifest specifies information such as the model pre- and post-processing steps, its model sources (graph and weight), and its software stack..
Built-in Pre-/Post-Processing Functions − As vision models are widely used and their pre- and
post-processing operations are less diverse, MLModelScope allows for common pre-processing
image operations (e.g. image decoding, resizing, and normalization) and post-processing operations
(e.g. topK, IOU, mAP, etc.) to be used within the manifest without writing code. ..
[Listing. 1]: 

    PNG
    media_image8.png
    724
    506
    media_image8.png
    Greyscale

The examiner interprets where evaluation library is shown in Dakkak teaching that all models are described using a model specification manifest (evaluation configuration) that specifies software stack components, containers, and allows defining pre- and post-processing functions using standard or custom code libraries. Specifically, Dakkak describes Built-in Pre-/Post-Processing Functions, where it allows for common post-processing operations (such as topK, IOU, mAP, etc.) "to be used within the manifest without writing code. Internally, MLModelScope invokes built-in pre- and post-processing code to perform these operations".) Dakkak further teaches in response to the one or more user inputs ([Section 3.3]: “… Figure 2 shows the evaluation flow of a user’s request. 1 On system startup, each agent publishes the hardware it is running on to the registry. This information is made visible to the MLModelScope orchestration layer. 2 A user then uses MLModelScope’s UI to request an evaluation by specifying the model, framework, and hardware constraints. 3 An API request is then performed to the remote API handler, which then 4 queries the registry to find an agent which satisfies the user’s constraints. 5 The request is then forwarded to one (or all) of the agents capable of running the evaluation. The agents then provision the hardware and software environment and run the model. 6 The agents then collect and publish the results to a centralized evaluation database. 7 Finally, an evaluation summary is presented to the user.”

[FIG. 2]: 

    PNG
    media_image9.png
    780
    740
    media_image9.png
    Greyscale

 The examiner interprets where  in response to the one or more user inputs is shown in performing processing actions (such as generating variants, saving configurations, executing evaluations) in response to receiving user inputs) creating, storing, and/or updating the evaluation configuration including at least indications of the first evaluation data set, the first evaluation library, and the first subset definition; (See [Section 3.1]: The examiner interprets where creating, storing, and/or updating the evaluation configuration including at least indications of the first evaluation data set, the first evaluation library, and the first subset definition is shown in creating and storing the "evaluation manifest" (evaluation configuration) that includes indications of the dataset (e.g., ILSVRC 2012), the software libraries/frameworks, and pre-/post-processing functions. When combined, the processor of Szeto creates, stores, or updates the evaluation configuration (comprising the model parameters, DataSource, and Evaluator metric library/segment filters) in the database.)
	It would have been obvious to a POSITA before the effective filing date of the invention to integrate Dakkak’s modular, manifest-defined "evaluation libraries" (built-in post-processing functions or libraries that define evaluation metrics like topK, IOU, mAP) into the platform of Szeto, and to provide them as selectable options in the GUI of Szeto.
The modification represents a straightforward combination of known prior art elements according to established techniques to yield predictable results. Specifically, utilizing Dakkak’s technique of encapsulating standard evaluation metrics into modular software libraries and specifying them in the evaluation configuration allows developers to easily swap, reuse, and standardize evaluation metrics across different model variants without manual code modification, reducing development overhead and ensuring consistent benchmarking.
Szeto and Dakkak fails to explicitly teach a specific domain modeling objectives (e.g., real estate /house price prediction modeling objectives). However, Ravikumar further teaches a third user input specifying at least a first evaluation library for the evaluation configuration, wherein the first evaluation library defines at least a first one or more evaluation metrics ([Abstract]: “… The results and evaluation of these models using the machine learning and advanced data mining tools like Weka, Rapid Miner will have the more influence in the price prediction.”

[Section 5.2]:

    PNG
    media_image10.png
    446
    698
    media_image10.png
    Greyscale

The examiner interprets where a third user input specifying at least a first evaluation library for the evaluation configuration, wherein the first evaluation library defines at least a first one or more evaluation metrics is shown in specifying evaluation libraries/tools (Weka, RapidMiner, R) and metrics (RMSE, R²) a fourth user input specifying at least a first subset definition for the evaluation configuration, wherein the first subset definition identifies a first plurality of subsets of the first evaluation data set ([Section 3.5.1]:

    PNG
    media_image11.png
    264
    708
    media_image11.png
    Greyscale

[Section 3.5.2]:

    PNG
    media_image12.png
    128
    700
    media_image12.png
    Greyscale

The examiner interprets where  a fourth user input specifying at least a first subset definition for the evaluation configuration, wherein the first subset definition identifies a first plurality of subsets of the first evaluation data set is shown in defining data subsets/attributes and evaluating models across partitioned data fields.)
It would have been obvious to a person of ordinary skill in the art (POSITA) before the effective filing date of the invention to combine the teachings of Dakkak, Szeto, and Ravikumar. A POSITA would have been motivated to modify Dakkak’s Web UI by incorporating Szeto's interactive graphical configuration editor (Element 2584) and query segmentation filters in order to provide an intuitive, non-coded GUI interface for creating, storing, and editing model parameter configurations and data subset definitions. Furthermore, a POSITA would have been motivated to incorporate Ravikumar's teaching of integrating modular third-party evaluation libraries (Weka, RapidMiner) and dataset subset partitioning into Dakkak’s extensible manifest framework to enable slice-based benchmarking across housing price-specific modeling objectives. Combining these known teachings applies routine software engineering principles to yield the predictable result of a standardized, user-friendly model evaluation and management platform.
Regarding Claim 2
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 1, wherein the evaluating comprises at least: (See claim 1) executing each of the one or more models associated with the defined modeling objective on the first evaluation data set; (Szeto: {See [FIG. 3A-B], [FIG. 8], [0264]: “…One or more metrics can be defined to compare predicted results returned from the engine with actual results among the validation data. The goal of engine parameter tuning is to determine an optimal engine parameter set that maximizes evaluation metric scores. The higher the score, the better the engine parameter set...” 
[0290]: “According to various embodiments, the evaluation and tuning processes may start at either deployment platform 812 or user application 880. For example, after deployment platform 812 deploys engine variant 820 and engine variant 822, user application 880 may send three queries Q1, Q2, and Q3 (882) to the PredictionIO or machine learning platform 805. In some embodiments, a query may include identifying information including user ID, product ID, time, and location...
Deployed engine variant 820 then generates predicted results 824 including predicted result P1 with replay ID X, and predicted result P3 with replay ID Z. Replay IDs in this example are alphanumeric tracking tags specific to individual queries...” The examiner interprets where executing each of the one or more models associated with the defined modeling objective on the first evaluation data set is shown in using the processor to execute the models on an evaluation dataset. Specifically, teaches that "the validation set is used to validate the engine by querying the engine with the validation set data" (validation set representing the evaluation dataset). [0290] teaches that "user application 880 may send three queries Q1, Q2, and Q3... Deployed engine variant 820 then generates predicted results 824...". FIG. 3B illustrates the querying (executing) of the predictive engine 320 to produce prediction results. 
See Dakkak: {[Section 3.1]: The examiner interprets where executing each of the one or more models associated with the defined modeling objective on the first evaluation data set is shown in the platform executing model evaluations over the user-specified benchmarking dataset (e.g., ILSVRC 2012 / ImageNet) by deploying the models in a containerized environment and performing inference on the dataset.) determining, for each of the one or more models, and based on the first evaluation library and outputs of each of the one or more models, overall values of the first one or more evaluation metrics; and (Szeto: {See [0257], [0264], [0293], [0060]: ”Actual result” and “A” includes correct values, actual events, as well as user actions or “subsequent end-user behaviors.” Actual results can be correct values to predictive problems such as classifications, actual outcomes or results of future events, and/or any user actions or behaviors from the end-user device specifying what the end-user has done in response to a prediction result provided in response to a query, and so on. Actual results include actual outcomes in the case of a prediction engine predicting actual events..” 
[0063]: “…A prediction score is calculated by at least one pre-defined or operator-defined score function, based on prediction result(s) and actual result(s) associated with the query.” 
The examiner interprets where determining, for each of the one or more models, and based on the first evaluation library and outputs of each of the one or more models, overall values of the first one or more evaluation metrics is shown in determining overall values of evaluation metrics based on model outputs. Specifically, teaches that the "Evaluator component 450 receives data from Serving component 440, and applies one or more metrics to compute evaluation result 455 as an output... An evaluation metric may quantify prediction accuracy with a numerical score." [0264] teaches that "One or more metrics can be defined to compare predicted results returned from the engine with actual results among the validation data... the goal of engine parameter tuning is to determine an optimal engine parameter set that maximizes evaluation metric scores... a precision score may be used to measure..." [0293] teaches that "two metrics 842 and 844 are used within evaluator 840... Evaluation results are sent to auto parameter tuning..." . This represents determining overall values of the metrics across the entire validation dataset. See Dakkak: {[Section 3.1]: The examiner interprets where determining, for each of the one or more models, and based on the first evaluation library and outputs of each of the one or more models, overall values of the first one or more evaluation metrics is shown in the platform loads pre-configured software stack containers (framework libraries) and invokes post-processing functions (evaluation libraries) to compute overall performance and accuracy metrics (e.g., topK, IOU, mAP) based on the outputs of the models. A POSITA would combine Szeto and Dakkak to compute these overall metrics using Dakkak’s modular, manifest-specified evaluation libraries.) determining, for each of the first plurality of subsets of the first evaluation data set, and based on the first evaluation library and outputs of each of the one or more models, subset values of the first one or more evaluation metrics. (Szeto: {See [FIG. 10, 13],  [0230], [0233], [0324]: “... A replay group refers to a pre-defined or operator-defined segment of queries that satisfy one or more conditions as provided through query segment filters. Replay groups may be created for textual or visual displays. Examples of query segment filters include engine variant filters, user attribute filters, item attribute filters, query attribute filters, and other property filters or conditional filters capable of selecting a subset of available queries for performance analysis and monitoring…
 an item attribute filter may be applied if queries contain at least an item; and an query time attribute filter may be applied if queries have associated timestamps...
since query segment filters select subsets of queries without necessarily affecting the prediction process, they may be applied during any stage of the predictive engine tuning, evaluation, and replay process...”
See [FIG. 9 & 11, 14-18] for support.
The examiner interprets where determining, for each of the first plurality of subsets of the first evaluation data set, and based on the first evaluation library and outputs of each of the one or more models, subset values of the first one or more evaluation metrics is shown in calculating evaluation metrics over each of the subsets of the evaluation dataset. Specifically, [0230] teaches that "prediction performances may be evaluated in terms of prediction scores... By segmenting available replay data, prediction performances of different engines or engine variants may be compared and studied conditionally." [0233] teaches that "Instead of just returning prediction success or failure rates... operators can replay the whole prediction scenario... according to different segmentation rules." [0324] teaches that "Replay Groups refer to a pre-defined or operator-defined segment of queries... query segment filters include engine variant filter, user attribute filter, item attribute filter... capable of selecting a subset... in addition, since query segment filters select subsets of queries... they may be applied during any stage of the predictive engine tuning, evaluation, and replay process.". Applying these segmentation rules/filters (subset definitions) to partition the evaluation data and conditionally evaluate performances represents determining subset values of the evaluation metrics for each of the subsets. 
See Dakkak: {[Section 3.1] ], [Abstract]: “... This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking. We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope’s capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.” 
The examiner interprets where determining, for each of the first plurality of subsets of the first evaluation data set, and based on the first evaluation library and outputs of each of the one or more models, subset values of the first one or more evaluation metrics is shown in built-in post-processing functions (evaluation libraries) defining evaluation metrics. In the combination, it would have been obvious to a POSITA to utilize Dakkak’s modular, manifest-specified evaluation libraries to calculate the metrics not only globally (overall), but also conditionally over the partitioned data subsets (Replay Groups) of Szeto’s to track subpopulation-specific performance.)
	Motivation to combine/ Rationale to combine similar to dependent on claim. See dependent on claim.
Regarding Claim 3
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 2, (See claim 2) wherein the first evaluation data set includes first expected values, (Szeto: {See [0060], [FIG. 4, 8, 20], [0264], [0293], [0258]: “.. An actual result refers to a correct prediction result or an actual outcome of a prediction task. If a predicted result is the same as an actual result, the predicted result can be considered as an excellent prediction. Recall the exemplary queries and corresponding predicted results discussed with reference to FIG. 2A. In the classification task, an actual result may be the string “complaint”, which is a correct classification of the text input. In the similar item recommendation task, an actual result may be product IDs (P10, P20), indicating that products P10 and P20 are similar to given items (P1, P2, P3), although the predictive engine suggests products P10 and P11. In a personalized recommendation task, an actual user behavior may be product IDs (P10, P20), indicating that the user selected products P10 and P20 for further viewing and purchase, after products P10 and P11 are recommended by the predictive engine. Another example of actual results is in algorithmic trading, where an actual result may be the actual opening or closing price of a particular stock on the next day. Actual results may be collected through user devices, read from storage, or simulated.”
[FIG. 2A]:

    PNG
    media_image13.png
    784
    1010
    media_image13.png
    Greyscale

The examiner interprets where  the first evaluation data set includes first expected values is shown in the validation/testing data includes ground-truth values (termed "actual results" or "A"). Specifically, [0060] defines "Actual result" and "A" as including "correct values, actual events... Actual results can be correct values to predictive problems such as classifications, actual outcomes or results of future events..." [0258] teaches that "an actual result refers to a correct prediction result or an actual outcome of a prediction task" [0264] states that "actual results among the validation data" are used during validation”. These "actual results" are the structural and functional equivalents of "expected values" under the BRI. 
See Dakkak: {[Section 3.3], [Section 3.1]: The examiner interprets where  the first evaluation data set includes first expected values is shown in running model evaluations over standard datasets (e.g., ILSVRC 2012 / ImageNet) which contain ground-truth target annotations (first expected values).)  and wherein the overall values of the first one or more evaluation metrics and the subset values of the first one or more evaluation metrics are determined based, at least in part, on the first expected values.  (Szeto: {See [0060, [0257-0258], [0264], [FIG. 8, 5A-7B, 13, 19-20] [0292-0293]: “In this embodiment, at user application 880, user actions and/or behaviors collected subsequent to receiving predicted results P1, P2, and P3 (886) from the PredictionIO or machine learning platform 805 are considered as actual results A1, A2, and A3 (884) respectively, and tagged with corresponding Replay IDs. Such user actions may be collected in real-time, or over a given time span such as a few hours, a day, or a week. Recall that each query evokes a prediction process to generate a predicted result, and each query is uniquely identified by a replay ID. Hence, multiple user actions or actual results corresponding to a particular query with a given replay ID may be tagged with the same replay ID. For example, actual result A1 shown in FIG. 8 may represent a sequence of user clicks and browsed product pages, all corresponding to query Q1, product recommendation P1 and replay ID X. After actual results 884 are transferred to datastore 830, engine variant parameter sets, queries, predicted results, and actual results corresponding to the same Replay ID are aggregated within datastore 830, using the data source (data reader) or event datastore mentioned above. Aggregated data sets 832 are sent to evaluator 840 for evaluation...” 
See [0327] for support.
 The examiner interprets where the overall values of the first one or more evaluation metrics and the subset values of the first one or more evaluation metrics are determined based, at least in part, on the first expected values is shown in both overall and subset evaluation metrics calculated based on comparing model outputs with these actual results (expected values). Specifically, [0060] teaches that "The deviation of P [predicted result] from A [actual result] can be used to define a metric of the accuracy or correctness". [0257] teaches that "Evaluator component 450 receives data... and applies one or more metrics to compute evaluation result 455". [0264] teaches that "One or more metrics can be defined to compare predicted results returned from the engine with actual results among the validation data". This represents calculating overall values of the metrics across the entire validation dataset. [0293] discloses that actual results are aggregated with predicted results and queries under the same Replay ID (representing subsets/replay groups) and "Aggregated data sets 832 are sent to evaluator 840 for evaluation". [0327] teaches that "performance... over a replay group is quantified... by a prediction score... calculated by at least one... score function based on the predicted result(s) and actual result(s) associated with the query". Since a "replay group" is a subset of the dataset, determining these prediction scores (metrics) for the replay group based on actual results (expected values) directly teaches determining subset values of the metrics based on the expected values. FIG. 19 and FIG. 20 show tables (table 1980, table 2080) of query records for replay groups showing the mapping of Predicted Result (P) and Actual Result (A) to compute a Prediction Score (Score). 
See Dakkak: {[Section 3.1]: The examiner interprets where the overall values of the first one or more evaluation metrics and the subset values of the first one or more evaluation metrics are determined based, at least in part, on the first expected values is shown in built-in post-processing functions (evaluation libraries) defining evaluation metrics like topK, IOU, and mAP, which inherently require comparing the model outputs (predictions) against the ground-truth expected values in the dataset. In the combination, it would have been obvious to a POSITA to utilize Dakkak’s modular, manifest-specified evaluation libraries to calculate these metrics (such as topK accuracy or mAP) globally (overall) and subset-wise (over Replay Groups) by performing the necessary comparison against the expected values (actual results) included in the evaluation dataset of Szeto.)
	The motivation to combine is similar to the dependent on claim. See dependent on claim.

Regarding Claim 4
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 3, (See claim 3) wherein: specifying the first evaluation library includes at least: identifying a first data field of the first expected values; (Szeto: {See [0060], [0258], [0292-0293], [0327], [FIG. 8, 19-20]: The examiner interprets where identifying a first data field of the first expected values is shown in actual results (the first expected values) stored in the event datastore and aggregated with predicted results and queries under corresponding tracking/replay IDs. For example, FIG. 19 and FIG. 20 show data records mapping the query, predicted result, and actual result (A) in tables 1980/2080. The "actual result" represents a specific data column or field in the aggregated dataset. 
See Dakkak: {[Section 3.1], [FIG. 1-2], [Section 3.3], [Listing 1], [Section A3.2]: “As discussed in the  main body of the paper, replication of model accuracy and performance results is dependent on: the usage of specific HW/SW stack; the training dataset; and the pre/post-processing steps on the inputs and outputs. MLModelScope specifies these requirements via a model manifest file described in Section 3. The manifest tells MLModelScope the HW/SW stack to instantiate and how to evaluate the model…”
See [Section A.3.4] for support.
The examiner interprets identifying a first data field of the first expected values is shown in the evaluation configuration is specified in a model manifest file (model specification) that defines the dataset, software stack, and evaluation metrics. The dataset manager exposes a consistent API to retrieve values and iterate through the dataset. In the combination, standard evaluation libraries and metrics (e.g., accuracy, precision, topK, Mean Squared Error) compute their values by comparing predicted results against ground-truth actual results (expected values). In any standard evaluation library or API (such as those in scikit-learn, TensorFlow, or PyTorch), this computation inherently requires identifying which data field or column in the database/dataset contains the true ground-truth targets (expected values) to serve as the true label input (e.g., y_true). Thus, specifying the first evaluation library in the combined system of Szeto and Dakkak would have rendered it obvious to a POSITA to identify a first data field of the expected values (actual results) within the dataset layout or database schema so the evaluation library can access the values and compute the performance metrics.) and identifying or defining a second data field for model output values. (Szeto: {[0292-0293], [FIG. 8, 19, 20]}: The examiner interprets where identifying or defining a second data field for model output values is shown in the model output values (predicted results) are generated by the deployed variants in response to user queries, and passed to the event data store where they are stored as Predicted Result (e.g. P1, P2, P3). FIG. 19 and FIG. 20 show tables 1980/2080 displaying predicted results (P) alongside actual results (A). 
See Dakkak: {See [Section 3.1], [FIG. 1], [Section A.3.3]: “A predictor is a thin abstraction layer that exposes a framework through a common API. A predictor is responsible for evaluating models (using the manifest file) and capturing the results along with the framework’s profile information. A predictor publishes its HW/SW stack information to MLModelScope’s registry at startup, can have multiple instantiations across the system, and is managed by MLModelScope’s middleware”
 [Section 2]: “...The model inference stage calls the framework’s inference API on the processed input and produces an output. The post-processing stage transforms the model output to a form that can be viewed by a user or used to compute metrics. Framework level ( L2 ) performs model inference by executing the layers in the model graph using a framework such as TensorFlow, MXNet, or PyTorch. Layer level ( L3 ) executes a sequence of ML library calls for layers such as convolution, normalization, or softmax. ML Library level ( L4 ) invokes a chain of system library calls for functions in ML libraries such as cuDNN (Chetlur et al., 2014), MKL-DNN (MKL-DNN) or OpenBLAS (Xianyi et al., 2014)...”
 The examiner interprets where identifying or defining a second data field for model output values is shown in the model inference stage calls the framework's inference API on the processed input and produces an output, and that the post-processing stage transforms the model output to a form used to compute metrics. The predictor captures the results along with the framework's profile information. In the combination, computing evaluation metrics (e.g., precision, accuracy, topK, Mean Squared Error) inherently requires identifying or defining the model output field (predictions) from the model variants to serve as the predicted input (e.g., y_pred) to the evaluation metrics. Thus, specifying the evaluation library in the combined system of Szeto and Dakkak would have rendered it obvious to a POSITA to identify or define a second data field for the model output values (predictions) within the database schema or dataset layout so that the evaluation library can retrieve the predictions and align them with the expected values to compute the metrics.
	The motivation to combine is similar to the dependent on claim. See dependent on claim.
Regarding Claim 5
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 2, (See claim 2) further comprising, by the one or more hardware processors executing program instructions:
causing display, via one or more graphical user interfaces, of results of the evaluating, (Szeto: {[0230], [0234], [FIG. 10-11], [0234]: “…Instead of just returning prediction success or failure rates for determining whether one variant performs better than another, disclosed embodiments can replay the whole prediction scenario, from engine parameters, queries, prediction results, to actual results, user interactions, and evaluation metrics, to help developers understand particular behaviors of engine variants of interest, and to tailor and improve prediction engine design. The graphical or textual visual replay of evaluation and tuning results not only makes the whole process easier to use, but also allows interactive engine parameter tuning by an operator.” 
 [0413]: “Still further benefits of the replay means include providing of visual elements in a visual replay mode which are graphical and/or textual, giving more insight to the operators; evaluating user interactions and more intuitive means by which to tune the engine and algorithms, for instance, by changing the scenario ad-hoc based on replay results such as making modifications to the email header, and replaying how results will perform for that engine variant, etc. ...”
[FIG. 25]:

    PNG
    media_image14.png
    1146
    1110
    media_image14.png
    Greyscale

See [0304], [0311-0312], [0450-0451] for support.
The examiner interprets where causing display, via one or more graphical user interfaces, of results of the evaluating is shown in Szeto explicitly discloses causing the display of evaluation results via a graphical user interface (GUI). Specifically, [0230] teaches that "Prediction performances may be evaluated in terms of prediction scores and visualized through plots and diagrams." [0234] teaches that "The graphical or textual visual replay of evaluation and tuning results... makes the whole process easier to use." [0413] teaches "providing of visual elements in a visual replay mode which are graphical and/or textual, giving more insight to the operators." FIG. 25 depicts an exemplary graphical interface 2500 operating at a computing device 2501 (e.g., personal computer, mobile, tablet) containing a Model Configuration Editor 2584. [0469] further teaches that operator user systems 2712 run browsers (HTTP clients) allowing them to view pages, forms, applications, and graphical user interfaces on a display screen. 
See Dakkak: {[Section 3.1], [Section 3.3], [Abstract], [Section 3.2]: 
“…Web UI and Command Line Interface − Users interact with MLModelScope through its web UI
or command-line interface by specifying model and hardware constraints. These constraints are
used to query the database for previous evaluations or to schedule new ones. Users can integrate
MLModelScope within their existing tools or pipelines by using its REST or RPC APIs.”
[Listing 3]:

    PNG
    media_image15.png
    284
    990
    media_image15.png
    Greyscale

The examiner interprets where  causing display, via one or more graphical user interfaces, of results of the evaluating is shown in Section 3.2 describing its "Web UI... Users interact with MLModelScope through its web UI or command-line interface." Section A.3.1 describes that "The website... allow users to evaluate and profile models without familiarity with the underlying frameworks." Section 3.3 Step 7 teaches that "Finally, an evaluation summary is presented to the user.". Section A.3.2 discloses that "MLModelScope summarizes and generates plots to aid in comparing the performance across experiment.") wherein the results include at least, for each of the one or more models: the overall values of the first one or more evaluation metrics; and (Szeto: {[0311-0312], [FIG. 10, 14], [0355]}: “More particularly, FIG. 14 depicts an exemplary visual display 1400 of prediction performances over two replay groups, according to one embodiment. In addition to Replay Group 1 as represented by the curve 1450, accumulated scores for queries within Replay group 2 is visualized as curve 1460...”
 The examiner interprets where the results include at least, for each of the one or more models: the overall values of the first one or more evaluation metrics is shown in the displayed results include overall evaluation metric values for each engine variant (model) under evaluation. [0311] discloses that "reports of prediction results may be viewed graphically... FIG. 10 shows the number of prediction successes and failures over a four-day time-span. The piecewise-linear success and failure curves may refer to a particular engine variant, or all variants..." These plots and corresponding tables display the overall accuracy metrics across the entire validation dataset over time. Furthermore, [0335] and FIG. 14 teach a graphical visual display 1400 where checking the "Whole Period" checkbox (element 1472) "turns each of curves 1450 and 1460 into a single data point... under the whole period option, all queries within the time period of the chart will be summarized to generate a single accumulated prediction score". This single data point represents the overall, aggregate value of the evaluation metric for the model variant.
 See Dakkak: { [Section 3.3], [Section A.3.2]: The examiner interprets where the results include at least, for each of the one or more models: the overall values of the first one or more evaluation metrics is shown in the Web UI summarizing overall evaluation metrics and historical performance profiles (e.g., accuracy, latency, memory usage) for the benchmarked models in a centralized evaluation database.) for each of the first plurality of subsets, the subset values of the first one or more evaluation metrics. (Szeto: {[0230], [FIG. 17-18], [0324], [0345}: “… In FIG. 17, visualization 1700 shows how well one engine variant performs over a given one-month period for three different user segments divided by age groups. Curves 1750, 1760, and 1770 correspond to Replay Groups 1, 2, and 3 respectively, as indicated by legend 1730. Queries are divided into below-30, 30-to-60, and above-60 age groups, and queries within each replay group are processed through engine variant e_v_111...” The examiner interprets where for each of the first plurality of subsets, the subset values of the first one or more evaluation metrics is shown in displaying subset values of evaluation metrics for each of a plurality of subsets (Replay Groups). Specifically, [0324] explains that "Replay groups refer to a pre-defined or operator-defined segment of queries [subsets]... query segment filters include engine variant filters, user attribute filters, item attribute filters... capable of selecting a subset... they may be applied during any stage of the predictive engine tuning, evaluation, and replay process.". [0345] and FIG. 17 explicitly disclose an interactive visual display 1700 showing "how well one engine variant performs... for three different user segments divided by age groups. Curves 1750, 1760, and 1770 correspond to Replay Groups 1, 2, and 3 respectively... processed through engine variant e_v_111." This is an explicit, express disclosure of displaying the subset-specific values (the accumulated prediction scores shown by curves 1750, 1760, and 1770) of the evaluation metrics (prediction scores) for each of the plurality of subsets (the three age segments) on the graphical interface. Furthermore, FIG. 20 (table 2080) and [0353] show displaying query records and their subset scores (average accumulated scores, element 2086) for multiple Replay Groups simultaneously.)
	The motivation to combine is similar to the dependent on claim. See dependent on claim.
Regarding Claim 6
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 1, (See claim 1) wherein the one or more models associated with the defined modeling objective comprises a plurality of models, ((Szeto: {See [0220], [FIG. 1B], [0076]: “... As shown here, there is a commit 130 process versioned B43da1 which outputs model 1 at element 155 and model 2 at element 160. However, subsequent commit process 135 db48a3 outputs model 3 at element 165 and model 4 at element 170.”  [0078]: “According to additional embodiments there is support provided for experiments using any existing or new commit processes. For instance, developers and certain advanced users can define multiple data sources and multiple experiments to be run in parallel…” 
[0322] “In yet other embodiments, a PredictionIO or machine learning platform may perform evaluation, tuning, and/or comparison of multiple engines. For example, multiple engines may be implemented by different developers and data scientists for a particular prediction problem such as classification of incoming mail as spam or non-spam, or recommendation of similar items. A PredictionIO or machine learning platform may provide, to externally or internally implemented predictive engines, engine evaluation, engine parameter set tuning, prediction history tracking, and replay services as discussed throughout the current disclosure. For multiple engines targeting the same prediction problem, the PredictionIO or machine learning platform may serve as an interface for cross-comparison and engine selection. For multiple engines targeting different prediction problems based on queries from the same user, PredictionIO or machine learning platform may serve as an interface for cross-examination, selection, and aggregation.”

[FIG. 12]:

    PNG
    media_image16.png
    826
    592
    media_image16.png
    Greyscale

See [0081-0082] for support.
The examiner interprets where the one or more models associated with the defined modeling objective comprises a plurality of models that is shown in a customer project or tenant (modeling objective) being associated with a plurality of models. [0076] and FIG. 1B teach that a Tenant 1 owns a repository containing machine learning code, where a commit 130 B43da1 outputs a plurality of models (model 1 and model 2), and a subsequent commit 135 db48a3 outputs model 3 and model 4. [0078] teaches that "multiple data sources and multiple experiments [engine variants/models] to run in parallel... may be defined". FIG. 12 shows a project (predictive engine 1240) associated with a plurality of models, namely engine variant 1 (1242), engine variant 2 (1244), and engine variant 3 (1246). [0322] further teaches "multiple engines targeting the same prediction problem... implementing multiple predictive engines..." See Dakkak: {See [FIG. 2], [Section A.2]: “... A user can then 7 selects one or more models and 8 selects one or more systems to run the evaluation on. The input can be specified as a URL, data from disk, or dataset 8 and once complete the user can perform the evaluation 9 . This 10 will run the evaluation on the remote system and 11 display the evaluation results along with summary of the execution flow.” 
[FIG. 10]:

    PNG
    media_image17.png
    598
    916
    media_image17.png
    Greyscale

The examiner interprets where wherein the one or more models associated with the defined modeling objective comprises a plurality of models is shown in In Section A.2 (FIG. 10), describing a website workflow where a user selects a task (modeling objective, such as object detection), which displays a list of models available for that task. Step 7 teaches that the user "selects one or more models" (a plurality of models) from the list to be evaluated together.) and wherein the evaluating is performed for each of the plurality of models. (Szeto: {See [FIG. 8, 12], [0229-0230], [0290], [0081-0082] “…Developers may further execute multiple evaluations, experiments, and specify multiple deployments simultaneously, including deployment of multiple concurrent or simultaneous experiments on live or simulated or historical data depending upon the particular needs of the developer. Where necessary, developers are further enabled to control evaluation and deployment schedules including scheduling future (e.g., non-real-time) deployments and specify alerts to be triggered upon any evaluation that fails to meet expectations according to one or more pre-specified conditions or comparisons, such as A/B testing and comparisons for determining which of multiple prediction engine variants are yield the best results according to the developer's specified criteria....”
 [0219] “Additionally supported is A/B testing or comparisons of two or more different trained model variants such that an evaluation or experiment may determine which of the two or more possible trained model variants yield the preferred real-world behaviors according to some specified condition...”
 [0294] “… Evaluator 840 may sort or rank the performances of such multiple engine variants, with pair-wise or multiple-way comparisons, before generating new engine variants for further deployment and evaluation.”
 [0317] “Recall from the discussion with reference to FIG. 8, that multiple engine variants may be tested and studied at the same time, with a split test controller determining to which engine variant any given a user query is dispatched.”
 See [0320] for support.
  The examiner interprets where the evaluating is performed for each of the plurality of models is shown in evaluating each of a plurality of models. [0081] teaches that "developers may further execute multiple evaluations, experiments... simultaneously, including deployment of multiple concurrent or simultaneous experiments on live or simulated or historical data...". [0082] teaches "A/B testing and comparisons for determining which of multiple prediction engine variants yield the best results" . [0219] teaches "A/B testing or comparisons of two or more different trained model variants such that an evaluation or experiment may determine which of the two or more possible trained model variants yield the preferred real-world behaviors...". [0229] teaches "Multiple instances of a predictive engine, or engine variants, may be evaluated at the same time and subsequently compared...". [0230] teaches that "prediction performances of different engines or engine variants may be compared and studied conditionally..." [0290] teaches that split test controller 860 can route queries to multiple deployed variants (e.g., first variant 820 and second variant 822) to generate respective predictions and collect tracking data for evaluation. [0294] teaches "Evaluator 840 may sort or rank the performances of such multiple engine variants... with pairwise or multiple-way comparisons..." . [0320] teaches that forward/splitter 1220 "may direct the same predictive traffic to multiple engine variants, thus enabling direct comparison of prediction results and prediction accuracy across the multiple engine variants." 
See Dakkak: {See [Abstract], [Section 3.2,], [Section A.2], [Section A.3.2]: The examiner interprets where the evaluating is performed for each of the plurality of models is shown in  performing "parallel evaluation... to analyze and study" the models. Section A.2 Step 8-11 teaches that after selecting a plurality of models, the system "will run the evaluation on the remote system and display the evaluation results" for each model, showing comparative performance. Section A.3.2 teaches that after resolving the constraints, "The model evaluation can then be run on one of (or, at the user request, all) the predictors [systems where models are deployed]".)
	The motivation to combine in similar to the dependent on claim. See dependent on claim.
Regarding Claim 7
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 1, wherein: (See claim 1) the one or more user inputs further include: a fifth user input specifying that newly submitted or updated models associated with the defined modeling objective are to be automatically evaluated upon submission of a new model or update of a models; (Szeto: {See [0081-0082], [0363-0364], [0129-0130]: “As stated above, future scheduling of evaluation runs is additionally supported, as are periodic runs as is execution through the underlying continuous integration infrastructure, for instance, simply by instructing the PredictionIO or machine learning platform to execute the following exemplary periodic repeating 5 minute command execution: lambda schedule “H/5 * * * *”. It is also possible to automate the execution of evaluations such that they are checked and evaluations are run only when there is a code change detected, for instance, by executing the following command, lambda schedule “H * * * *” s cm, the PredictionIO or machine learning platform will detect whether or not a code change is present by checking the repository every hour.”
See [0072-0073], [FIG. 1A], [0084] for support. 
The examiner interprets where the one or more user inputs further include: a fifth user input specifying that newly submitted or updated models associated with the defined modeling objective are to be automatically evaluated upon submission of a new model is shown in users inputting configuration commands or schedules (representing a "fifth user input") to specify that evaluations of newly submitted or updated models are to be executed automatically. [0364] teaches that continuous evaluations "may be run directly under the management by PredictionIO... It is also possible to set trigger criteria for running one or more evaluations. For example, the code repository may be checked periodically, and evaluation runs may be performed wherever there is a code change." Furthermore, [0129] and [0130] disclose explicit command-line user inputs to schedule and automate these trigger-based runs. Specifically, [0129] teaches scheduling periodic repeats using cron expressions: lambda schedule "H/5 * * * *".  [0130] teaches automating the execution of evaluations to run only when there is a code change detected: lambda schedule "H * * * *" --cm (where --cm is a flag instructing the platform to check the repository and run evaluations when a code change is present). [0082] further teaches "the automated deployment of new yet to be created models once those models are generated and have passed evaluations as defined by the developer," establishing that the user defines these evaluation and deployment criteria beforehand to run automatically upon new model generation.) and the computer-implemented method further comprises, by the one or more hardware processors executing program instructions: in response to association of a new model with the defined modeling objective and/or updating of a model of the one or more models associated with the defined modeling objective, automatically evaluating, based on the evaluation configuration, the new model and/or the model. (Szeto: {See [0072-0073], [0076], [0081-0082], [0361], [0363-0364], [0129-0130], [FIG. 1B & 22]: The examiner interprets where the computer-implemented method further comprises, by the one or more hardware processors executing program instructions: in response to association of a new model with the defined modeling objective and/or updating of a model of the one or more models, automatically evaluating, based on the evaluation configuration, the new model and/or the model is shown in response to the association of a new model (e.g., a code commit) or an update to a model, the processors of the machine learning platform automatically evaluate the model. FIG. 1B, [0076], and [0077] show a tenant repository where a subsequent Git commit db48a3 outputs newly updated models (Model 3 and Model 4), which are associated with the same tenant project (defined modeling objective). [0364] teaches that "evaluation runs may be performed wherever there is a code change [a model update/submission]," which is executed automatically by the processors leveraging continuous integration (CI) tools (e.g., Jenkins, Travis) integrated into the platform. [0130] confirms that the system automatically "will detect whether or not a code change is present by checking the repository... [and] automate the execution of evaluations". See Dakkak: {See [Section 3.1], [Section 3.2], [Section 3.3]: The examiner interprets where the computer-implemented method further comprises, by the one or more hardware processors executing program instructions: in response to association of a new model with the defined modeling objective and/or updating of a model of the one or more models, automatically evaluating, based on the evaluation configuration, the new model and/or the model is shown in the modular "evaluation configuration" aspect. Section 3.2 teaches that user evaluations specify the model, framework, and hardware requirements, which are executed using the manifest file (evaluation configuration) to launch the evaluation flow. Integrating this manifest-defined execution flow allows the automated CI/CE triggers of Szeto to parse a uniform, standard evaluation configuration and evaluate the newly submitted or updated model (via Docker containers) using the identical evaluation datasets and metric libraries specified by the user.
	The motivation to combine is similar to the dependent on claim. See the dependent on claim.
Regarding Claim 8
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 1, (See claim 1) wherein the computer-implemented method further comprises, by the one or more hardware processors executing program instructions: (See claim 1) causing display, via the one or more graphical user interfaces, of a listing of one or more selectable evaluation libraries, (Szeto: {See [0072-0073], [FIG. 24-25], [0407]: “Certain embodiments provide a library/engine templates gallery so developers can build their own engines or customize templates to their own needs; ready to use right away and also customizable...” 
[0450-0451]: “… or select changeable values from a template or otherwise specify tuning parameters and other configuration information, such as algorithms and algorithm parameters for use in training the machine learning model for the creation of the new predictive engine variant.”
See [0444] & [0446] for support.
The examiner interprets where causing display, via the one or more graphical user interfaces, of a listing of one or more selectable evaluation libraries is shown in causing the display of a listing of selectable configuration templates and libraries via a graphical user interface. [0407] explicitly discloses that the system provides a "library/engine templates gallery" representing a listing of selectable configurations. It teaches that the "parameter generator deploys the variants via, for instance, engine.json which will thus contain a list of parameters that an operator can tune". [0444] and [0446] teach user systems 2406A-C communicate with web-server 2475 to display web-based interfaces and graphical displays on display screens. [0451] and FIG. 25 disclose that Model Configuration Editor 2584 displays a graphical interface 2500 on computing device 2501 where an operator can "select changeable values from a template or otherwise specify tuning parameters and other configuration information, such as algorithms and algorithm parameters". These templates, algorithms, and parameter sets are the structural and functional equivalents of evaluation libraries under the BRI. 
See Dakkak: {See [Section 3.1], [Section A.2], [FIG. 10], [Section A.3.1]: “MLModelScope can be used as an application or as a library. Users interact with MLModelScope application through its website, command line, or its API interface. The website and command line interface allow users to evaluate and profile models without familiarity with the underlying frameworks or profiling tools…”

[FIG. 11]:

    PNG
    media_image18.png
    706
    736
    media_image18.png
    Greyscale


The examiner interprets where causing display, via the one or more graphical user interfaces, of a listing 
of one or more selectable evaluation libraries is shown in causing the display of a listing of selectable
evaluation metrics and libraries. Section 3.1 teaches "Built-in Pre-/Post-Processing Functions... 
MLModelScope allows for common... post-processing operations (e.g. topK, IOU, mAP, etc.) [evaluation 
libraries] to be used within the manifest without writing code". Section A.2 and Figure 10 describe a 
centralized Web UI (website interface)where users are presented with a visual landing page, select 
tasks, and are displayed a "listing of selectable models and software configurations" available for the 
tasks. Section A.3.1 and Figure 11 teach that the website allows users to "evaluate and profile models” 
without familiarity with the underlying frameworks," displaying available models, metrics, and hardware 
configurations) wherein specifying the first evaluation library includes at least: (Szeto & Dakkak: {See 
citations of the previous two limitations: The examiner interprets where specifying the first evaluation 
library includes at least: is shown in specifying a model evaluation configuration or engine tuning setup 
incorporates selecting and defining the algorithms, metrics libraries, or templates to be executed.) 
receiving user input selection of the first evaluation library from the of the one or more selectable evaluation libraries. (Szeto: {See [FIG. 23, 25], [0444], [0451], [0408]: “…allows selection of variants by the user based on a template provided by PredictionIO; supports automatic generation; provides templates which give the engine parameters that the user can tune with the default settings in which the parameter generator deploys the variants via, for instance, engine.json which will thus contain a list of parameters that an operator can tune.”
 The examiner interprets where receiving user input selection of the first evaluation library from the of the one or more selectable evaluation libraries is shown in receiving user input selection of a template or algorithm configuration from the displayed listing. [0408] teaches that the system "allows selection of [engine] variants by the user based on a template provided by PredictionIO" . [0444] teaches that web-server 2475 receives "interactions with displayed graphical user interfaces and displays at the user client devices". [0451] and FIG. 25 disclose that Model Configuration Editor 2584 receives user inputs to "select changeable values from a template" or algorithm parameters on GUI 2500. 
See Dakkak: {See [Section A.2], [FIG. 10]: The examiner interprets where receiving user input selection of the first evaluation library from the of the one or more selectable evaluation libraries is shown in Section A.2 (Step 7), it describes that the user "selects one or more models [configurations/libraries]" from the displayed listing on the Web UI to specify their benchmarking experiment. Step 9 then receives the user input to trigger and execute the evaluation based on the selected configuration.
	The motivation to combine is similar to the dependent on claim. See dependent on claim.
Regarding Claim 9
Szeto in combination with Dakkak teaches The method of claim 1, (See claim 1) wherein the multiple transactions within the training data include any one or more of: simulated transactions; past actual user transactions; user records; product records; behavioral data; user activities data; events data; images; voice data; and text. (Szeto: {See [0408], [0451]: The examiner interprets where determining, based on evaluation library definitions, selectable evaluation libraries is shown in utilizing templates with default values to determine configuration options. See Dakkak: {See [Section 3.2], [Section A.3.2], [Section A.3.4: The examiner interprets where determining, based on evaluation library definitions, selectable evaluation libraries is shown in MLModelScope's orchestration layer "solves the constraint" against these definitions to dynamically determine and return the selectable libraries/predictors that satisfy the constraint (Section A.3.2, A.3.4, Section 3.2).
	The motivation to combine is similar to dependent on claim. See dependent on claim.
Regarding Claim 10
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 9, (See claim 9) wherein the one or more evaluation library definitions each include at least: respective parsable descriptions of the evaluation libraries, (Szeto: {See [0408]}: The examiner interprets where respective parsable descriptions of the evaluation libraries is shown in using structured parsable files like engine.json. See Dakkak: {See [Listing 1 & 2], [Section A.3.4]: The examiner interprets where respective parsable descriptions of the evaluation libraries is shown in all models described using a model specification (manifest) which is a "text file" (parsable description) containing pre-processing, post-processing, and framework requirements.) respective identifications of user definable data fields, (Szeto: {See [0056]}: “A data item can be a data representation or container for information representing a specific thing in the world that have a number of definable properties. For example, a data item can represent an entity such as a physical object, a parcel of land or other real property, a market instrument, a policy or contract, or other noun. Each data item may be associated with a unique identifier that uniquely identifies the data item. The item's attributes (e.g. metadata about the object) may be represented in one or more properties. Attributes may include, for example, a geographic location associated with the item, a value associated with the item, a probability associated with the item, an event associated with the item, and so forth.”
 [0480]: “… Each table generally contains one or more data categories logically arranged as columns or fields in a viewable schema. Each row or record of a table contains an instance of data for each category defined by the fields. For example, a CRM database may include a table that describes a customer with fields for basic contact information such as name, address, phone number, fax number, etc. Another table might describe a purchase order, including fields for information such as customer, product, sale price, date, etc. In some multi-tenant database systems, standard entity tables might be provided for use by all tenants. For CRM database applications, such standard entities might include tables for Account, Contact, Lead, and Opportunity data, each containing pre-defined fields. It is understood that the word “entity” may also be used interchangeably herein with “object” and “table.”” The examiner interprets where respective identifications of user definable data fields is shown in customizable DASE inputs and tables arranged as columns or fields. 
See Dakkak: {See [Listing 1], [Section A.3.4],

[Listing 4]:

    PNG
    media_image19.png
    586
    952
    media_image19.png
    Greyscale

See [Listing 5] for support.
The examiner interprets where respective identifications of user definable data fields is shown in specifying model inputs and outputs in the manifest, including element_type, layer_name, layout, and parameters which represent identifications of data fields and parameters that are user-definable within the text specification.) and respective specifications for determining related evaluation metrics. (Szeto: {See [0257]}: The examiner interprets where respective specifications for determining related evaluation metrics is shown in applying evaluation metrics to compute evaluation scores. See Dakkak: {See [Section 3.1], [Section A3.4], [Listing 1 & 2]: The examiner interprets where respective specifications for determining related evaluation metrics is shown in the evaluation manifest specifies post-processing operations, including built-in metrics calculation steps (such as topK, IOU, mAP) or custom arbitrary Python post-processing functions (Listing 1, Lines 31-34; Section 3.1), which represent specifications for determining related evaluation metrics.)
	The motivation to combine is similar to dependent on claim. See dependent on claim.
Regarding Claim 11
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 10, (See claim 10) wherein the listing of the one or more selectable evaluation libraries includes respective evaluation library descriptions determined based on the respective parsable descriptions of the respective library definitions. (Dakkak: {See [Section 3.2], [Section A.2], [FIG. 10]: The examiner interprets where the listing of the one or more selectable evaluation libraries includes respective evaluation library descriptions determined based on the respective parsable descriptions of the respective library definitions is shown in a Web UI that parses the manifests in the registry and presents a listing of selectable models/libraries to the user, displaying their metadata/descriptions so the user can easily choose and configure an evaluation.
The motivation to combine is similar to dependent on claim. See dependent on claim.
Regarding Claim 12
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 9, (See claim 9) wherein the one or more evaluation library definitions are user definable as computer-readable code. (Szeto: {See [0408], [0451], [0257]}: “Once initialized, evaluations may be performed or added to under the management of the PredictionIO or machine learning platform by executing the add evaluation command abbreviated as “aec”, as follows: lambda aec “python nb.py”. Execution of such a command creates a new evaluation based on the quoted Python command. If the lamda status check is now executed again, the user may verify that one evaluation is now under the management of the PredictionIO or machine learning platform, as follows.” The examiner interprets where the one or more evaluation library definitions are user definable as computer-readable code is shown in user-defined evaluation python scripts  ([0096] lambda aec "python nb.py") and json templates. 
See Dakkak: {See [Section 3.1], [Section A.3.2], [Listing 1]: The examiner interprets where the one or more evaluation library definitions are user definable as computer-readable code is shown in the evaluation manifests (library definitions) are user-definable files where users can specify arbitrary Python code snippets (computer-readable code) inside the pre- and post-processing functions (Listing 1, Lines 23-26, 31-34; Section 3.1). These Python code blocks are executed by the runtime inside Python subsessions / sub-interpreters (Section 3.1, A.3.2).)
The motivation to combine is similar to dependent on claim. See dependent on claim.
Regarding Claim 13
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 8, (See claim 8) wherein the listing of the one or more selectable evaluation libraries includes core evaluation libraries and user-defined evaluation libraries.  (Szeto: {See [0407] & [0257]}: The examiner interprets where the listing of the one or more selectable evaluation libraries includes core evaluation libraries and user-defined evaluation libraries is shown "pre-defined" evaluation metrics with default computation steps [0257], "customizable" metrics designed by developers [0257] and building custom engines in a templates gallery [0407], and  displaying standard templates and user-customized variants in its gallery listing [0407]. See Dakkak: {See [Section 3.1], [Section A.3.2], [Section A.2]: The examiner interprets where the listing of the one or more selectable evaluation libraries includes core evaluation libraries and user-defined evaluation libraries is shown in pre-integrated, preloaded, and predefined evaluation libraries ("core evaluation libraries") that allow common resizing, normalization, topK, IOU, and mAP metrics to be evaluated "without writing code" (Section 3.1, 3.26, A.3.2), custom "user-defined evaluation libraries" created by users that allow specifying "arbitrary Python functions" (computer-readable code snippets) for custom metrics (Section 3.1, Listing 1, Lines 23-26, 31-34), dynamically displaying available pre-integrated (core) model configurations and custom user-defined model manifests on its Web UI listing for user selection (Section A.2, A.3.2).)
The motivation to combine is similar to dependent on claim. See dependent on claim.
Regarding Claim 14
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 8, (See claim 8) wherein the listing of the one or more selectable evaluation libraries is searchable (Szeto: {See [0365]]}: The examiner interprets where the listing of the one or more selectable evaluation libraries is searchable is shown in database storage (SQL/NoSQL HBase) and query interfaces ([0221], [0455]) providing "display, search, rank, rating, and similar management features" for stored data assets ([0365]) to enable storage and retrieval across many projects . It would have been obvious to a POSITA to apply these search and retrieval features of Szeto and MLMODELSCOPE to the interactive listing (templates gallery) of Szeto, rendering the listing searchable. See Dakkak: {See [Section 3.2], [Section A.2], [Section A.3.1],[Section A.3.4]: The examiner interprets where all evaluation libraries/manifests are stored in a distributed key-value registry (Section 3.2, A.3.4). To store these manifests, MLModelScope teaches utilizing a database registry styled after "Hyperdex: A distributed, searchable key-value store" (Section 3.2, Reference Escriva et al., 2012, P.4). This database registry natively enables searchable and multi-attribute querying of manifests and users specifying model, framework, or hardware "constraints" using its Web UI or command line, which are resolved by the system to locate, filter, and retrieve specific evaluation configurations (Section 3.2, A.3.1, A.3.2). Specifically, on the Web UI, displaying of available models and evaluation configurations are dynamically filtered (searched) by task type (Section A.2, "clicking demo displays... task types... selecting object detection filters and displays available models").)
The motivation to combine is similar to dependent on claim. See dependent on claim.
Regarding Claim 15
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 8, (See claim 8) wherein the listing of the one or more selectable evaluation libraries is determined based on permissions associated with the user (Szeto: {See [FIG. 24], 
 [0464] "…Thus, different users will have different capabilities with regard to accessing and modifying application and database information, depending on a user's security or permission level.”
[0467]: "… With a multi-tenant system, data for multiple tenants may be stored in the same physical database object, however, tenant data typically is arranged so that data of one tenant is kept logically separate from that of other tenants so that one tenant does not have access to another tenant's data, unless such data is expressly shared...”
 [0478]: "..Because an MTS might support multiple tenants including possible competitors, the MTS may have security protocols that keep data, applications, and application use separate. Also, because many tenants may opt for access to an MTS rather than maintain their own system, redundancy, up-time, and backup are additional functions that may be implemented in the MTS. In addition to user-specific data and tenant specific data, system 2716 might also maintain system level data usable by multiple tenants or other data. Such system level data might include industry reports, news, postings, and the like that are sharable among tenants.”
See [0437-0438], [0441], [0449] [0471], [FIG. 27A], [FIG. 27B] for support.
The examiner interprets where the listing of the one or more selectable evaluation libraries is determined based on permissions associated with the user is shown in a cloud-hosted, multi-tenant database system (MTS) that hosts various standard and custom applications for different tenant organizations ( [0437], [0467]), and "tenant data typically arranged so that data of one tenant is kept logically separate from that of other tenants so that one tenant does not have access to another tenant's data, unless such data is expressly shared" ( [0467]). This isolation explicitly extends to applications and application use, as Szeto teaches that "because many tenants may opt for access to an MTS... the MTS may have security protocols that keep data, applications, and application use separate" ([0478]). A tenant's custom models and templates (such as the evaluation library manifests of MLModelScope) are stored in logically isolated tenant storage areas ( [0471], [0473]). Therefore, a user belonging to Tenant A is prevented by the system's security protocols from accessing or selecting the templates/libraries of Tenant B.  Furthermore,  it is shown within any organization/tenant, "different users will have different capabilities with regard to accessing and modifying application and database information, depending on a user's security or permission level" ([0464]), verified by an authenticator 2440 ([0449]). When displaying the templates gallery (selectable library listing) of Szeto ( [0407]) or the evaluation configurations listing of MLMODELSCOPE, it would have been obvious to a POSITA to determine and filter the selectable options based on these tenant boundaries and user permission levels taught by Szeto. This ensures that the user is only displayed a listing of the specific evaluation libraries they have permissions to view and select, directly satisfying the limitation.
The motivation to combine is similar to dependent on claim. See dependent on claim.
Regarding Claim 16
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 1, wherein: specifying the first subset definition includes at least: identifying at least a first data field for subsetting; and (Szeto: {See [FIG. 15], [0324], [0350],  [0325]}: "…. For instance, age, gender, sign up date, plan or service a user has signed-up for, range of user ids, dates, and so on. If the system contains users' behavior data, the filter can even go further to select queries that have targeted users who have performed certain actions during a certain time range. For example, one or more filters may be applied to generate a replay group by selecting queries for recommending five products to female users when they are in San Francisco.” 
[0337]: ".... Labels 1572, 1582, and 1592 indicate user attributes that can be set by the operator. Such user attributes may be pre-defined or operator-defined. In addition, the PredictionIO or machine learning platform may assess all available queries to determine if users are present, and if so, which user attributes are present and can be selected for generating replay groups. In this particular example, age, sign-up date, and gender are three available user attributes…” 
The examiner interprets where specifying the first subset definition includes at least: identifying at least a first data field for subsetting is shown in when configuring an evaluation run, the user specifying dynamic segment filters (subset definitions) based on database attributes (data fields). Specifically, [0324] teaches selecting "user attribute filters, item attribute filters, query attribute filters... capable of selecting a subset of available queries...". [0325] teaches selecting user attributes such as "age, gender, sign-up date, plan or service...". [0337] and Figure 15 teach that "age, sign-up date, and gender are three available user attributes" for subsetting. This represents identifying at least a first data field (e.g. user age, user gender) for subsetting.) identifying, for the first data field, at least one of: data buckets, data ranges, or a number of data buckets or ranges; and (Szeto: {See [FIG. 10, 13-18], [0312], [0324-0325], [0337], [0345], [0329-0330]: "In this and subsequent illustrative examples shown in FIGS. 13 to 18, prediction performances are plotted in terms of accumulated prediction scores over time. Here an accumulated prediction score is calculated by an accumulation function that summarizes the prediction scores of all queries of a replay group within defined time intervals over a given time period. For example, each query may have an associated timestamp, representing the time at which the query was received by the predictive engine. According to such timestamps, queries within a replay group may be segmented for computing accumulated prediction scores. In another example, a timestamp may represent when a prediction has been made, or a sign-up date/time at which a user has signed-up for prediction service. Generally, computation of accumulated predicted scores may be carried out over any categorization or segmentation of queries within a replay group...” 
[0332]: "In some other embodiments, Replay Group 1 may be generated by selecting queries containing users who have signed up for prediction service during January, 2015. Generally, the time period 1315 may refer to any time-related query attribute. In other embodiments, prediction scores may be accumulated over different categories such as user gender, leading to accumulated score plots similar to the diagram shown in FIG. 11. ...” 
See [0346], [0308] for support.
The examiner interprets where identifying, for the first data field, at least one of: data buckets, data ranges, or a number of data buckets or ranges is shown in the filter definitions segment the chosen data field into specific categorical divisions (data buckets) or logical intervals (data ranges). Specifically, [0324]–[0325] and [0345]–[0346], and Figures 12, 17, 18 show selecting user age groups (ranges/buckets of "user age < 30", "30 <= user age <= 60", "user age > 60"). [0337] and Figure 15 teach determining a user age group (using a threshold, e.g., value 30) or a date range ("after Jun. 1, 2014, but before Aug. 1, 2014"). Paragraph [0312] teaches binned groupings where the system "take into account the amount of data available to dynamically determine optimal time scale values for binning purposes" (defining a number of data buckets/ranges). [0330] teaches grouping data points by time intervals/ranges (e.g. two-day intervals, or specific timestamp ranges). [0332] teaches grouping data into buckets like user gender.) the computer-implemented method further comprises, by the one or more hardware processors executing program instructions: causing display, via the one or more graphical user interfaces, of one or more visualizations of the first evaluation data set based at least in part on the first data field for subsetting. (Szeto: {See [FIG. 11, 13, 15-18], [0311-0312], [0324-0325], [0330], [0337], [0345-0346], [0313]}: "As previously discussed, data may be grouped by cohort, session, and other types of data characteristics in generating useful statistics for analyzing prediction results. FIG. 11 is a bar chart of prediction successes and failures plotted against different genders. By considering different genders separately, it becomes clear that the current engine or engine variant under consideration is more tailored for male users instead of female users. Consequently, an operator or developer may decide to include gender as an additional variable in the predictive model. In some embodiments, other types of charts such as histograms and scatter plots may be displayed.” [0340-0341]: "In some embodiments, accumulated prediction scores of one or more replay groups within the time period of interest can be displayed on the visual chart through different graphical representations such as line plots, histograms, bar charts, and scatter plots. FIG. 16 depicts an exemplary histogram representing prediction performances over two replay groups in accordance with described embodiments.” The examiner interprets where causing display, via the one or more graphical user interfaces, of one or more visualizations of the first evaluation data set based at least in part on the first data field for subsetting is shown in displaying visual charts (line plots, histograms, bar charts, scatter plots) representing prediction performances (such as accumulated prediction scores or successes/failures) of the evaluation dataset over the specified subsets/Replay Groups. Specifically, [0311]–[0313] and Figure 11 show a bar chart of prediction successes and failures plotted against different genders (categories/buckets of a data field). [0330] and Figure 13 show displaying a piecewise linear curve representing accumulated prediction scores over time intervals (ranges) for Replay Group 1. [0337] teaches plotting window 1505 for visualizing accumulated prediction scores of the dynamically created Replay Group 1. [0340] teaches that accumulated prediction scores of one or more replay groups can be displayed on the visual chart through different graphical representations such as line plots, histograms, bar charts, and scatter plots. [0341] and Figure 16 show an illustrative histogram representing prediction performances over two replay groups (week-long intervals). [0345] and Figure 17 show a visualization comparing prediction performances over a given period for three different user segments divided by age groups (curves 1750, 1760, and 1770 corresponding to below-30, 30-to-60, and above-60 age groups). [0346] and Figure 18 show another visualization comparing how three engine variants perform for the below-30 age group.)
The motivation to combine is similar to dependent on claim. See dependent on claim.
Regarding Claim 17
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 16, (See claim 16) wherein specifying the first subset definition further includes at least: identifying at least a second data field for subsetting; and (Szeto: {See [FIG. 15], [0337]}: The examiner interprets where specifying the first subset definition further includes at least: identifying at least a second data field for subsetting is show when specifying a segment filter (subset definition), the system selecting and combine multiple database attributes (data fields) to segment data. Specifically, Paragraph [0337] and Figure 15 explicitly teach that "Age," "Sign-up Date," and "Gender" are three available user attributes that can be used simultaneously as segment filters to define "Replay Group 1". In Figure 15, the operator specifies a filter for "Age" (first data field) and dynamically adds a filter for "Gender" (second data field) or "Sign-up Date" (third data field) to construct a highly targeted cohort. This represents identifying at least a second data field for subsetting.)  identifying, for the second data field, at least one of: data buckets, data ranges, or a number of data buckets or ranges. (Szeto: {See [FIG. 15], [0337]}: The examiner interprets where identifying, for the second data field, at least one of: data buckets, data ranges, or a number of data buckets or ranges is shown in the filter definitions segment the second data field into specific categorical divisions (data buckets) or logical intervals (data ranges). Specifically, Figure 15 shows that for the second data field "Gender," the operator selects from options "male" and "female" (data buckets). For the other available attribute "Sign-up Date," the operator defines a date range using "After: Jun 1, 2014" and "Before: Aug 1, 2014" inputs (data ranges). This represents identifying, for the second data field, at least one of: data buckets or data ranges.)
The motivation to combine is similar to dependent on claim. See dependent on claim. 
Regarding Claim 18
Szeto in combination with Dakkak and Ravikumar The computer-implemented method of claim 1, wherein at least one of: the second user input includes specifying a second evaluation data set; (Szeto: {See [0365], [0132-134]: “For instance, consider a developer seeking to run the same evaluation command on two different data sources. The developer will first define two data sources via the lamda datasource add command, as follows:
lambda datasource add data1 training1.csv 
lambda datasource add data2 training2.csv”
[0136-0138]: “In the examples used here, the data sources reside within the global scope and are therefore manually associated with the respective project, as follows:
lambda aeds data1
lambda aeds data2”
[0143]: “Git Commit: 8037562633ccdb71fd0e2efb808f9cb0dce6bbea
Eval Data Source: data1”
[0149]: “Eval Command: python nb.py
=> (no result)”
[0153-0156]: “Execution of the lambda eval command will then launch four evaluation runs for parallel execution. All present data sources may be listed at any time via the lambda datasource list command, which will provide output such as follows:
Data Source ID: data1
Value: training1.csv
Data Source ID: data2
Value: training2.csv” 
The examiner interprets where the second user input includes specifying a second evaluation data set is shown in receiving user inputs specifying a second evaluation dataset(a second data source) to enable running multiple evaluations in parallel. Specifically, [0132] teaches: "The developer will first define two data sources via the lambda datasource add command" [715]. [0137]–[0138] show registering data1 and data2 [715]. [0154]–[0156] list data1 (associated with training1.csv) and data2(associated with training2.csv) [716]. [0365] teaches: "In some embodiments, PredictionIO... utilizes data sources to enable the running of multiple evaluations in parallel. For example, x data sources and y evaluations may be defined... running xy evaluation instances in parallel" See Dakkak: {See [Section 3.1], [Section A.3.2]: The examiner interprets where the second user input includes specifying a second evaluation data set is shown in versioning, managing, and running evaluations across multiple datasets (e.g., ILSVRC 2012, COCO, etc.) on demand during evaluation runs.) the third user input includes specifying a second evaluation library, wherein the second evaluation library defines at least a second one or more evaluation metrics; or (Szeto: {See [0293], [0431], [0286-0287]: “An exemplary value of the parameter set 813 is as follows:
 
    PNG
    media_image20.png
    384
    1002
    media_image20.png
    Greyscale
Parameter set 813 states that variant 820 uses DataSource x2, and Algorithms 4 and 2. The values of algorithm parameter1 and algorithm parameter2 of Algorithm 4 are set to b1 and a2 respectively, while the value of the parameter Y of Algorithm 2 is set to 33.”
 [0432]: “In accordance with another embodiment of method 2300, the configuration input received by the machine learning platform includes the specification of the configuration to alter generated prediction engine variants according to developer specification or according to developer selectable templates with changeable default values or according to automatically generated default configuration parameters provided by the machine learning platform.” The examiner interprets where the third user input includes specifying a second evaluation library, wherein the second evaluation library defines at least a second one or more evaluation metrics is shown in the evaluator component evaluating the predictive process using multiple metrics concurrently (e.g., metric 842 and second metric 844) individually or in combination. [0431] teaches receiving configuration inputs specifying multiple hyperparameters or business logic parameters for multiple algorithms (e.g., Algo 4 and Algo 2). See Dakkak: {See [Listing 4, 5], [FIG. 10], [Section 3.1], [Section A.1]: “Listing 4 shows the manifest of SSD_MobileNet_v1_COCO, an objection detection model, for TensorFlow. This model embeds the pre-processing operations in the model graph, and thus requires no normalization, cropping, or resizing. The major difference from a image classification model manifest is the task type (being object_detection) and the outputs. There are three output tensors for this model (boxes, probabilities, and classes). These output tensors are processed by MLModelScope to produce a single object detection feature array, which can then be visualized or used to calculate the metrics (e.g. mean average precision). Listing 5 shows the manifest of Mask_RCNN_ResNet50_v2_Atrous_COCO, an instance segmentation model, for MXNet. The major difference from the object detection model in Listing 4 is the task type (being instance_segmentation) and the outputs. Listing 5 shows four outputs for this model (boxes, probabilities, classes, and masks). These output tensors are processed by MLModelScope to produce a single instance segmentation feature array. Note that unlike TensorFlow, MXNet uses layer indices in place of layer names to get the tensor objects.” The examiner interprets where the third user input includes specifying a second evaluation library, wherein the second evaluation library defines at least a second one or more evaluation metrics is shown in a model manifest file (configuration) that can specify multiple software stack components, output layers, and post-processing functions (representing multiple modular evaluation libraries). For example, Listing 1 and Section A.1 show specifying multiple output metrics/tensors (e.g., boxes, probabilities, and classes) which are processed via built-in post-processing code (evaluation libraries) to compute multiple different metrics (e.g. latency, topK, mAP) concurrently. A POSITA would have found it obvious to specify a second evaluation library (or select multiple library definitions via the GUI) to compute a second set of evaluation metrics, utilizing the modular manifest structure of Dakkak within the platform of Szeto.) the fourth user input includes specifying a second subset definition, wherein the second subset definition identifies a second plurality of subsets of the first evaluation data set and second evaluation data set, if specified. (Szeto: {See [0324-0325], [0337], [0346]: The examiner interprets where the fourth user input includes specifying a second subset definition, wherein the second subset definition identifies a second plurality of subsets of the first evaluation data set and second evaluation data set, if specified is shown in the user specifying multiple segment filters (subset definitions) jointly to identify a second plurality of subsets (Replay Groups/cohorts) across evaluation dataset(s). Specifically, [0324] teaches: "Multiple query segment filters [subset definitions] may be used jointly, and filtered results may be combined as intersections or unions of query segments... each query may be associated with one or more replay group IDs". [0325] teaches combining a first filter (gender female) and a second filter (location San Francisco). [0337] and Figure 15 show an interactive display 1560 where the user specifies Age group (first subset definition), Sign-up Date (second subset definition), and Gender (third subset definition) to segment the evaluation [737]. [0346] teaches obtaining a subset by applying a user attribute filter as well as an engine variant filter. Since Szeto teaches running evaluations on multiple datasets (data1, data2), applying these joint filters segments both the first and second datasets (data1 and data2) into corresponding subsets, identifying a second plurality of subsets of the first and second datasets.
The motivation to combine is similar to dependent on claim. See dependent on claim. 
Regarding Claim 19
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 18, (See claim 18) wherein the evaluating comprises at least: executing each of the one or more models associated with the defined modeling objective on the first evaluation data set and the second evaluation data set, if specified; (Szeto: {See [0153], [0365], [0131-0132]: “Numerous data sources may be utilized with the PredictionIO or machine learning platform providing a highly versatile platform upon which to schedule and run multiple evaluations in parallel with a single command. For instance, by defining x data sources and y evaluations, running the evaluation command will automatically run all x times y evaluations in parallel in a fully automated fashion, with each of the evaluations being a unique combination of a particular data source and a particular command. For instance, consider a developer seeking to run the same evaluation command on two different data sources. The developer will first define two data sources via the lamda datasource add command, as follows:”
 [0205]: “Because the runs and models of different projects are completely isolated, the developer may wish to switch between tenants by executing the following command: lambda switch my-text-cls, which will change context for the tenancy associated with the project in question in the event that a single developer needs to work on multiple different tenant projects.” The examiner interprets where executing each of the one or more models associated with the defined modeling objective on the first evaluation data set and the second evaluation data set, if specified is shown in the developer specifying a first data source (data1) and a second data source (data2), and that executing the evaluation command launches parallel execution of evaluations across both datasets. [0365] teaches: "PredictionIO... utilizes data sources to enable the running of multiple evaluations in parallel. For example, x data sources and y evaluations may be defined, and PredictionIO... may be instructed... to run each of the y evaluations x times, each time with a different data source, for a total of xy evaluation instances in parallel" . This represents executing each of the models (engine variants) on the first and second evaluation datasets, if specified. See Dakkak: {See [Section 3.1], [Section A.3.2]: The examiner interprets where executing each of the one or more models associated with the defined modeling objective on the first evaluation data set and the second evaluation data set, if specified is shown in the predictor responsible for evaluating models under dataset constraints, running the model on the specified datasets (e.g. ImageNet, COCO) on demand.) determining, for each of the one or more models, and based on the first evaluation library and outputs of each of the one or more models, overall values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified; and (Szeto: {See [0257], [0229], [0293], [0431], [0055]: ““Evaluator” or “Evaluation” component refers to a component of a predictive engine for evaluating the performance of the prediction process to compare different algorithms as well as different engine variants.” The examiner interprets where determining, for each of the one or more models, based on the first evaluation library and outputs of each of the one or more models, overall values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified is shown in the evaluator component 450 (or evaluator 840) receiving outputs from the model variants and applies metrics to compute evaluation results (macro-average prediction scores). Specifically, [0293] teaches that "two metrics 842 and 844 are used within evaluator 840, individually or in combination" to compute overall scores based on the models' outputs. [0431] teaches configuring prediction engine level comparisons and metrics. See Dakkak: {See [Section 3.1], [Section A.3.2]: The examiner interprets where determining, for each of the one or more models, based on the first evaluation library and outputs of each of the one or more models, overall values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified is shown in specifying built-in or custom post-processing functions (evaluation libraries) within the manifest to compute overall values for multiple distinct evaluation metrics (such as topK, latency, and mean average precision/mAP) concurrently from the model outputs. A POSITA would find it obvious to determine overall values of both first and second metrics by specifying and invoking Dakkak’s modular evaluation libraries within Szeto’s platform.) determining, for each of the subsets of each of the evaluation data sets, and based on the first evaluation library and outputs of each of the one or more models, subset values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified. (Szeto: {See [0064], [0230], [0324], [0329-0330], [0337], [0350-0353], [0342]: “For example, FIG. 16 shows an illustrative histogram 1600 representing prediction performances over two replay groups, according to one embodiment. The same Replay Groups 1 and 2 from FIG. 14 are shown here as identified by element 1630. Each bar, such as bars 1640 and 1650, corresponds to prediction scores accumulated over one-week intervals during the one-month period of January, 2015.” The examiner interprets where determining, for each of the subsets of each of the evaluation data sets, and based on the first evaluation library and outputs of each of the one or more models, subset values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified is shown in calculating prediction scores (subset values of metrics) for segmented subgroups (subsets of datasets). [0230] teaches: "By segmenting available replay data, prediction performances of different engines or engine variants may be compared and studied conditionally". [0329] teaches that "accumulated prediction score is calculated by an accumulation function that summarizes the prediction scores of all queries of a replay group [subset] within defined time intervals". Replay Groups (subpopulations) are defined using query segment filters. Furthermore, Figure 15 ( [0337]) shows specifying subset filters (Age < 30, Gender, Sign-up Date) [1634], and Figure 20 ( [0353]) shows a graphical table 2082 displaying computed scores for each specific subset (e.g. Replay Group 1 and Replay Group 2). Since Szeto’s teaches evaluating multiple datasets (data1 and data2) applying these joint filters calculates first and second evaluation metrics for each subset across both datasets. See Dakkak: {See [Section 3.1], [Section A.1]: The examiner interprets where determining, for each of the subsets of each of the evaluation data sets, and based on the first evaluation library and outputs of each of the one or more models, subset values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified is shown in performing data slicing to segment datasets and benchmarking model performance (computing latency, topK, or mAP metrics) specifically at the subpopulation/subset level. Combined, the system calculates overall and subset metrics for both datasets based on the evaluation libraries.)
The motivation to combine is similar to dependent on claim. See dependent on claim. 
Regarding Claim 20
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 19, (See claim 19) wherein the computer-implemented method further comprises, by the one or more hardware processors executing program instructions: causing display, via one or more graphical user interfaces, of results of the evaluating, wherein the results include at least, for each of the one or more models: the overall values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified; and (Szeto: {See [0257] & [0177]:

    PNG
    media_image21.png
    396
    892
    media_image21.png
    Greyscale

The examiner interprets where the overall values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified is shown in displaying overall values of metrics (e.g., overall accuracy, macro-average precision and recall, and support) in a tabular format on screen or as overall score curves. [0177] explicitly lists global macro-level metric values (Accuracy, Precision, Recall, F1-Score) displayed on screen for a model variant. See Dakkak: {See [Section A.3.2], [Section 3]: “We propose MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable
distributed system design to perform model evaluation and benchmarking at scale. MLModelScope consists of a model evaluation specification and a distributed runtime.” The examiner interprets where the overall values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified is shown in displaying overall values of metrics (such as overall latency, throughput, and macro-average accuracy/topK/mAP) for each model on the graphical Web user interface. A PHOSITA would find it obvious to display overall values of both first and second metrics by invoking Dakkak’s modular metrics libraries and displaying them on Szeto’s platform GUI.) for each of the subsets of each of the evaluation data sets, the subset values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified. (Szeto: {See [0230] & [0329], [0337], [0345-0346], [0353]: The examiner interprets where for each of the subsets of each of the evaluation data sets, the subset values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified is shown in displaying subset values of evaluation metrics for segmented subgroups (subpopulation subsets). [0345] teaches: "In FIG. 17, visualization 1700 shows how well one engine variant performs over a given one-month period for three different user segments [subsets] divided by age groups. Curves 1750, 1760, and 1770 correspond to Replay Groups 1, 2, and 3 [subsets]". Figure 18 similarly compares curves of three model variants over a specific subset. Furthermore, Figure 20 ( [0353]) shows a graphical table 2082 displaying computed scores for each specific subset (Replay Group 1 and Replay Group 2). Applying these filters over first and second datasets and metrics displays subset values for both metrics across all subsets on the visual interface.  See Dakkak: {See [Section A.3.2], [Section 3.1], [Section A.1]: The examiner interprets where for each of the subsets of each of the evaluation data sets, the subset values of the first one or more evaluation metrics and the second one or more evaluation metrics, if specified is shown in displaying subset-specific values of evaluation metrics (e.g., subset latency or accuracy) for each data slice on the Web user interface.)
The motivation to combine is similar to dependent on claim. See dependent on claim. 
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over US PGPUB No. US20170124487A1 by Szeto & Chan in view of NPL: "A DISTRIBUTED PLATFORM FOR MODELEVALUATION AND BENCHMARKING AT SCALE" by Dakkak et al. in further view of NPL: “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation” by Liu et al.

Regarding Claim 21
Szeto in combination with Dakkak and Ravikumar teaches The computer-implemented method of claim 1, wherein the computer-implemented method further comprises, by the one or more hardware processors executing program instructions: communicating, to a large language model (“LLM”), one or more items of information associated with the defined modeling objective; (Szeto: {See [0005], [0072], 
[0446-0448]: “In one embodiment, requests 2415 are received at, or submitted to, a web-server 2475 within host organization 2410. Host organization 2410 may receive a variety of requests for processing by the host organization 2410 and its database system 2430. Incoming requests 2415 received at web-server 2475 may specify which services from the host organization 2410 are to be provided, such as query requests, search request, status requests, database transactions, graphical user interface requests and interactions, processing requests to retrieve, update, or store data on behalf of one of the customer organizations 2405A-C, code execution requests, and so forth. Web-server 2475 may be responsible for receiving requests 2415 from various customer organizations 2405A-C via network 2425 on behalf of the query interface 2480 and for providing a web-based interface or other graphical displays to an end-user user client device 2406A-C or machine originating such data requests 2415.
…
The host organization 2410 additionally provides a query interface 2480 capable of receiving and executing requested queries against the databases and storage components of the database system 2430 so as to return a result set, response, or other requested data in furtherance of the methodologies described...”
See [0208], [0240-0241 [0280], [0460] for support.
The examiner interprets where communicating, to a large language model (“LLM”), one or more items of information associated with the defined modeling objective is shown in communicating information associated with a modeling objective/project (such as free-form text input paragraphs, query strings, user preferences, or task parameters) to a predictive/language processing model via REST/RPC APIs. [0434] explicitly describes receiving an alphanumerical string representing a paragraph of free-form text input from a user and communicating it to a natural language processing / text classification model to process text for a modeling objective (such as classifying complaints vs suggestions). [0446]–[0448] describe web servers 2475 and query interfaces 2480 communicating data inputs and API requests to models.)  receiving, from the LLM, an output indicative of at least one of: the defined modeling objective, identify additional models that may be related to the defined modeling objective, (Szeto: {See [0283-0295], [FIG. 8], [0408]: The examiner interprets where identify additional models that may be related to the defined modeling objective is shown in [0283]–[0295] and FIG. 8 describing Evaluator 840 and Auto Parameter Tuning Variant Generator 850 receiving project context and generating/recommending new related model variants (V3, V4) for the modeling objective. [0408] explicitly teaches "selection of variants by the user based on a template... supports automatic generation") the first evaluation data set, the first evaluation library, the first subset definitions, (Szeto: {See [0289], [0431-0433]: The examiner interprets where the first evaluation data set, the first evaluation library, the first subset definitions is shown in [0289] and [0431]–[0433] describing receiving model recommendations specifying data sources (datasets) and algorithm/evaluator parameters (metric libraries)  the one or more evaluation metrics, or an evaluation of evaluation metric values; and (Szeto: {See [0241], [0434-0435]: “In accordance with another embodiment of method 2300, the input includes an alphanumerical string representing all or part of a paragraph of free form text input from a user and a classification indicating whether the alphanumerical string corresponds to a customer complaint, a customer suggestion, or a customer compliment.
In accordance with another embodiment of method 2300, the prediction output from the machine learning platform specifies a set of multiple item IDs, a set of recommended item IDs, a set of product IDs indicated as being predictively similar to input product IDs, one or more numerical scores returned responsive to a query and/or non-numerical criteria returned as the prediction output responsive to a query.” The examiner interprets the one or more evaluation metrics, or an evaluation of evaluation metric values is shown in [0241] and [0435] teach receiving output specifying numerical scores or non-numerical criteria evaluating prediction performance.)  parsing the output and updating the one or more graphical user interfaces based on the output. (Szeto: {See [0293-0296], [0460], [0337], [0451-0453], [0250]: “... The returned predicted result 345 may be automatically structured into a programming object easily convertible to other formats by the PredictionIO or machine learning platform 300.” 
See [0412] for support.
The examiner interprets where parsing the output and updating the one or more graphical user interfaces based on the output is shown in parsing the returned output from the models or parameter generators and updating the graphical user interface based on the parsed output. [0250] teaches that returned predicted results are "automatically structured into a programming object easily convertible to other formats" (parsing). [0296] and [0412] teach parsing evaluation and replay outputs to render visual/textual elements on visual replay dashboard 890. [0451]–[0453] teach updating Model Configuration Editor 2584 with recommended or entered configuration parameters. [0460] teaches query interface 2635 rendering output results on user displays. 
See Dakkak: {[Section 3.2], [Section A.2], [Section A.3.1] : The examiner interprets where parsing the output and updating the one or more graphical user interfaces based on the output is shown in parsing returned JSON/manifest outputs from predictors and updating ready-made web interface displays.
	Szeto in combination with Dakkak and Ravikumar fails to explicitly teach communicating, to a large language model (“LLM”), one or more items of information associated with the defined modeling objective. Liu teaches communicating, to a large language model (“LLM”), one or more items of information associated with the defined modeling objective ([Abstract], [Section 1, 2, 5]: The examiner interprets where communicating, to a large language model (“LLM”), one or more items of information associated with the defined modeling objective is shown in Large Language Models ("LLM") such as ChatGPT, GPT-4, CodeGen, Vicuna, SANTACODER, and PolyCoder, and communicating prompt context (problem descriptions, function signatures, docstrings) associated with a modeling objective to an LLM via API queries. Communicating information associated with a modeling objective to an LLM as taught by Liu in Szeto’s API pipeline satisfies.) Liu further teaches receiving, from the LLM, an output indicative of at least one of: the defined modeling objective, identify additional models that may be related to the defined modeling objective, the first evaluation data set, the first evaluation library, the first subset definitions, the one or more evaluation metrics, or an evaluation of evaluation metric values ([Section 2.1, 3, 4], [Table 4]: The examiner interprets where receiving, from the LLM, an output indicative of at least one of: the defined modeling objective, identify additional models that may be related to the defined modeling objective, the first evaluation data set, the first evaluation library, the first subset definitions, the one or more evaluation metrics, or an evaluation of evaluation metric values is shown in receiving outputs from LLMs (ChatGPT, GPT-4, CodeGen) including generated code, test inputs, and evaluation metric values (pass@k)
It would have been obvious to a POSITA before the effective filing date of the invention to substitute/incorporate an LLM service as taught by Liu into Szeto’s platform pipeline. A POSITA would be motivated by simple substitution of known equivalents (KSR Rationale 2), applying a known technique to a known device to yield predictable results (KSR Rationale 1), and design incentives to automate complex model configuration workflows (KSR Rationale 5). Such a combination allows the platform to accept natural language objective descriptions, automatically infer candidate models, datasets, metric libraries, and subset rules via the LLM, parse the output, and update GUI fields, with a high and reasonable expectation of success.
Regarding Claim 22
Szeto in combination with Dakkak and Ravikumar teaches A system comprising: one or more computer-readable storage mediums having program instructions embodied therewith; and one or more processors configured to execute the program instructions to cause the system to perform (Szeto: {See [0436], [0454-0455], [FIG. 26, 28],
 [0482-0486]: “…The exemplary computer system 2800 includes a processor 2802, a main memory 2804 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc., static memory such as flash memory, static random access memory (SRAM), volatile but high-data rate RAM, etc.), and a secondary memory 2818 (e.g., a persistent storage device including hard disk drives and a persistent database and/or a multi-tenant database implementation), which communicate with each other via a bus 2830. Main memory 2804 includes machine learning models 2824 and a database 2823 to store predictive engine variants, transactions, customer data, etc. Additionally depicted here is main memory 2804 which includes training input data 2825 for use with training the machine learning models 2824 in accordance with the embodiments described herein. Main memory 2804 and its sub-elements are operable in conjunction with processing logic 2826 and processor 2802 to perform the methodologies discussed herein.” The examiner interprets where  A system comprising: one or more computer-readable storage mediums having program instructions embodied therewith; and one or more processors configured to execute the program instructions to cause the system to perform is shown in a computer system 2800 comprising a processor 2802 (one or more general-purpose processors, CPUs, microprocessors), main memory 2804 (RAM, ROM), secondary memory 2818 (non-transitory machine-readable storage media), and database 2650/2823. [0436] explicitly claims non-transitory computer-readable storage media storing instructions executed by a processor and memory of a machine learning platform)  the computer-implemented method of claim 1. (See claim 1)
The motivation to combine is similar to dependent on claim. See dependent on claim. 
Regarding Claim 23
Szeto in combination with Dakkak and Ravikumar teaches A computer program product comprising one or more computer-readable storage mediums having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform (Szeto: {See [[0482-0486], [Claim 17], [FIG. 28], [0069]: “Embodiments may be provided as a computer program product, or software, that may include a machine-readable medium having stored thereon instructions, which may be used to program a computer system (or other electronic devices) to perform a process according to the disclosed embodiments. A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium (e.g., read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices, etc.), a machine (e.g., computer) readable transmission medium (electrical, optical, acoustical), etc.”  The examiner interprets where A computer program product comprising one or more computer-readable storage mediums having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform is shown in non-transitory computer-readable storage media (e.g., secondary memory 2818, storage medium 2831, main memory 2804, ROM, RAM, magnetic disk, flash memory) storing software program instructions 2822 executed by processor 2802/2690 to perform machine learning model management and evaluation operations. Claim 17 ([0436]) explicitly claims non-transitory computer-readable storage media having stored instructions that, when executed by a processor, cause performance of platform operations.) the computer-implemented method of claim 1. (See claim 1)
The motivation to combine is similar to dependent on claim. See dependent on claim. 


Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARIC RAYJEE MARKS whose telephone number is (571)467-6372. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ryan Pitaro can be reached at (571) 272-4071. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.





/AARIC R MARKS/Examiner, Art Unit 2188                                                                                                                                                                                                        
/RYAN F PITARO/Supervisory Patent Examiner, Art Unit 2188
Read full office action
FRAMEWORK FOR EVALUATION OF COMPUTER-BASED MODELS

Examiner Intelligence

Office Action

Prosecution Timeline

Strategy Recommendation AI-generated — please review before filing

Prosecution Projections

Ready to respond to this office action?

FRAMEWORK FOR EVALUATION OF COMPUTER-BASED MODELS

Examiner Intelligence

Office Action

Prosecution Timeline

Strategy Recommendation AI-generated — please review before filing

Prosecution Projections

Ready to respond to this office action?

Sign in with your work email
