DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This office action is in response to the amendment filed on 01/28/2026. Claims remain pending in the application. Claims 1 and 5 are independent.
Drawings
Applicant's amendment to specification corrects previous objections; therefore, the previous objections are withdrawn.
Specification
Applicant's amendment to specification corrects previous objections; therefore, the previous objections are withdrawn.
Claim Objections
Applicant's amendment to claims corrects previous objections; therefore, the previous objections are withdrawn.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: "input module", "auto-learning module", and "predictive analysis module" in Claim 6.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
Applicant's amendment to claims corrects some of previous rejections; therefore, some of previous rejections are withdrawn. The remaining rejections are shown below.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6-10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim limitation "input module", "auto-learning module", and "predictive analysis module" in Claim 6 invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. The disclosure is completely silent on "input module"; i.e., the disclosure is devoid of any structure that performs the function for "input module". Also, no association between the structure and the function for "auto-learning module" (recited in ¶¶ [012]-[016], [018]-[019], and [034]) and "predictive analytics module" (recited in ¶¶ [012], [016]-[017], [028], and [033]) can be found in the specification. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claims 7-10 are rejected for fully incorporating the deficiency of their respective base claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-10 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea without significantly more.
The claim(s) recite(s) "identifying a set of relevant training data samples from the input training data" (Claims 1-5), "selecting a suitable type and configuration of a model from a plurality of types and configurations of models for processing the set of relevant training data samples" (Claims 1-10), "validating the set of relevant training data samples" (Claims 1-10), "identifying a plurality of features from the input training data" (Claims 2 and 7), "selecting a set of relevant features from the plurality of features" (Claims 2 and 7), "generating one or more clusters from the selected set of relevant features based on one or more distance-based techniques" (Claims 2 and 7), "ranking the generated one or more clusters" (Claims 2 and 7), "determining a sample selection number from the ranked one or more clusters and creating a sample arrangement" (Claims 2 and 7); "selecting a sample from the created sample arrangement" (Claims 2 and 7), "creating a reduced dataset from the selected sample" (Claims 2 and 7), "monitoring performance of each of the plurality of types and configurations of models with the set of relevant training data samples" (Claims 3-4 and 8-9), "ranking each of the plurality of types and configurations of models in relation to the set of relevant training data samples, based on the performance" (Claims 3-4 and 8-9), "selecting the suitable type and configuration of machine learning model from the plurality of types and configurations of the machine learning models, based on the ranking" (Claims 3-4 and 8-9), "pre-processing the set of relevant training data sample" (Claims 4 and 9), "imputing missing values" (Claims 4 and 9), "removing outlier values" (Claims 4 and 9), "handling categorical variable values" (Claims 4 and 9), "removing duplicate data samples" (Claims 4 and 9), "correcting inconsistent data samples" (Claims 4 and 9), "obtaining, from the suitable type and configuration of model, a prediction and an accuracy probability score associated with the prediction" (Claims 5 and 10), "comparing the accuracy probability score with a threshold score" (Claims 5 and 10), "validating the set of relevant training data samples based on the comparison" (Claims 5 and 10), "determining a set of relevant training data samples from the input training data" (Claims 6-10), and "determining a right set of features for training the machine learning model" (Claim 7) which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or mathematical concepts/algorithms/calculations.
This judicial exception is not integrated into a practical application because the claim(s) recite(s) additional elements/limitations of "optimization device" (Claims 1-5), "system" (Claims 6-10), "one or more computing devices" (Claims 6-10), "input module" (Claims 6-10), "auto-learning module" (Claims 6-10), "predictive analysis module" (Claims 6-10), "executor" (Claims 6-10), "optimizing training of a machine learning model" (Claims 1-5), "receiving/receive input training data comprising a plurality of training data samples" (Claims 1-10), and "machine learning" (Claims 1-10) which only amount to "apply it" with the use of generic computer components or insignificant extra solution activity. None of the additional elements/limitations, taken alone or in combination, integrate the abstract idea into a practical application.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because (a) the additional limitations/elements of "optimizing training of a machine learning model" (Claims 1-5) and "machine learning" (Claims 1-10) are well-understood, routine and conventional (WURC) activity similar to "performing repetitive calculation" (see MPEP 2106.05(d), "Performing repetitive calculations, Flook, 437 U.S. at 594, 198 USPQ2d at 199 (recomputing or readjusting alarm limit values)"); and (b) the additional limitation/element of "receiving/receive input training data comprising a plurality of training data samples" (Claims 1-10) is also well-understood, routine and conventional (WURC) activity similar to "receiving or transmitting data over a network" (see MPEP 2106.05(d), "Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)"). Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5-7, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over HIGGINS et al. (US 2021/0065053 A1, pub. date on 03/04/2021), hereinafter HIGGINS in view of Karumanchi et al. (US 2017/0140297 A1, pub. date on 05/18/2017), hereinafter Karumanchi.
Independent Claim 1
HIGGINS discloses a method of optimizing training of a machine learning model (HIGGINS, ¶ [0038] with FIG. 1B: model selection platform 104 may tune hyper parameters in connection with selection of a machine learning model; model selection platform 104 may evaluate a set of types of machine learning models, and may select a particular type of machine learning model with a greatest quantity of optimal hyper parameters; model selection platform 104 uses hyper parameter optimization to select a machine learning model; ¶¶ [0083] and [0117]: automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models; ¶¶ [0085] and [0119]: tuning the set of parameters of the particular type of machine learning model includes optimizing hyper parameters of the particular type of machine learning model, and retraining the machine learning model based on optimizing the hyper parameters of the machine learning model), the method comprising:
receiving, by an optimization device (HIGGINS, ¶¶ [0050]-[0051], [0062], and [0038] with FIGS. 1A-B, 210 in FIG. 2, and 300 in FIG. 3: data management platform 210 includes a data processing platform 210-1 (e.g., data processing platform 102) and a model selection platform 210-2 (e.g., model selection platform 104) to perform automated data processing and machine learning model generation; device 300 may correspond to data management platform 210, data processing platform 210-1, model selection platform 210-2, computing resource 215, and/or client device 230), input training data comprising a plurality of training data samples (HIGGINS, ¶¶ [0015] with 150 in FIG. 1A: by reference number 150, data processing platform 102 may receive training data from data sources 108; e.g., data processing platform 102 may receive a natural language processing data set, an image processing and/or computer vision data set, and/or the like; ¶ [0029]: data processing platform 102 may use an oversampling technique to generate synthetic examples for the training data set, to avoid under-sampled classes and/or clusters within the training data; ¶ [0071] with 410 in FIG. 4: obtaining first data relating to a machine learning model; ¶ [0089] with 510 in FIG. 5: receiving input data from a data source (block 510); ¶ [0108] with 610 in FIG. 6: obtaining first data relating to a natural language processing or image processing task (block 610));
identifying, by the optimization device, a set of relevant training data samples from the input training data (HIGGINS, ¶¶ [0015]-[0029] with FIG. 1A: data processing platform 102 may provide a user interface identifying characteristics of the training data; data processing platform 102 may obtain contextual information relating to the training data, such as information identifying input fields, output fields, a clustering parameter for clustering-based pre-processing, and/or the like; by reference number 152, data processing platform 102 may curate the training data; e.g., data processing platform 102 may perform a rule-based pre-processing procedure, a clustering-based pre-processing procedure, and/or the like to curate and/or transform the training data; data processing platform 102 may apply one or more training data modification filtering rules to perform rule-based pre-processing; e.g., data processing platform 102 may apply one or more rules regarding a type of data that is to be received as training data, and may perform one or more procedures to enforce the one or more rules, such as performing a space trimming procedure, a case lowering procedure, a stop-words removal procedure, and/or the like; data processing platform 102 may parse the training data to determine if a data entry has a regular expression pattern matching a particular regular expression that is configured for data filtering and may filter the data entry accordingly; data processing platform 102 may generate a set of clusters using the training data to perform a cluster analysis; data processing platform 102 may generate the set of clusters and may identify outlier data based on the set of clusters; data processing platform 102 may automatically transform the training data by removing or altering the outlier data; data processing platform 102 may apply labels to data entries based on the training data and/or may apply labels to clusters of data entries based on generating a set of clusters of data entries; data processing platform 102 may determine a data risk value (e.g., which may be a categorical value, such as low risk, medium risk, high risk, and/or the like) based on the clustering index value; e.g., data processing platform 102 may determine that a data set within the training data with a relatively high data purity may be classified as a low risk data set, whereas a data set with a relatively low data purity may be classified as a high risk data set; data processing platform 102 classifies the data set to enable selection of a low risk portion of the data set for training one or more models, resulting in a higher accuracy of the one or more models; data processing platform 102 may alter a parameter of the training data set based on determining the data purity and data risk; data processing platform 102 may alter labels of data based on the data purity and/or data risk determination; data processing platform 102 may determine whether the dataset has a threshold imbalance, a threshold skew of an attribute, and/or the like; data processing platform 102 may determine a data quality based on the balance metric and for different types of data (e.g., integer data, categorical data, date data, identifier data, text data, and/or the like); data processing platform 102 may provide information identifying the data quality as output, may automatically select a subset of the training data for use in machine learning to ensure a threshold level of data quality in data selected for the machine learning, and/or the like; data processing platform 102 may automatically identify a data type of a data entry in the training data (e.g., a determination of whether an attribute value in a data entry matches a regular expression); data processing platform 102 may identify categorical data, identifier data, textual data, and/or the like; data processing platform 102 may use an association technique to account for multicollinearity of attributes; data processing platform 102 may determine the labels based on a cluster; data processing platform 102 may use an unsupervised machine learning algorithm to enable data cleaning, data set alteration, labeling, and/or the like; data processing platform 102 may automatically adjust missing values, remove skewness of the data (e.g., using logarithmic, normalization, standardization, and/or the like techniques), add labels to the data, filter attributes of the data, remove or alter outliers in the data, and/or the like to improve a data quality of data used for machine learning; data processing platform 102 may use an oversampling technique to generate synthetic examples for the training data set, to avoid under-sampled classes and/or clusters within the training data; ¶¶ [0072]-[0073] with 420-430 in FIG. 4: pre-processing the first data to alter the first data to generate second data (block 420); processing the second data to select a set of features from the second data (block 430); ¶¶ [0081]-[0082]: processing the second data includes identifying a plurality of features of the second data, and performing a feature reduction procedure to identify the set of features from the plurality of features of the second data; analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes classifying the plurality of types of machine learning models based on a type of problem associated with the first data; ¶¶ [0090]-[0091] with 520-530 in FIG. 5: pre-processing and filtering the input data to generate intermediate data based on receiving the input data (block 520); labeling one or more missing labels in the intermediate data to generate output data based on generating the intermediate data (block 530); ¶ [0099]: identifying the plurality of languages, and training, for the machine learning model, a plurality of sub-models for the plurality of languages; ¶¶ [0103]-[0104]: the cluster analysis identifies a plurality of predicted sources of error in the input data; when pre-processing and filtering the input data, are configured to perform at least one of a space-trimming procedure, a case lowering procedure, a stop words removal procedure, a Boolean logic filtering procedure, or a regular expression pattern matching procedure; ¶¶ [0109]-[0110] with 620-630 in FIG. 6: pre-processing and filtering the first data to generate second data (block 620); processing the second data to select a set of features from a plurality of features of the second data (block 630));
selecting, by the optimization device, a suitable type and configuration of a machine learning model from a plurality of types and configurations of machine learning models for processing the set of relevant training data samples (HIGGINS, ¶¶ [0030]-[0039] with FIGS. 1A-1B: by reference number 154, data processing platform 102 may provide curated training data to model selection platform 104; by reference number 156, model selection platform 104 may obtain algorithms for a set of types of machine learning models from model repository 110; e.g., model selection platform 104 may obtain stored algorithms representing classification models (e.g., a random forest classifier model, a k-nearest neighbors model, a decision tree model, a multilayer perceptron model, a stochastic gradient descent classifier model, a logistic regression model, a linear support vector classifier model, a naive Bayes model, a gradient boosting model, and/or the like), regression models ( e.g., a ridge regression model), deep learning classification models (e.g., a convolutional neural network model, an Inception V3 model, and/or the like), and/or the like; by reference number 158, model selection platform 104 may automatically select, train, and deploy a particular machine learning model of the set of possible machine learning models; e.g., model selection platform 104 may classify a task type, select a machine learning model category for the task type, select a machine learning model type from the model category, tune parameters of the machine learning model, and train the machine learning model; model selection platform 104 may determine one or more parameters relating to deploying a particular type of machine learning model; model selection platform 104 may determine that the training data relates to a classification task, a regression task, and/or the like, and may filter the set of types of machine learning models based on the type of task; model selection platform 104 may determine that a task is a natural language processing task, an image processing task, a computer vision task, and/or the like; model selection platform 104 may define one or more use cases for a machine learning model, define one or more inputs or outputs for the machine learning model, and/or the like to enable selection of a machine learning model; based on obtaining information identifying a set of types of machine learning models and the information identifying a task type (e.g., a service management task, an Information Technology (IT) task, an application development task, a service transition task, and/or the like) model selection platform 104 may determine that a machine learning model is deployed for the same task, for a similar task (e.g., based on a similarity score calculated based on training data parameters), and/or the like; in this case, model selection platform 104 may determine to modify the existing machine learning model (e.g., by retraining the machine learning model on new training data); model selection platform 104 may use a local interpretable model agnostic explanation (LIME) model to identify an influence of features on the prediction instances of multiple types of machine learning models; model selection platform 104 may automatically select the features for one or more types of machine learning model; e.g., based on the data curation, model selection platform 104 may identify low risk data entries and associated variables to use as features for training a machine learning model; model selection platform 104 may tune hyper parameters in connection with selection of a machine learning model; model selection platform 104 may evaluate a set of types of machine learning models, and may select a particular type of machine learning model with a greatest quantity of optimal hyper parameters; model selection platform 104 uses hyper parameter optimization to select a machine learning model; when a particular type of machine learning model is selected (e.g., using another technique as described herein), model selection platform 104 may automatically tune hyper parameters of the particular machine learning model to enable the particular type of machine learning model to perform predictions using subsequent prediction data; to identify best estimated hyper parameters of multiple types of machine learning models, model selection platform 104 may use a tree-structured Parzen estimator to identify best estimated hyper parameters within a high dimensional search space of parameters of the multiple types of machine learning models; in this way, model selection platform 104 may enable selection from multiple types of machine learning models with multiple sets of parameters and a high dimensional search space; model selection platform 104 may determine a score ( e.g., based on tuning hyper parameters), and may select a particular type of machine learning model based on the score; model selection platform 104 may determine a timeline of a model's performance at evaluating training data, and may select a particular type of machine learning model based on a score, the timeline, and/or the like; ¶¶ [0074]-[0076] with 440-460 in FIG. 4: analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features (block 440); selecting a particular type of machine learning model, of the plurality of types of machine learning models, for the set of features, based on analyzing the set of features to evaluate the plurality of types of machine learning models (block 450); tuning a set of parameters of the particular type of machine learning model, to train the machine learning model (block 460); ¶¶ [0083]-[0085]: analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models; analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes providing, via a user interface, a visualization of model performance of the plurality of types of machine learning models, and receiving, via the user interface, a selection of the particular type of machine learning model based on providing the visualization of the model performance of the plurality of types of machine learning models; tuning the set of parameters of the particular type of machine learning model includes optimizing hyper parameters of the particular type of machine learning model, and retraining the machine learning model based on optimizing the hyper parameters of the machine learning; ¶¶ [0092]-[0093] with 540-550 in FIG. 5: selecting, for the output data, a machine learning model, of a plurality of types of machine learning models, to apply to the output data (block 540); tuning a set of hyper-parameters for the machine learning model based on selecting the machine learning model (block 550); ¶ [0105]: the plurality of types of machine learning models includes at least one of a random forest classifier model, a k-nearest neighbor model, a decision tree model, a multilayer perceptron model, a stochastic gradient descent classifier model, a logistic regression model, a linear support vector classifier model, a naive Bayes model, a ridge regression model, a convolutional neural network model, or an Inception v3 model; ¶¶ [0111]-[0112] with 640-650 in FIG. 6: applying the set of features to a plurality of types of machine learning models to evaluate the plurality of types of machine learning models with respect to completing the natural language processing or image processing task (block 640); selecting a particular type of machine learning model, from the plurality of types of machine learning models, for the set of features, based on applying the set of features to the plurality of types of machine learning models (block 650); ¶¶ [0117]-[0119]: automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models; providing, via a user interface, a visualization of model performance of the plurality of types of machine learning models at completing the natural language processing or image processing task; and receiving, via the user interface, a selection of the particular type of machine learning model based on providing the visualization of the model performance of the plurality of types of machine learning models; optimizing hyper parameters of the particular type of machine learning model; and retraining the particular type of machine learning model based on optimizing the hyper parameters of the particular type of machine learning model); and
obtaining a prediction for (HIGGINS, ¶¶ [0036] and [0039]: provide information identifying results from test data applied to a set of types of machine learning models to provide information, to a user, regarding how each type of machine learning model may respond to a scenario; determine a test accuracy score (e.g., an Fl score), a positive predictive value score (e.g., a precision score), a sensitivity score (e.g., a recall score), and/or the like; provide a user interface to visualize results of testing a set of different types of machine learning models (e.g., a set of hyper parameters, a timeline of model performance, a score, and/or the like); ¶¶ [0077]-[0079] with 470- in FIG. 4: providing access to the particular type of machine learning model via an interface (block 470); receiving, as input via the interface, third data for prediction using the particular type of machine learning model (block 480); providing, as output via the interface, a prediction using the particular type of machine learning model based on receiving the third data (block 490); ¶¶ [0094]-[0097] with 560-590 in FIG. 5: establishing a model pipeline for the machine learning model based on tuning the set of hyper-parameters (block 560); receiving prediction data based on establishing the model pipeline (block 570); performing a prediction using the machine learning model and using the prediction data (block 580); providing the prediction for display via a user interface based on performing the prediction (block 590); ¶¶ [0113]-[0115] with 660-680 in FIG. 6: providing access to the particular type of machine learning model via an interface (block 660); receiving, as input via the interface, third data for prediction using the particular type of machine learning model (block 670); providing, as output via the interface, a prediction, using the particular type of machine learning model, based on receiving the third data, wherein the prediction is a text-completion prediction or an image recognition prediction (block 680)).
HIGGINS fails to explicitly disclose validating the set of relevant training data samples.
Karumanchi teaches a system and a method relating to machine learning (Karumanchi, ABSTRACT), wherein validating the set of relevant training data samples (Karumanchi, ¶¶ [0047]-[0049] with FIG. 3: ¶¶ [0047]-[0049] with FIG. 3: continues to progressively sample the clusters until a prediction accuracy threshold is met by training a prediction model using the sampled data or until a sampling memory usage threshold has been met; the prediction accuracy threshold may be replaced with a comparison of progressive accuracies; when the accuracy converges or does not improve, the progressive sampling stops at that point for output of the sampled data and the trained ML model from a previous progressive sampling iteration; obtain a classification accuracy for the trained ML model on a held-out test data set or using k-fold cross validation on the obtained training data set, and compares the classification accuracy with an accuracy from a previous sampling of the data; upon a determination that the classification accuracy improves over the accuracy from the previous sampling of the data, perform incremental sampling to a second sampling percentage (e.g., higher than the first percentage); upon a determination that the classification accuracy converges or does not improve over the accuracy from the previous sampling of the data, or a total sampling size is larger than a predetermined sampling size threshold, output the sampled data and the trained ML model from a previous progressive sampling iteration; combine the sampled representative data to generate training data for processing predictive analytics; predict data relevance, such as business data relevance (e.g., classified/unclassified, secure/unsecure, sensitive/non-sensitive, etc.) by learning from the sampled training data to predict different categories (e.g., classified, unclassified; private, public, etc.); ¶¶ [0070]-[0071] with FIG. 8: the progressive incremental sampling processing 810 starts from a relatively small sampling percentage; the previous clustering-based sampling is applied to obtain a training data set 750, which is combined with existing training data; content processing 820 processes the actual content of a data file and applies keywords for pattern matching to determine, e.g., confidentiality; the output of the content processing (confidential 825 and non-confidential 830) is input to train a machine learning model(s) 620 and obtain its classification accuracy 840 on a held-out test dataset or using k-fold cross validation on all the obtained training data; the ML model(s) output is input to the business relevance classifier 630; the resulting accuracy is compared with the accuracy from the previous running (set the previous accuracy to O for the first running); if the accuracy improves, then incremental sampling Δxi is performed; if the accuracy converges or does not improve, or the total sampling size is larger than a predetermined threshold (beyond the memory usage and/or processing latency), the progressive incremental sampling processing 810 stops and outputs the sampled data and the trained machine learning model at the previous step; ¶ [0072] with FIG. 9: in block 920, process 900 perform clustering on features (S1 440 x S2 450 x S4 470, FIG. 4) extracted from the data; in block 930 sampling Δxi is performed from each cluster; in block 940 the actual content S3 460 of the sampled data is processed to obtain the respective classification labels; in block 950 the current sampled data is combined with existing sampled data to form a new set of training data; in block 960 the ML model(s) is trained using the set of training data; in block 970 classification accuracy is determined using k-fold cross validation or on a held out test dataset; in block 980 it is determined whether the accuracy improves or converges or does not improve; if the accuracy improves, process 900 proceeds to block 930, otherwise process 900 proceeds to block 990; in block 990, process 900 outputs the sampled data and the trained ML model at for the previous processing iteration; ¶ [0075]: perform progressive sampling of the multiple clusters by sampling the multiple clusters with a first sampling percentage, applying a previous clustering-based sampling to obtain a training data set, combining the training data set with previous determined training data, training an ML model and obtaining a classification accuracy for the ML model on a held-out test data set or using k-fold cross validation on the obtained training data set, and comparing the classification accuracy with an accuracy from a previous sampling of the data).
HIGGINS and Karumanchi are analogous art because they are from the same field of endeavor, a system and a method relating to machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Karumanchi to HIGGINS. Motivation for doing so would ensure that samples data appropriate for ML training are efficiently obtained to improve accuracy of ML (Karumanchi, [0002], [0043]-[0044], [0047]-[0048], and [0062]).
Claim 2
HIGGINS in view of Karumanchi discloses all the elements as stated in Claim 1 and further discloses wherein identifying the set of relevant training data samples from the input training data comprises at least one of identifying a plurality of features from the input training data; selecting a set of relevant features from the plurality of features; generating one or more clusters from the selected set of relevant features based on one or more distance-based techniques; ranking the generated one or more clusters; determining a sample selection number from the ranked one or more clusters and creating a sample arrangement; selecting a sample from the created sample arrangement; and creating a reduced dataset from the selected sample (HIGGINS, ¶¶ [0015]-[0029] with FIG. 1A: data processing platform 102 may provide a user interface identifying characteristics of the training data; data processing platform 102 may obtain contextual information relating to the training data, such as information identifying input fields, output fields, a clustering parameter for clustering-based pre-processing, and/or the like; by reference number 152, data processing platform 102 may curate the training data; e.g., data processing platform 102 may perform a rule-based pre-processing procedure, a clustering-based pre-processing procedure, and/or the like to curate and/or transform the training data; data processing platform 102 may apply one or more training data modification filtering rules to perform rule-based pre-processing; e.g., data processing platform 102 may apply one or more rules regarding a type of data that is to be received as training data, and may perform one or more procedures to enforce the one or more rules, such as performing a space trimming procedure, a case lowering procedure, a stop-words removal procedure, and/or the like; data processing platform 102 may parse the training data to determine if a data entry has a regular expression pattern matching a particular regular expression that is configured for data filtering and may filter the data entry accordingly; data processing platform 102 may generate a set of clusters using the training data to perform a cluster analysis; data processing platform 102 may generate the set of clusters and may identify outlier data based on the set of clusters; data processing platform 102 may automatically transform the training data by removing or altering the outlier data; data processing platform 102 may apply labels to data entries based on the training data and/or may apply labels to clusters of data entries based on generating a set of clusters of data entries; data processing platform 102 may determine a data risk value (e.g., which may be a categorical value, such as low risk, medium risk, high risk, and/or the like) based on the clustering index value; e.g., data processing platform 102 may determine that a data set within the training data with a relatively high data purity may be classified as a low risk data set, whereas a data set with a relatively low data purity may be classified as a high risk data set; data processing platform 102 classifies the data set to enable selection of a low risk portion of the data set for training one or more models, resulting in a higher accuracy of the one or more models; data processing platform 102 may alter a parameter of the training data set based on determining the data purity and data risk; data processing platform 102 may alter labels of data based on the data purity and/or data risk determination; data processing platform 102 may determine whether the dataset has a threshold imbalance, a threshold skew of an attribute, and/or the like; data processing platform 102 may determine a data quality based on the balance metric and for different types of data (e.g., integer data, categorical data, date data, identifier data, text data, and/or the like); data processing platform 102 may provide information identifying the data quality as output, may automatically select a subset of the training data for use in machine learning to ensure a threshold level of data quality in data selected for the machine learning, and/or the like; data processing platform 102 may automatically identify a data type of a data entry in the training data (e.g., a determination of whether an attribute value in a data entry matches a regular expression); data processing platform 102 may identify categorical data, identifier data, textual data, and/or the like; data processing platform 102 may use an association technique to account for multicollinearity of attributes; data processing platform 102 may label one or more data entries that are missing labels; data processing platform 102 may determine the labels based on a cluster; data processing platform 102 may use an unsupervised machine learning algorithm to enable data cleaning, data set alteration, labeling, and/or the like; data processing platform 102 may automatically adjust missing values, remove skewness of the data (e.g., using logarithmic, normalization, standardization, and/or the like techniques), add labels to the data, filter attributes of the data, remove or alter outliers in the data, and/or the like to improve a data quality of data used for machine learning; data processing platform 102 may use an oversampling technique to generate synthetic examples for the training data set, to avoid under-sampled classes and/or clusters within the training data; ¶¶ [0072]-[0073] with 420-430 in FIG. 4: pre-processing the first data to alter the first data to generate second data (block 420); processing the second data to select a set of features from the second data (block 430); ¶¶ [0081]-[0082]: processing the second data includes identifying a plurality of features of the second data, and performing a feature reduction procedure to identify the set of features from the plurality of features of the second data; analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes classifying the plurality of types of machine learning models based on a type of problem associated with the first data; ¶¶ [0090]-[0091] with 520-530 in FIG. 5: pre-processing and filtering the input data to generate intermediate data based on receiving the input data (block 520); labeling one or more missing labels in the intermediate data to generate output data based on generating the intermediate data (block 530); ¶ [0099]: identifying the plurality of languages, and training, for the machine learning model, a plurality of sub-models for the plurality of languages; ¶¶ [0103]-[0104]: the cluster analysis identifies a plurality of predicted sources of error in the input data; when pre-processing and filtering the input data, are configured to perform at least one of a space-trimming procedure, a case lowering procedure, a stop words removal procedure, a Boolean logic filtering procedure, or a regular expression pattern matching procedure; ¶¶ [0109]-[0110] with 620-630 in FIG. 6: pre-processing and filtering the first data to generate second data (block 620); processing the second data to select a set of features from a plurality of features of the second data (block 630)) (Karumanchi, ABSTRACT and ¶¶ [0003], [0042], [0044], and [0046] with 310 in FIG. 3: the clustering processor 310 clusters the data into multiple clusters based on similarities of the obtained data across an entire storage stack; employ a clustering based sampling component to group data points into clusters across three layers (storage infrastructure metrics, file metadata, and applications); ¶ [0068] with FIGS. 4 and 6: the training data with features 610 are obtained across the feature space S1 440 x S2 450 x S4 470; the training data structure 640 that is generated by the processing system includes the data identifier, the features S1 440 x S2 450 x S4 470 and the class labels assigned by S3 460; the clustering based sampling component provides for clustering all the data points across the feature space S1 440 x S2 450 x S4 470 using data 730 and the feature space of the training data structure 640, obtains the cluster centroids 720 and computes the percentage of data assigned to each cluster shown as 750; the clustering may use Vertex Substitution Heuristic (VSH) processing, which is a distance-based clustering algorithm; K-means or other clustering algorithms which clusters data points in a vector space can also be used since the data points 410 are also described in feature vectors; random sampling is performed from each cluster proportionally with respect to the previously obtained percentage; ¶ [0072] with 920 in FIG. 9: in block 920, process 900 perform clustering on features (S1 440 x S2 450 x S4 470, FIG. 4) extracted from the data; in block 930 sampling Δxi is performed from each cluster).
Claim 5
HIGGINS in view of Karumanchi discloses all the elements as stated in Claim 1 and further discloses wherein validating the set of relevant training data samples comprises: obtaining, from the suitable type and configuration of machine learning model, a prediction and an accuracy probability score associated with the prediction; comparing the accuracy probability score with a threshold score; and validating the set of relevant training data samples based on the comparison (HIGGINS, ¶¶ [0036] and [0039]: provide information identifying results from test data applied to a set of types of machine learning models to provide information, to a user, regarding how each type of machine learning model may respond to a scenario; determine a test accuracy score (e.g., an Fl score), a positive predictive value score (e.g., a precision score), a sensitivity score (e.g., a recall score), and/or the like; provide a user interface to visualize results of testing a set of different types of machine learning models (e.g., a set of hyper parameters, a timeline of model performance, a score, and/or the like); ¶¶ [0077]-[0079] with 470- in FIG. 4: providing access to the particular type of machine learning model via an interface (block 470); receiving, as input via the interface, third data for prediction using the particular type of machine learning model (block 480); providing, as output via the interface, a prediction using the particular type of machine learning model based on receiving the third data (block 490); ¶¶ [0094]-[0097] with 560-590 in FIG. 5: establishing a model pipeline for the machine learning model based on tuning the set of hyper-parameters (block 560); receiving prediction data based on establishing the model pipeline (block 570); performing a prediction using the machine learning model and using the prediction data (block 580); providing the prediction for display via a user interface based on performing the prediction (block 590); ¶¶ [0113]-[0115] with 660-680 in FIG. 6: providing access to the particular type of machine learning model via an interface (block 660); receiving, as input via the interface, third data for prediction using the particular type of machine learning model (block 670); providing, as output via the interface, a prediction, using the particular type of machine learning model, based on receiving the third data, wherein the prediction is a text-completion prediction or an image recognition prediction (block 680)) (Karumanchi, ¶¶ [0047]-[0049] with FIG. 3: continues to progressively sample the clusters until a prediction accuracy threshold is met by training a prediction model using the sampled data or until a sampling memory usage threshold has been met; the prediction accuracy threshold may be replaced with a comparison of progressive accuracies; when the accuracy converges or does not improve, the progressive sampling stops at that point for output of the sampled data and the trained ML model from a previous progressive sampling iteration; obtain a classification accuracy for the trained ML model on a held-out test data set or using k-fold cross validation on the obtained training data set, and compares the classification accuracy with an accuracy from a previous sampling of the data; upon a determination that the classification accuracy improves over the accuracy from the previous sampling of the data, perform incremental sampling to a second sampling percentage (e.g., higher than the first percentage); upon a determination that the classification accuracy converges or does not improve over the accuracy from the previous sampling of the data, or a total sampling size is larger than a predetermined sampling size threshold, output the sampled data and the trained ML model from a previous progressive sampling iteration; combine the sampled representative data to generate training data for processing predictive analytics; predict data relevance, such as business data relevance (e.g., classified/unclassified, secure/unsecure, sensitive/non-sensitive, etc.) by learning from the sampled training data to predict different categories (e.g., classified, unclassified; private, public, etc.); ¶¶ [0070]-[0071] with FIG. 8: the progressive incremental sampling processing 810 starts from a relatively small sampling percentage; the previous clustering-based sampling is applied to obtain a training data set 750, which is combined with existing training data; content processing 820 processes the actual content of a data file and applies keywords for pattern matching to determine, e.g., confidentiality; the output of the content processing (confidential 825 and non-confidential 830) is input to train a machine learning model(s) 620 and obtain its classification accuracy 840 on a held-out test dataset or using k-fold cross validation on all the obtained training data; the ML model(s) output is input to the business relevance classifier 630; the resulting accuracy is compared with the accuracy from the previous running (set the previous accuracy to O for the first running); if the accuracy improves, then incremental sampling Δxi is performed; if the accuracy converges or does not improve, or the total sampling size is larger than a predetermined threshold (beyond the memory usage and/or processing latency), the progressive incremental sampling processing 810 stops and outputs the sampled data and the trained machine learning model at the previous step; ¶ [0072] with FIG. 9: in block 920, process 900 perform clustering on features (S1 440 x S2 450 x S4 470, FIG. 4) extracted from the data; in block 930 sampling Δxi is performed from each cluster; in block 940 the actual content S3 460 of the sampled data is processed to obtain the respective classification labels; in block 950 the current sampled data is combined with existing sampled data to form a new set of training data; in block 960 the ML model(s) is trained using the set of training data; in block 970 classification accuracy is determined using k-fold cross validation or on a held out test dataset; in block 980 it is determined whether the accuracy improves or converges or does not improve; if the accuracy improves, process 900 proceeds to block 930, otherwise process 900 proceeds to block 990; in block 990, process 900 outputs the sampled data and the trained ML model at for the previous processing iteration; ¶ [0075]: perform progressive sampling of the multiple clusters by sampling the multiple clusters with a first sampling percentage, applying a previous clustering-based sampling to obtain a training data set, combining the training data set with previous determined training data, training an ML model and obtaining a classification accuracy for the ML model on a held-out test data set or using k-fold cross validation on the obtained training data set, and comparing the classification accuracy with an accuracy from a previous sampling of the data).
Independent Claim 6
HIGGINS discloses a system, comprising: one or more computing devices (HIGGINS, ¶¶ [0050]-[0051], [0062], and [0038] with FIGS. 1A-B, 210 in FIG. 2, and 300 in FIG. 3: data management platform 210 includes a data processing platform 210-1 (e.g., data processing platform 102) and a model selection platform 210-2 (e.g., model selection platform 104) to perform automated data processing and machine learning model generation; device 300 may correspond to data management platform 210, data processing platform 210-1, model selection platform 210-2, computing resource 215, and/or client device 230) configured to:
receive, by an input module, input training data comprising a plurality of training data samples (HIGGINS, ¶¶ [0015] with 150 in FIG. 1A: by reference number 150, data processing platform 102 may receive training data from data sources 108; e.g., data processing platform 102 may receive a natural language processing data set, an image processing and/or computer vision data set, and/or the like; ¶ [0029]: data processing platform 102 may use an oversampling technique to generate synthetic examples for the training data set, to avoid under-sampled classes and/or clusters within the training data; ¶ [0071] with 410 in FIG. 4: obtaining first data relating to a machine learning model; ¶ [0089] with 510 in FIG. 5: receiving input data from a data source (block 510); ¶ [0108] with 610 in FIG. 6: obtaining first data relating to a natural language processing or image processing task (block 610));
determining, by the auto-learning module, a set of relevant training data samples from the input training data (HIGGINS, ¶¶ [0015]-[0029] with FIG. 1A: data processing platform 102 may provide a user interface identifying characteristics of the training data; data processing platform 102 may obtain contextual information relating to the training data, such as information identifying input fields, output fields, a clustering parameter for clustering-based pre-processing, and/or the like; by reference number 152, data processing platform 102 may curate the training data; e.g., data processing platform 102 may perform a rule-based pre-processing procedure, a clustering-based pre-processing procedure, and/or the like to curate and/or transform the training data; data processing platform 102 may apply one or more training data modification filtering rules to perform rule-based pre-processing; e.g., data processing platform 102 may apply one or more rules regarding a type of data that is to be received as training data, and may perform one or more procedures to enforce the one or more rules, such as performing a space trimming procedure, a case lowering procedure, a stop-words removal procedure, and/or the like; data processing platform 102 may parse the training data to determine if a data entry has a regular expression pattern matching a particular regular expression that is configured for data filtering and may filter the data entry accordingly; data processing platform 102 may generate a set of clusters using the training data to perform a cluster analysis; data processing platform 102 may generate the set of clusters and may identify outlier data based on the set of clusters; data processing platform 102 may automatically transform the training data by removing or altering the outlier data; data processing platform 102 may apply labels to data entries based on the training data and/or may apply labels to clusters of data entries based on generating a set of clusters of data entries; data processing platform 102 may determine a data risk value (e.g., which may be a categorical value, such as low risk, medium risk, high risk, and/or the like) based on the clustering index value; e.g., data processing platform 102 may determine that a data set within the training data with a relatively high data purity may be classified as a low risk data set, whereas a data set with a relatively low data purity may be classified as a high risk data set; data processing platform 102 classifies the data set to enable selection of a low risk portion of the data set for training one or more models, resulting in a higher accuracy of the one or more models; data processing platform 102 may alter a parameter of the training data set based on determining the data purity and data risk; data processing platform 102 may alter labels of data based on the data purity and/or data risk determination; data processing platform 102 may determine whether the dataset has a threshold imbalance, a threshold skew of an attribute, and/or the like; data processing platform 102 may determine a data quality based on the balance metric and for different types of data (e.g., integer data, categorical data, date data, identifier data, text data, and/or the like); data processing platform 102 may provide information identifying the data quality as output, may automatically select a subset of the training data for use in machine learning to ensure a threshold level of data quality in data selected for the machine learning, and/or the like; data processing platform 102 may automatically identify a data type of a data entry in the training data (e.g., a determination of whether an attribute value in a data entry matches a regular expression); data processing platform 102 may identify categorical data, identifier data, textual data, and/or the like; data processing platform 102 may use an association technique to account for multicollinearity of attributes; data processing platform 102 may determine the labels based on a cluster; data processing platform 102 may use an unsupervised machine learning algorithm to enable data cleaning, data set alteration, labeling, and/or the like; data processing platform 102 may automatically adjust missing values, remove skewness of the data (e.g., using logarithmic, normalization, standardization, and/or the like techniques), add labels to the data, filter attributes of the data, remove or alter outliers in the data, and/or the like to improve a data quality of data used for machine learning; data processing platform 102 may use an oversampling technique to generate synthetic examples for the training data set, to avoid under-sampled classes and/or clusters within the training data; ¶¶ [0072]-[0073] with 420-430 in FIG. 4: pre-processing the first data to alter the first data to generate second data (block 420); processing the second data to select a set of features from the second data (block 430); ¶¶ [0081]-[0082]: processing the second data includes identifying a plurality of features of the second data, and performing a feature reduction procedure to identify the set of features from the plurality of features of the second data; analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes classifying the plurality of types of machine learning models based on a type of problem associated with the first data; ¶¶ [0090]-[0091] with 520-530 in FIG. 5: pre-processing and filtering the input data to generate intermediate data based on receiving the input data (block 520); labeling one or more missing labels in the intermediate data to generate output data based on generating the intermediate data (block 530); ¶ [0099]: identifying the plurality of languages, and training, for the machine learning model, a plurality of sub-models for the plurality of languages; ¶¶ [0103]-[0104]: the cluster analysis identifies a plurality of predicted sources of error in the input data; when pre-processing and filtering the input data, are configured to perform at least one of a space-trimming procedure, a case lowering procedure, a stop words removal procedure, a Boolean logic filtering procedure, or a regular expression pattern matching procedure; ¶¶ [0109]-[0110] with 620-630 in FIG. 6: pre-processing and filtering the first data to generate second data (block 620); processing the second data to select a set of features from a plurality of features of the second data (block 630)),
selecting, by the predictive analysis module, a suitable type and configuration of a machine learning model from a plurality of types and configurations of machine learning models for processing the set of relevant training data samples (HIGGINS, ¶¶ [0030]-[0039] with FIGS. 1A-1B: by reference number 154, data processing platform 102 may provide curated training data to model selection platform 104; by reference number 156, model selection platform 104 may obtain algorithms for a set of types of machine learning models from model repository 110; e.g., model selection platform 104 may obtain stored algorithms representing classification models (e.g., a random forest classifier model, a k-nearest neighbors model, a decision tree model, a multilayer perceptron model, a stochastic gradient descent classifier model, a logistic regression model, a linear support vector classifier model, a naive Bayes model, a gradient boosting model, and/or the like), regression models ( e.g., a ridge regression model), deep learning classification models (e.g., a convolutional neural network model, an Inception V3 model, and/or the like), and/or the like; by reference number 158, model selection platform 104 may automatically select, train, and deploy a particular machine learning model of the set of possible machine learning models; e.g., model selection platform 104 may classify a task type, select a machine learning model category for the task type, select a machine learning model type from the model category, tune parameters of the machine learning model, and train the machine learning model; model selection platform 104 may determine one or more parameters relating to deploying a particular type of machine learning model; model selection platform 104 may determine that the training data relates to a classification task, a regression task, and/or the like, and may filter the set of types of machine learning models based on the type of task; model selection platform 104 may determine that a task is a natural language processing task, an image processing task, a computer vision task, and/or the like; model selection platform 104 may define one or more use cases for a machine learning model, define one or more inputs or outputs for the machine learning model, and/or the like to enable selection of a machine learning model; based on obtaining information identifying a set of types of machine learning models and the information identifying a task type (e.g., a service management task, an Information Technology (IT) task, an application development task, a service transition task, and/or the like) model selection platform 104 may determine that a machine learning model is deployed for the same task, for a similar task (e.g., based on a similarity score calculated based on training data parameters), and/or the like; in this case, model selection platform 104 may determine to modify the existing machine learning model (e.g., by retraining the machine learning model on new training data); model selection platform 104 may use a local interpretable model agnostic explanation (LIME) model to identify an influence of features on the prediction instances of multiple types of machine learning models; model selection platform 104 may automatically select the features for one or more types of machine learning model; e.g., based on the data curation, model selection platform 104 may identify low risk data entries and associated variables to use as features for training a machine learning model; model selection platform 104 may tune hyper parameters in connection with selection of a machine learning model; model selection platform 104 may evaluate a set of types of machine learning models, and may select a particular type of machine learning model with a greatest quantity of optimal hyper parameters; model selection platform 104 uses hyper parameter optimization to select a machine learning model; when a particular type of machine learning model is selected (e.g., using another technique as described herein), model selection platform 104 may automatically tune hyper parameters of the particular machine learning model to enable the particular type of machine learning model to perform predictions using subsequent prediction data; to identify best estimated hyper parameters of multiple types of machine learning models, model selection platform 104 may use a tree-structured Parzen estimator to identify best estimated hyper parameters within a high dimensional search space of parameters of the multiple types of machine learning models; in this way, model selection platform 104 may enable selection from multiple types of machine learning models with multiple sets of parameters and a high dimensional search space; model selection platform 104 may determine a score ( e.g., based on tuning hyper parameters), and may select a particular type of machine learning model based on the score; model selection platform 104 may determine a timeline of a model's performance at evaluating training data, and may select a particular type of machine learning model based on a score, the timeline, and/or the like; ¶¶ [0074]-[0076] with 440-460 in FIG. 4: analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features (block 440); selecting a particular type of machine learning model, of the plurality of types of machine learning models, for the set of features, based on analyzing the set of features to evaluate the plurality of types of machine learning models (block 450); tuning a set of parameters of the particular type of machine learning model, to train the machine learning model (block 460); ¶¶ [0083]-[0085]: analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models; analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes providing, via a user interface, a visualization of model performance of the plurality of types of machine learning models, and receiving, via the user interface, a selection of the particular type of machine learning model based on providing the visualization of the model performance of the plurality of types of machine learning models; tuning the set of parameters of the particular type of machine learning model includes optimizing hyper parameters of the particular type of machine learning model, and retraining the machine learning model based on optimizing the hyper parameters of the machine learning; ¶¶ [0092]-[0093] with 540-550 in FIG. 5: selecting, for the output data, a machine learning model, of a plurality of types of machine learning models, to apply to the output data (block 540); tuning a set of hyper-parameters for the machine learning model based on selecting the machine learning model (block 550); ¶ [0105]: the plurality of types of machine learning models includes at least one of a random forest classifier model, a k-nearest neighbor model, a decision tree model, a multilayer perceptron model, a stochastic gradient descent classifier model, a logistic regression model, a linear support vector classifier model, a naive Bayes model, a ridge regression model, a convolutional neural network model, or an Inception v3 model; ¶¶ [0111]-[0112] with 640-650 in FIG. 6: applying the set of features to a plurality of types of machine learning models to evaluate the plurality of types of machine learning models with respect to completing the natural language processing or image processing task (block 640); selecting a particular type of machine learning model, from the plurality of types of machine learning models, for the set of features, based on applying the set of features to the plurality of types of machine learning models (block 650); ¶¶ [0117]-[0119]: automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models; providing, via a user interface, a visualization of model performance of the plurality of types of machine learning models at completing the natural language processing or image processing task; and receiving, via the user interface, a selection of the particular type of machine learning model based on providing the visualization of the model performance of the plurality of types of machine learning models; optimizing hyper parameters of the particular type of machine learning model; and retraining the particular type of machine learning model based on optimizing the hyper parameters of the particular type of machine learning model); and
obtaining a prediction for (HIGGINS, ¶¶ [0036] and [0039]: provide information identifying results from test data applied to a set of types of machine learning models to provide information, to a user, regarding how each type of machine learning model may respond to a scenario; determine a test accuracy score (e.g., an Fl score), a positive predictive value score (e.g., a precision score), a sensitivity score (e.g., a recall score), and/or the like; provide a user interface to visualize results of testing a set of different types of machine learning models (e.g., a set of hyper parameters, a timeline of model performance, a score, and/or the like); ¶¶ [0077]-[0079] with 470- in FIG. 4: providing access to the particular type of machine learning model via an interface (block 470); receiving, as input via the interface, third data for prediction using the particular type of machine learning model (block 480); providing, as output via the interface, a prediction using the particular type of machine learning model based on receiving the third data (block 490); ¶¶ [0094]-[0097] with 560-590 in FIG. 5: establishing a model pipeline for the machine learning model based on tuning the set of hyper-parameters (block 560); receiving prediction data based on establishing the model pipeline (block 570); performing a prediction using the machine learning model and using the prediction data (block 580); providing the prediction for display via a user interface based on performing the prediction (block 590); ¶¶ [0113]-[0115] with 660-680 in FIG. 6: providing access to the particular type of machine learning model via an interface (block 660); receiving, as input via the interface, third data for prediction using the particular type of machine learning model (block 670); providing, as output via the interface, a prediction, using the particular type of machine learning model, based on receiving the third data, wherein the prediction is a text-completion prediction or an image recognition prediction (block 680)).
HIGGINS fails to explicitly disclose validating the set of relevant training data samples.
Karumanchi teaches a system and a method relating to machine learning (Karumanchi, ABSTRACT), wherein validating the set of relevant training data samples (Karumanchi, ¶¶ [0047]-[0049] with FIG. 3: ¶¶ [0047]-[0049] with FIG. 3: continues to progressively sample the clusters until a prediction accuracy threshold is met by training a prediction model using the sampled data or until a sampling memory usage threshold has been met; the prediction accuracy threshold may be replaced with a comparison of progressive accuracies; when the accuracy converges or does not improve, the progressive sampling stops at that point for output of the sampled data and the trained ML model from a previous progressive sampling iteration; obtain a classification accuracy for the trained ML model on a held-out test data set or using k-fold cross validation on the obtained training data set, and compares the classification accuracy with an accuracy from a previous sampling of the data; upon a determination that the classification accuracy improves over the accuracy from the previous sampling of the data, perform incremental sampling to a second sampling percentage (e.g., higher than the first percentage); upon a determination that the classification accuracy converges or does not improve over the accuracy from the previous sampling of the data, or a total sampling size is larger than a predetermined sampling size threshold, output the sampled data and the trained ML model from a previous progressive sampling iteration; combine the sampled representative data to generate training data for processing predictive analytics; predict data relevance, such as business data relevance (e.g., classified/unclassified, secure/unsecure, sensitive/non-sensitive, etc.) by learning from the sampled training data to predict different categories (e.g., classified, unclassified; private, public, etc.); ¶¶ [0070]-[0071] with FIG. 8: the progressive incremental sampling processing 810 starts from a relatively small sampling percentage; the previous clustering-based sampling is applied to obtain a training data set 750, which is combined with existing training data; content processing 820 processes the actual content of a data file and applies keywords for pattern matching to determine, e.g., confidentiality; the output of the content processing (confidential 825 and non-confidential 830) is input to train a machine learning model(s) 620 and obtain its classification accuracy 840 on a held-out test dataset or using k-fold cross validation on all the obtained training data; the ML model(s) output is input to the business relevance classifier 630; the resulting accuracy is compared with the accuracy from the previous running (set the previous accuracy to O for the first running); if the accuracy improves, then incremental sampling Δxi is performed; if the accuracy converges or does not improve, or the total sampling size is larger than a predetermined threshold (beyond the memory usage and/or processing latency), the progressive incremental sampling processing 810 stops and outputs the sampled data and the trained machine learning model at the previous step; ¶ [0072] with FIG. 9: in block 920, process 900 perform clustering on features (S1 440 x S2 450 x S4 470, FIG. 4) extracted from the data; in block 930 sampling Δxi is performed from each cluster; in block 940 the actual content S3 460 of the sampled data is processed to obtain the respective classification labels; in block 950 the current sampled data is combined with existing sampled data to form a new set of training data; in block 960 the ML model(s) is trained using the set of training data; in block 970 classification accuracy is determined using k-fold cross validation or on a held out test dataset; in block 980 it is determined whether the accuracy improves or converges or does not improve; if the accuracy improves, process 900 proceeds to block 930, otherwise process 900 proceeds to block 990; in block 990, process 900 outputs the sampled data and the trained ML model at for the previous processing iteration; ¶ [0075]: perform progressive sampling of the multiple clusters by sampling the multiple clusters with a first sampling percentage, applying a previous clustering-based sampling to obtain a training data set, combining the training data set with previous determined training data, training an ML model and obtaining a classification accuracy for the ML model on a held-out test data set or using k-fold cross validation on the obtained training data set, and comparing the classification accuracy with an accuracy from a previous sampling of the data).
HIGGINS and Karumanchi are analogous art because they are from the same field of endeavor, a system and a method relating to machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Karumanchi to HIGGINS. Motivation for doing so would ensure that samples data appropriate for ML training are efficient.
Claim 7
HIGGINS in view of Karumanchi discloses all the elements as stated in Claim 6 and further discloses wherein determining a right set of features for training the machine learning model is based on at least one of: identifying a plurality of features from the input training data; selecting a set of relevant features from the plurality of features; generating one or more clusters from the selected set of relevant features based on one or more distance-based techniques; ranking the generated one or more clusters; determining a sample selection number from the ranked one or more clusters and creating a sample arrangement; selecting a sample from the created sample arrangement; and creating a reduced dataset from the selected sample (see also 112(b) Rejections) (HIGGINS, ¶¶ [0015]-[0029] with FIG. 1A: data processing platform 102 may provide a user interface identifying characteristics of the training data; data processing platform 102 may obtain contextual information relating to the training data, such as information identifying input fields, output fields, a clustering parameter for clustering-based pre-processing, and/or the like; by reference number 152, data processing platform 102 may curate the training data; e.g., data processing platform 102 may perform a rule-based pre-processing procedure, a clustering-based pre-processing procedure, and/or the like to curate and/or transform the training data; data processing platform 102 may apply one or more training data modification filtering rules to perform rule-based pre-processing; e.g., data processing platform 102 may apply one or more rules regarding a type of data that is to be received as training data, and may perform one or more procedures to enforce the one or more rules, such as performing a space trimming procedure, a case lowering procedure, a stop-words removal procedure, and/or the like; data processing platform 102 may parse the training data to determine if a data entry has a regular expression pattern matching a particular regular expression that is configured for data filtering and may filter the data entry accordingly; data processing platform 102 may generate a set of clusters using the training data to perform a cluster analysis; data processing platform 102 may generate the set of clusters and may identify outlier data based on the set of clusters; data processing platform 102 may automatically transform the training data by removing or altering the outlier data; data processing platform 102 may apply labels to data entries based on the training data and/or may apply labels to clusters of data entries based on generating a set of clusters of data entries; data processing platform 102 may determine a data risk value (e.g., which may be a categorical value, such as low risk, medium risk, high risk, and/or the like) based on the clustering index value; e.g., data processing platform 102 may determine that a data set within the training data with a relatively high data purity may be classified as a low risk data set, whereas a data set with a relatively low data purity may be classified as a high risk data set; data processing platform 102 classifies the data set to enable selection of a low risk portion of the data set for training one or more models, resulting in a higher accuracy of the one or more models; data processing platform 102 may alter a parameter of the training data set based on determining the data purity and data risk; data processing platform 102 may alter labels of data based on the data purity and/or data risk determination; data processing platform 102 may determine whether the dataset has a threshold imbalance, a threshold skew of an attribute, and/or the like; data processing platform 102 may determine a data quality based on the balance metric and for different types of data (e.g., integer data, categorical data, date data, identifier data, text data, and/or the like); data processing platform 102 may provide information identifying the data quality as output, may automatically select a subset of the training data for use in machine learning to ensure a threshold level of data quality in data selected for the machine learning, and/or the like; data processing platform 102 may automatically identify a data type of a data entry in the training data (e.g., a determination of whether an attribute value in a data entry matches a regular expression); data processing platform 102 may identify categorical data, identifier data, textual data, and/or the like; data processing platform 102 may use an association technique to account for multicollinearity of attributes; data processing platform 102 may label one or more data entries that are missing labels; data processing platform 102 may determine the labels based on a cluster; data processing platform 102 may use an unsupervised machine learning algorithm to enable data cleaning, data set alteration, labeling, and/or the like; data processing platform 102 may automatically adjust missing values, remove skewness of the data (e.g., using logarithmic, normalization, standardization, and/or the like techniques), add labels to the data, filter attributes of the data, remove or alter outliers in the data, and/or the like to improve a data quality of data used for machine learning; data processing platform 102 may use an oversampling technique to generate synthetic examples for the training data set, to avoid under-sampled classes and/or clusters within the training data; ¶¶ [0072]-[0073] with 420-430 in FIG. 4: pre-processing the first data to alter the first data to generate second data (block 420); processing the second data to select a set of features from the second data (block 430); ¶¶ [0081]-[0082]: processing the second data includes identifying a plurality of features of the second data, and performing a feature reduction procedure to identify the set of features from the plurality of features of the second data; analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes classifying the plurality of types of machine learning models based on a type of problem associated with the first data; ¶¶ [0090]-[0091] with 520-530 in FIG. 5: pre-processing and filtering the input data to generate intermediate data based on receiving the input data (block 520); labeling one or more missing labels in the intermediate data to generate output data based on generating the intermediate data (block 530); ¶ [0099]: identifying the plurality of languages, and training, for the machine learning model, a plurality of sub-models for the plurality of languages; ¶¶ [0103]-[0104]: the cluster analysis identifies a plurality of predicted sources of error in the input data; when pre-processing and filtering the input data, are configured to perform at least one of a space-trimming procedure, a case lowering procedure, a stop words removal procedure, a Boolean logic filtering procedure, or a regular expression pattern matching procedure; ¶¶ [0109]-[0110] with 620-630 in FIG. 6: pre-processing and filtering the first data to generate second data (block 620); processing the second data to select a set of features from a plurality of features of the second data (block 630)) (Karumanchi, ABSTRACT and ¶¶ [0003], [0042], [0044], and [0046] with 310 in FIG. 3: the clustering processor 310 clusters the data into multiple clusters based on similarities of the obtained data across an entire storage stack; employ a clustering based sampling component to group data points into clusters across three layers (storage infrastructure metrics, file metadata, and applications); ¶ [0068] with FIGS. 4 and 6: the training data with features 610 are obtained across the feature space S1 440 x S2 450 x S4 470; the training data structure 640 that is generated by the processing system includes the data identifier, the features S1 440 x S2 450 x S4 470 and the class labels assigned by S3 460; the clustering based sampling component provides for clustering all the data points across the feature space S1 440 x S2 450 x S4 470 using data 730 and the feature space of the training data structure 640, obtains the cluster centroids 720 and computes the percentage of data assigned to each cluster shown as 750; the clustering may use Vertex Substitution Heuristic (VSH) processing, which is a distance-based clustering algorithm; K-means or other clustering algorithms which clusters data points in a vector space can also be used since the data points 410 are also described in feature vectors; random sampling is performed from each cluster proportionally with respect to the previously obtained percentage; ¶ [0072] with 920 in FIG. 9: in block 920, process 900 perform clustering on features (S1 440 x S2 450 x S4 470, FIG. 4) extracted from the data; in block 930 sampling Δxi is performed from each cluster).
Claim 10
HIGGINS in view of Karumanchi discloses all the elements as stated in Claim 6 (see also Claim Objections) and further disclose wherein validating the set of relevant training data samples comprises: obtaining, from the suitable type and configuration of machine learning model, a prediction and an accuracy probability score associated with the prediction; comparing the accuracy probability score with a threshold score; and validating the set of relevant training data samples based on the comparison (HIGGINS, ¶¶ [0036] and [0039]: provide information identifying results from test data applied to a set of types of machine learning models to provide information, to a user, regarding how each type of machine learning model may respond to a scenario; determine a test accuracy score (e.g., an Fl score), a positive predictive value score (e.g., a precision score), a sensitivity score (e.g., a recall score), and/or the like; provide a user interface to visualize results of testing a set of different types of machine learning models (e.g., a set of hyper parameters, a timeline of model performance, a score, and/or the like); ¶¶ [0077]-[0079] with 470- in FIG. 4: providing access to the particular type of machine learning model via an interface (block 470); receiving, as input via the interface, third data for prediction using the particular type of machine learning model (block 480); providing, as output via the interface, a prediction using the particular type of machine learning model based on receiving the third data (block 490); ¶¶ [0094]-[0097] with 560-590 in FIG. 5: establishing a model pipeline for the machine learning model based on tuning the set of hyper-parameters (block 560); receiving prediction data based on establishing the model pipeline (block 570); performing a prediction using the machine learning model and using the prediction data (block 580); providing the prediction for display via a user interface based on performing the prediction (block 590); ¶¶ [0113]-[0115] with 660-680 in FIG. 6: providing access to the particular type of machine learning model via an interface (block 660); receiving, as input via the interface, third data for prediction using the particular type of machine learning model (block 670); providing, as output via the interface, a prediction, using the particular type of machine learning model, based on receiving the third data, wherein the prediction is a text-completion prediction or an image recognition prediction (block 680)) (Karumanchi, ¶¶ [0047]-[0049] with FIG. 3: continues to progressively sample the clusters until a prediction accuracy threshold is met by training a prediction model using the sampled data or until a sampling memory usage threshold has been met; the prediction accuracy threshold may be replaced with a comparison of progressive accuracies; when the accuracy converges or does not improve, the progressive sampling stops at that point for output of the sampled data and the trained ML model from a previous progressive sampling iteration; obtain a classification accuracy for the trained ML model on a held-out test data set or using k-fold cross validation on the obtained training data set, and compares the classification accuracy with an accuracy from a previous sampling of the data; upon a determination that the classification accuracy improves over the accuracy from the previous sampling of the data, perform incremental sampling to a second sampling percentage (e.g., higher than the first percentage); upon a determination that the classification accuracy converges or does not improve over the accuracy from the previous sampling of the data, or a total sampling size is larger than a predetermined sampling size threshold, output the sampled data and the trained ML model from a previous progressive sampling iteration; combine the sampled representative data to generate training data for processing predictive analytics; predict data relevance, such as business data relevance (e.g., classified/unclassified, secure/unsecure, sensitive/non-sensitive, etc.) by learning from the sampled training data to predict different categories (e.g., classified, unclassified; private, public, etc.); ¶¶ [0070]-[0071] with FIG. 8: the progressive incremental sampling processing 810 starts from a relatively small sampling percentage; the previous clustering-based sampling is applied to obtain a training data set 750, which is combined with existing training data; content processing 820 processes the actual content of a data file and applies keywords for pattern matching to determine, e.g., confidentiality; the output of the content processing (confidential 825 and non-confidential 830) is input to train a machine learning model(s) 620 and obtain its classification accuracy 840 on a held-out test dataset or using k-fold cross validation on all the obtained training data; the ML model(s) output is input to the business relevance classifier 630; the resulting accuracy is compared with the accuracy from the previous running (set the previous accuracy to O for the first running); if the accuracy improves, then incremental sampling Δxi is performed; if the accuracy converges or does not improve, or the total sampling size is larger than a predetermined threshold (beyond the memory usage and/or processing latency), the progressive incremental sampling processing 810 stops and outputs the sampled data and the trained machine learning model at the previous step; ¶ [0072] with FIG. 9: in block 920, process 900 perform clustering on features (S1 440 x S2 450 x S4 470, FIG. 4) extracted from the data; in block 930 sampling Δxi is performed from each cluster; in block 940 the actual content S3 460 of the sampled data is processed to obtain the respective classification labels; in block 950 the current sampled data is combined with existing sampled data to form a new set of training data; in block 960 the ML model(s) is trained using the set of training data; in block 970 classification accuracy is determined using k-fold cross validation or on a held out test dataset; in block 980 it is determined whether the accuracy improves or converges or does not improve; if the accuracy improves, process 900 proceeds to block 930, otherwise process 900 proceeds to block 990; in block 990, process 900 outputs the sampled data and the trained ML model at for the previous processing iteration; ¶ [0075]: perform progressive sampling of the multiple clusters by sampling the multiple clusters with a first sampling percentage, applying a previous clustering-based sampling to obtain a training data set, combining the training data set with previous determined training data, training an ML model and obtaining a classification accuracy for the ML model on a held-out test data set or using k-fold cross validation on the obtained training data set, and comparing the classification accuracy with an accuracy from a previous sampling of the data)
Claims 3-4 and 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over HIGGINS in view of Karumanchi as applied to Claims 1 and 6 respectively above, and further in view of Moghadam et al. (US 2020/0334569 A1, pub. date: 10/22/2020), hereinafter Moghadam.
Claim 3
HIGGINS in view of Karumanchi discloses all the elements as stated in Claim 1 and further discloses wherein selecting a suitable type and configuration of machine learning model comprises: monitoring performance of each of the plurality of types and configurations of machine learning models with the set of relevant training data samples; scoring scoring (HIGGINS, HIGGINS, ¶¶ [0030]-[0039] with FIGS. 1A-1B: by reference number 154, data processing platform 102 may provide curated training data to model selection platform 104; by reference number 156, model selection platform 104 may obtain algorithms for a set of types of machine learning models from model repository 110; e.g., model selection platform 104 may obtain stored algorithms representing classification models (e.g., a random forest classifier model, a k-nearest neighbors model, a decision tree model, a multilayer perceptron model, a stochastic gradient descent classifier model, a logistic regression model, a linear support vector classifier model, a naive Bayes model, a gradient boosting model, and/or the like), regression models ( e.g., a ridge regression model), deep learning classification models (e.g., a convolutional neural network model, an Inception V3 model, and/or the like), and/or the like; by reference number 158, model selection platform 104 may automatically select, train, and deploy a particular machine learning model of the set of possible machine learning models; e.g., model selection platform 104 may classify a task type, select a machine learning model category for the task type, select a machine learning model type from the model category, tune parameters of the machine learning model, and train the machine learning model; model selection platform 104 may determine one or more parameters relating to deploying a particular type of machine learning model; model selection platform 104 may determine that the training data relates to a classification task, a regression task, and/or the like, and may filter the set of types of machine learning models based on the type of task; model selection platform 104 may determine that a task is a natural language processing task, an image processing task, a computer vision task, and/or the like; model selection platform 104 may define one or more use cases for a machine learning model, define one or more inputs or outputs for the machine learning model, and/or the like to enable selection of a machine learning model; based on obtaining information identifying a set of types of machine learning models and the information identifying a task type (e.g., a service management task, an Information Technology (IT) task, an application development task, a service transition task, and/or the like) model selection platform 104 may determine that a machine learning model is deployed for the same task, for a similar task (e.g., based on a similarity score calculated based on training data parameters), and/or the like; in this case, model selection platform 104 may determine to modify the existing machine learning model (e.g., by retraining the machine learning model on new training data); model selection platform 104 may use a local interpretable model agnostic explanation (LIME) model to identify an influence of features on the prediction instances of multiple types of machine learning models; model selection platform 104 may automatically select the features for one or more types of machine learning model; e.g., based on the data curation, model selection platform 104 may identify low risk data entries and associated variables to use as features for training a machine learning model; model selection platform 104 may tune hyper parameters in connection with selection of a machine learning model; model selection platform 104 may evaluate a set of types of machine learning models, and may select a particular type of machine learning model with a greatest quantity of optimal hyper parameters; model selection platform 104 uses hyper parameter optimization to select a machine learning model; when a particular type of machine learning model is selected (e.g., using another technique as described herein), model selection platform 104 may automatically tune hyper parameters of the particular machine learning model to enable the particular type of machine learning model to perform predictions using subsequent prediction data; to identify best estimated hyper parameters of multiple types of machine learning models, model selection platform 104 may use a tree-structured Parzen estimator to identify best estimated hyper parameters within a high dimensional search space of parameters of the multiple types of machine learning models; in this way, model selection platform 104 may enable selection from multiple types of machine learning models with multiple sets of parameters and a high dimensional search space; model selection platform 104 may determine a score ( e.g., based on tuning hyper parameters), and may select a particular type of machine learning model based on the score; model selection platform 104 may determine a timeline of a model's performance at evaluating training data, and may select a particular type of machine learning model based on a score, the timeline, and/or the like; ¶¶ [0074]-[0076] with 440-460 in FIG. 4: analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features (block 440); selecting a particular type of machine learning model, of the plurality of types of machine learning models, for the set of features, based on analyzing the set of features to evaluate the plurality of types of machine learning models (block 450); tuning a set of parameters of the particular type of machine learning model, to train the machine learning model (block 460); ¶¶ [0083]-[0085]: analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models; analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes providing, via a user interface, a visualization of model performance of the plurality of types of machine learning models, and receiving, via the user interface, a selection of the particular type of machine learning model based on providing the visualization of the model performance of the plurality of types of machine learning models; tuning the set of parameters of the particular type of machine learning model includes optimizing hyper parameters of the particular type of machine learning model, and retraining the machine learning model based on optimizing the hyper parameters of the machine learning; ¶¶ [0092]-[0093] with 540-550 in FIG. 5: selecting, for the output data, a machine learning model, of a plurality of types of machine learning models, to apply to the output data (block 540); tuning a set of hyper-parameters for the machine learning model based on selecting the machine learning model (block 550); ¶ [0105]: the plurality of types of machine learning models includes at least one of a random forest classifier model, a k-nearest neighbor model, a decision tree model, a multilayer perceptron model, a stochastic gradient descent classifier model, a logistic regression model, a linear support vector classifier model, a naive Bayes model, a ridge regression model, a convolutional neural network model, or an Inception v3 model; ¶¶ [0111]-[0112] with 640-650 in FIG. 6: applying the set of features to a plurality of types of machine learning models to evaluate the plurality of types of machine learning models with respect to completing the natural language processing or image processing task (block 640); selecting a particular type of machine learning model, from the plurality of types of machine learning models, for the set of features, based on applying the set of features to the plurality of types of machine learning models (block 650); ¶¶ [0117]-[0119]: automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models; providing, via a user interface, a visualization of model performance of the plurality of types of machine learning models at completing the natural language processing or image processing task; and receiving, via the user interface, a selection of the particular type of machine learning model based on providing the visualization of the model performance of the plurality of types of machine learning models; optimizing hyper parameters of the particular type of machine learning model; and retraining the particular type of machine learning model based on optimizing the hyper parameters of the particular type of machine learning model).
HIGGINS in view of Karumanchi fails to explicitly disclose ranking each of the plurality of types and configurations of machine learning models based on the performance; and selecting the suitable type and configuration of machine learning model from the plurality of types and configurations of the machine learning models, based on the ranking.
Moghadam teaches a system and a method relating to optimal selection of machine learning algorithms (Moghadam, ¶ [0001]), wherein ranking each of the plurality of types and configurations of machine learning models based on the performance; and selecting the suitable type and configuration of machine learning model from the plurality of types and configurations of the machine learning models, based on the ranking (Moghadam, ¶¶ [0019]-[0040] and [0048]-[0049] with FIG. 1: select (for training and/or inference) a few of machine learning algorithms 121-123 that could produce the best (most accurate, least error) results with sampled dataset 110; different configuration alternatives of machine learning algorithm 121 may be more suited or less suited for analyzing different categories of datasets; create or obtain hyperparameter predictors 135 for each of machine learning algorithms 121-123 to predict and produce optimal mini-model hyperparameters; a mini-model 140 closely tracks the RML model's score within some error bound (c), allowing a user to judge the relative performance of the RML model on a given dataset by simply using a mini-model in place of a corresponding RML model; create or obtain RML predictors for each of machine learning algorithms 121-123 to quickly and accurately predict the performance of each machine learning algorithm; a score may predictively measure how proficient (accuracy such as error rate) would a particular configuration of a particular machine learning algorithm become after training for a fixed duration with a particular training dataset, for which sampled dataset 110 is representative (e.g. small sample) of the training dataset; a score may instead predictively measure how much time does a particular configuration of a particular machine learning algorithm need to achieve a fixed proficiency for a particular training data set; a score may simply be a comparative measure of abstract suitability of a particular machine learning algorithm for a particular dataset; rank each machine learning algorithm based on an emitted score of RML predictor and select the best ranked machine learning algorithm as an optimal candidate for the input dataset; meta-features 131-133 and corresponding values 171-173 are generated based on the sampled dataset 110; the values of the meta-features are input to the hyperparameter predictors 135 which produce hyperparameters as output; the hyperparameters are used an input to the mini-model along with the sampled dataset 110 and/or values of the meta-features; the mini-model 140 produces a score as output and the score is used along with the meta-feature values as input to the RML predictor 145; the RML predictor 145 produces a score; the score is input to a ranking algorithm and the score of machine learning algorithm 121 is ranked alongside scores of other machine learning algorithms 122, 123; ¶¶ [0056]-[0060] with 310-314 in FIG. 3: in step 310, the respective MML model is trained that predicts a mini-model score; in step 312, a respective reference RML, predictor of said MML model is trained that predicts a respective RML predictor score; a reference RML predictor is trained using meta-features generated from samples of the data set generated in step 304, mini-model scores of said MML model such as generated in step 310, and a tuned score generated by applying the AHT to the data set; in step 314, for each MML model, a respective RML predictor score is calculated by invoking the respective reference RML predictor, wherein the respective RML predictor score is based on a respective subset of meta-feature values and respective mini-model score; each machine learning algorithm is ranked based on the scores; the machine learning algorithm corresponding to the RML predictor with the highest score is selected as the optimal machine learning algorithm for the data set).
HIGGIN and Moghadam are analogous art because they are from the same field of endeavor, a system and a method relating to machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Moghadam to HIGGINS in view of Karumanchi. Motivation for doing so would (1) avoid training of an excessive count of different machine learning algorithms by selecting well suited machine learning algorithms and/or their configurations.
Claim 4
HIGGINS in view of Karumanchi and Moghadam discloses all the elements as stated in Claim 3 and further discloses wherein selecting the suitable type and configuration of a machine learning model further comprises pre-processing the set of relevant training data samples, wherein the pre-processing comprises at least one of imputing missing values; removing outlier values; handling categorical variable values; removing duplicate data samples; and correcting inconsistent data samples (HIGGINS, ¶¶ [0016]-[0029] with FIG. 1A: by reference number 152, data processing platform 102 may curate the training data; e.g., data processing platform 102 may perform a rule-based pre-processing procedure, a clustering-based pre-processing procedure, and/or the like to curate and/or transform the training data; data processing platform 102 may apply one or more training data modification filtering rules to perform rule-based pre-processing; e.g., data processing platform 102 may apply one or more rules regarding a type of data that is to be received as training data, and may perform one or more procedures to enforce the one or more rules, such as performing a space trimming procedure, a case lowering procedure, a stop-words removal procedure, and/or the like; data processing platform 102 may parse the training data to determine if a data entry has a regular expression pattern matching a particular regular expression that is configured for data filtering and may filter the data entry accordingly; data processing platform 102 may generate a set of clusters using the training data to perform a cluster analysis; data processing platform 102 may generate the set of clusters and may identify outlier data based on the set of clusters; data processing platform 102 may automatically transform the training data by removing or altering the outlier data; data processing platform 102 may apply labels to data entries based on the training data and/or may apply labels to clusters of data entries based on generating a set of clusters of data entries; data processing platform 102 may determine a data risk value (e.g., which may be a categorical value, such as low risk, medium risk, high risk, and/or the like) based on the clustering index value; e.g., data processing platform 102 may determine that a data set within the training data with a relatively high data purity may be classified as a low risk data set, whereas a data set with a relatively low data purity may be classified as a high risk data set; data processing platform 102 classifies the data set to enable selection of a low risk portion of the data set for training one or more models, resulting in a higher accuracy of the one or more models; data processing platform 102 may alter a parameter of the training data set based on determining the data purity and data risk; data processing platform 102 may alter labels of data based on the data purity and/or data risk determination; data processing platform 102 may determine whether the dataset has a threshold imbalance, a threshold skew of an attribute, and/or the like; data processing platform 102 may determine a data quality based on the balance metric and for different types of data (e.g., integer data, categorical data, date data, identifier data, text data, and/or the like); data processing platform 102 may provide information identifying the data quality as output, may automatically select a subset of the training data for use in machine learning to ensure a threshold level of data quality in data selected for the machine learning, and/or the like; data processing platform 102 may automatically identify a data type of a data entry in the training data (e.g., a determination of whether an attribute value in a data entry matches a regular expression); data processing platform 102 may identify categorical data, identifier data, textual data, and/or the like; data processing platform 102 may use an association technique to account for multicollinearity of attributes; data processing platform 102 may label one or more data entries that are missing labels; data processing platform 102 may determine the labels based on a cluster; data processing platform 102 may use an unsupervised machine learning algorithm to enable data cleaning, data set alteration, labeling, and/or the like; data processing platform 102 may automatically adjust missing values, remove skewness of the data (e.g., using logarithmic, normalization, standardization, and/or the like techniques), add labels to the data, filter attributes of the data, remove or alter outliers in the data, and/or the like to improve a data quality of data used for machine learning; data processing platform 102 may use an oversampling technique to generate synthetic examples for the training data set, to avoid under-sampled classes and/or clusters within the training data; ¶ [0036] with FIG. 1: model selection platform 104 may use a local interpretable model agnostic explanation (LIME) model to identify an influence of features on the prediction instances of multiple types of machine learning models; model selection platform 104 may automatically select the features for one or more types of machine learning model; e.g., based on the data curation, model selection platform 104 may identify low risk data entries and associated variables to use as features for training a machine learning model; ¶¶ [0072]-[0073] with 420-430 in FIG. 4: pre-processing the first data to alter the first data to generate second data (block 420); processing the second data to select a set of features from the second data (block 430); ¶¶ [0081]-[0082]: processing the second data includes identifying a plurality of features of the second data, and performing a feature reduction procedure to identify the set of features from the plurality of features of the second data; analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes classifying the plurality of types of machine learning models based on a type of problem associated with the first data; ¶¶ [0090]-[0091] with 520-530 in FIG. 5: pre-processing and filtering the input data to generate intermediate data based on receiving the input data (block 520); labeling one or more missing labels in the intermediate data to generate output data based on generating the intermediate data (block 530); ¶ [0099]: identifying the plurality of languages, and training, for the machine learning model, a plurality of sub-models for the plurality of languages; ¶¶ [0103]-[0104]: the cluster analysis identifies a plurality of predicted sources of error in the input data; when pre-processing and filtering the input data, are configured to perform at least one of a space-trimming procedure, a case lowering procedure, a stop words removal procedure, a Boolean logic filtering procedure, or a regular expression pattern matching procedure; ¶¶ [0109]-[0110] with 620-630 in FIG. 6: pre-processing and filtering the first data to generate second data (block 620); processing the second data to select a set of features from a plurality of features of the second data (block 630)).
Claim 8
HIGGINS in view of Karumanchi discloses all the elements as stated in Claim 6 and further discloses wherein selecting a suitable type and configuration of machine learning model comprises: monitoring performance of each of the plurality of types and configurations of machine learning models with the set of relevant training data samples; scoring scoring (HIGGINS, HIGGINS, ¶¶ [0030]-[0039] with FIGS. 1A-1B: by reference number 154, data processing platform 102 may provide curated training data to model selection platform 104; by reference number 156, model selection platform 104 may obtain algorithms for a set of types of machine learning models from model repository 110; e.g., model selection platform 104 may obtain stored algorithms representing classification models (e.g., a random forest classifier model, a k-nearest neighbors model, a decision tree model, a multilayer perceptron model, a stochastic gradient descent classifier model, a logistic regression model, a linear support vector classifier model, a naive Bayes model, a gradient boosting model, and/or the like), regression models ( e.g., a ridge regression model), deep learning classification models (e.g., a convolutional neural network model, an Inception V3 model, and/or the like), and/or the like; by reference number 158, model selection platform 104 may automatically select, train, and deploy a particular machine learning model of the set of possible machine learning models; e.g., model selection platform 104 may classify a task type, select a machine learning model category for the task type, select a machine learning model type from the model category, tune parameters of the machine learning model, and train the machine learning model; model selection platform 104 may determine one or more parameters relating to deploying a particular type of machine learning model; model selection platform 104 may determine that the training data relates to a classification task, a regression task, and/or the like, and may filter the set of types of machine learning models based on the type of task; model selection platform 104 may determine that a task is a natural language processing task, an image processing task, a computer vision task, and/or the like; model selection platform 104 may define one or more use cases for a machine learning model, define one or more inputs or outputs for the machine learning model, and/or the like to enable selection of a machine learning model; based on obtaining information identifying a set of types of machine learning models and the information identifying a task type (e.g., a service management task, an Information Technology (IT) task, an application development task, a service transition task, and/or the like) model selection platform 104 may determine that a machine learning model is deployed for the same task, for a similar task (e.g., based on a similarity score calculated based on training data parameters), and/or the like; in this case, model selection platform 104 may determine to modify the existing machine learning model (e.g., by retraining the machine learning model on new training data); model selection platform 104 may use a local interpretable model agnostic explanation (LIME) model to identify an influence of features on the prediction instances of multiple types of machine learning models; model selection platform 104 may automatically select the features for one or more types of machine learning model; e.g., based on the data curation, model selection platform 104 may identify low risk data entries and associated variables to use as features for training a machine learning model; model selection platform 104 may tune hyper parameters in connection with selection of a machine learning model; model selection platform 104 may evaluate a set of types of machine learning models, and may select a particular type of machine learning model with a greatest quantity of optimal hyper parameters; model selection platform 104 uses hyper parameter optimization to select a machine learning model; when a particular type of machine learning model is selected (e.g., using another technique as described herein), model selection platform 104 may automatically tune hyper parameters of the particular machine learning model to enable the particular type of machine learning model to perform predictions using subsequent prediction data; to identify best estimated hyper parameters of multiple types of machine learning models, model selection platform 104 may use a tree-structured Parzen estimator to identify best estimated hyper parameters within a high dimensional search space of parameters of the multiple types of machine learning models; in this way, model selection platform 104 may enable selection from multiple types of machine learning models with multiple sets of parameters and a high dimensional search space; model selection platform 104 may determine a score ( e.g., based on tuning hyper parameters), and may select a particular type of machine learning model based on the score; model selection platform 104 may determine a timeline of a model's performance at evaluating training data, and may select a particular type of machine learning model based on a score, the timeline, and/or the like; ¶¶ [0074]-[0076] with 440-460 in FIG. 4: analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features (block 440); selecting a particular type of machine learning model, of the plurality of types of machine learning models, for the set of features, based on analyzing the set of features to evaluate the plurality of types of machine learning models (block 450); tuning a set of parameters of the particular type of machine learning model, to train the machine learning model (block 460); ¶¶ [0083]-[0085]: analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models; analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes providing, via a user interface, a visualization of model performance of the plurality of types of machine learning models, and receiving, via the user interface, a selection of the particular type of machine learning model based on providing the visualization of the model performance of the plurality of types of machine learning models; tuning the set of parameters of the particular type of machine learning model includes optimizing hyper parameters of the particular type of machine learning model, and retraining the machine learning model based on optimizing the hyper parameters of the machine learning; ¶¶ [0092]-[0093] with 540-550 in FIG. 5: selecting, for the output data, a machine learning model, of a plurality of types of machine learning models, to apply to the output data (block 540); tuning a set of hyper-parameters for the machine learning model based on selecting the machine learning model (block 550); ¶ [0105]: the plurality of types of machine learning models includes at least one of a random forest classifier model, a k-nearest neighbor model, a decision tree model, a multilayer perceptron model, a stochastic gradient descent classifier model, a logistic regression model, a linear support vector classifier model, a naive Bayes model, a ridge regression model, a convolutional neural network model, or an Inception v3 model; ¶¶ [0111]-[0112] with 640-650 in FIG. 6: applying the set of features to a plurality of types of machine learning models to evaluate the plurality of types of machine learning models with respect to completing the natural language processing or image processing task (block 640); selecting a particular type of machine learning model, from the plurality of types of machine learning models, for the set of features, based on applying the set of features to the plurality of types of machine learning models (block 650); ¶¶ [0117]-[0119]: automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models; providing, via a user interface, a visualization of model performance of the plurality of types of machine learning models at completing the natural language processing or image processing task; and receiving, via the user interface, a selection of the particular type of machine learning model based on providing the visualization of the model performance of the plurality of types of machine learning models; optimizing hyper parameters of the particular type of machine learning model; and retraining the particular type of machine learning model based on optimizing the hyper parameters of the particular type of machine learning model).
HIGGINS in view of Karumanchi fails to explicitly disclose ranking each of the plurality of types and configurations of machine learning models based on the performance; and selecting the suitable type and configuration of machine learning model from the plurality of types and configurations of the machine learning models, based on the ranking.
Moghadam teaches a system and a method relating to optimal selection of machine learning algorithms (Moghadam, ¶ [0001]), wherein ranking each of the plurality of types and configurations of machine learning models based on the performance; and selecting the suitable type and configuration of machine learning model from the plurality of types and configurations of the machine learning models, based on the ranking (Moghadam, ¶¶ [0019]-[0040] and [0048]-[0049] with FIG. 1: select (for training and/or inference) a few of machine learning algorithms 121-123 that could produce the best (most accurate, least error) results with sampled dataset 110; different configuration alternatives of machine learning algorithm 121 may be more suited or less suited for analyzing different categories of datasets; create or obtain hyperparameter predictors 135 for each of machine learning algorithms 121-123 to predict and produce optimal mini-model hyperparameters; a mini-model 140 closely tracks the RML model's score within some error bound (c), allowing a user to judge the relative performance of the RML model on a given dataset by simply using a mini-model in place of a corresponding RML model; create or obtain RML predictors for each of machine learning algorithms 121-123 to quickly and accurately predict the performance of each machine learning algorithm; a score may predictively measure how proficient (accuracy such as error rate) would a particular configuration of a particular machine learning algorithm become after training for a fixed duration with a particular training dataset, for which sampled dataset 110 is representative (e.g. small sample) of the training dataset; a score may instead predictively measure how much time does a particular configuration of a particular machine learning algorithm need to achieve a fixed proficiency for a particular training data set; a score may simply be a comparative measure of abstract suitability of a particular machine learning algorithm for a particular dataset; rank each machine learning algorithm based on an emitted score of RML predictor and select the best ranked machine learning algorithm as an optimal candidate for the input dataset; meta-features 131-133 and corresponding values 171-173 are generated based on the sampled dataset 110; the values of the meta-features are input to the hyperparameter predictors 135 which produce hyperparameters as output; the hyperparameters are used an input to the mini-model along with the sampled dataset 110 and/or values of the meta-features; the mini-model 140 produces a score as output and the score is used along with the meta-feature values as input to the RML predictor 145; the RML predictor 145 produces a score; the score is input to a ranking algorithm and the score of machine learning algorithm 121 is ranked alongside scores of other machine learning algorithms 122, 123; ¶¶ [0056]-[0060] with 310-314 in FIG. 3: in step 310, the respective MML model is trained that predicts a mini-model score; in step 312, a respective reference RML, predictor of said MML model is trained that predicts a respective RML predictor score; a reference RML predictor is trained using meta-features generated from samples of the data set generated in step 304, mini-model scores of said MML model such as generated in step 310, and a tuned score generated by applying the AHT to the data set; in step 314, for each MML model, a respective RML predictor score is calculated by invoking the respective reference RML predictor, wherein the respective RML predictor score is based on a respective subset of meta-feature values and respective mini-model score; each machine learning algorithm is ranked based on the scores; the machine learning algorithm corresponding to the RML predictor with the highest score is selected as the optimal machine learning algorithm for the data set).
HIGGINS in view of and Moghadam are analogous art because they are from the same field of endeavor, a system and a method relating to machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Moghadam to HIGGINS in view of Karumanchi. Motivation for doing so would (1) avoid training of an excessive count of different machine learning algorithms by selecting well suited machine learning algorithms and/or their configurations; (2) increase contextual suitability of selection by scoring based on fitness for actual dataset meta-feature values; and (3) accurately rank and select the best machine learning algorithm for the given dataset by improving the accuracy of scores of each mini-model .
Claim 9
HIGGINS in view of Karumanchi and Moghadam discloses all the elements as stated in Claim 8 and further discloses wherein selecting the suitable type and configuration of a machine learning model further comprises pre-processing the set of relevant training data samples, wherein the pre-processing comprises at least one of imputing missing values; removing outlier values; handling categorical variable values; removing duplicate data samples; and correcting inconsistent data samples (HIGGINS, ¶¶ [0016]-[0029] with FIG. 1A: by reference number 152, data processing platform 102 may curate the training data; e.g., data processing platform 102 may perform a rule-based pre-processing procedure, a clustering-based pre-processing procedure, and/or the like to curate and/or transform the training data; data processing platform 102 may apply one or more training data modification filtering rules to perform rule-based pre-processing; e.g., data processing platform 102 may apply one or more rules regarding a type of data that is to be received as training data, and may perform one or more procedures to enforce the one or more rules, such as performing a space trimming procedure, a case lowering procedure, a stop-words removal procedure, and/or the like; data processing platform 102 may parse the training data to determine if a data entry has a regular expression pattern matching a particular regular expression that is configured for data filtering and may filter the data entry accordingly; data processing platform 102 may generate a set of clusters using the training data to perform a cluster analysis; data processing platform 102 may generate the set of clusters and may identify outlier data based on the set of clusters; data processing platform 102 may automatically transform the training data by removing or altering the outlier data; data processing platform 102 may apply labels to data entries based on the training data and/or may apply labels to clusters of data entries based on generating a set of clusters of data entries; data processing platform 102 may determine a data risk value (e.g., which may be a categorical value, such as low risk, medium risk, high risk, and/or the like) based on the clustering index value; e.g., data processing platform 102 may determine that a data set within the training data with a relatively high data purity may be classified as a low risk data set, whereas a data set with a relatively low data purity may be classified as a high risk data set; data processing platform 102 classifies the data set to enable selection of a low risk portion of the data set for training one or more models, resulting in a higher accuracy of the one or more models; data processing platform 102 may alter a parameter of the training data set based on determining the data purity and data risk; data processing platform 102 may alter labels of data based on the data purity and/or data risk determination; data processing platform 102 may determine whether the dataset has a threshold imbalance, a threshold skew of an attribute, and/or the like; data processing platform 102 may determine a data quality based on the balance metric and for different types of data (e.g., integer data, categorical data, date data, identifier data, text data, and/or the like); data processing platform 102 may provide information identifying the data quality as output, may automatically select a subset of the training data for use in machine learning to ensure a threshold level of data quality in data selected for the machine learning, and/or the like; data processing platform 102 may automatically identify a data type of a data entry in the training data (e.g., a determination of whether an attribute value in a data entry matches a regular expression); data processing platform 102 may identify categorical data, identifier data, textual data, and/or the like; data processing platform 102 may use an association technique to account for multicollinearity of attributes; data processing platform 102 may label one or more data entries that are missing labels; data processing platform 102 may determine the labels based on a cluster; data processing platform 102 may use an unsupervised machine learning algorithm to enable data cleaning, data set alteration, labeling, and/or the like; data processing platform 102 may automatically adjust missing values, remove skewness of the data (e.g., using logarithmic, normalization, standardization, and/or the like techniques), add labels to the data, filter attributes of the data, remove or alter outliers in the data, and/or the like to improve a data quality of data used for machine learning; data processing platform 102 may use an oversampling technique to generate synthetic examples for the training data set, to avoid under-sampled classes and/or clusters within the training data; ¶ [0036] with FIG. 1: model selection platform 104 may use a local interpretable model agnostic explanation (LIME) model to identify an influence of features on the prediction instances of multiple types of machine learning models; model selection platform 104 may automatically select the features for one or more types of machine learning model; e.g., based on the data curation, model selection platform 104 may identify low risk data entries and associated variables to use as features for training a machine learning model; ¶¶ [0072]-[0073] with 420-430 in FIG. 4: pre-processing the first data to alter the first data to generate second data (block 420); processing the second data to select a set of features from the second data (block 430); ¶¶ [0081]-[0082]: processing the second data includes identifying a plurality of features of the second data, and performing a feature reduction procedure to identify the set of features from the plurality of features of the second data; analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features includes classifying the plurality of types of machine learning models based on a type of problem associated with the first data; ¶¶ [0090]-[0091] with 520-530 in FIG. 5: pre-processing and filtering the input data to generate intermediate data based on receiving the input data (block 520); labeling one or more missing labels in the intermediate data to generate output data based on generating the intermediate data (block 530); ¶ [0099]: identifying the plurality of languages, and training, for the machine learning model, a plurality of sub-models for the plurality of languages; ¶¶ [0103]-[0104]: the cluster analysis identifies a plurality of predicted sources of error in the input data; when pre-processing and filtering the input data, are configured to perform at least one of a space-trimming procedure, a case lowering procedure, a stop words removal procedure, a Boolean logic filtering procedure, or a regular expression pattern matching procedure; ¶¶ [0109]-[0110] with 620-630 in FIG. 6: pre-processing and filtering the first data to generate second data (block 620); processing the second data to select a set of features from a plurality of features of the second data (block 630)).
Response to Arguments
Applicant's arguments filed 01/28/2026 have been fully considered but they are not persuasive.
Applicant argues on Pages 11-15 of the Remarks regarding structure(s) of "input module", "auto-learning module", and "predictive analysis module" that the input module corresponds to the input training data block (108) cooperating with the data points block (114); auto-learning module corresponds to Feature extraction block (102-1), Feature reduction block (102-2), Distance measurement block (102-3), Clustering block (102-4), Sample ranking, selection, arrangement, and reduced dataset creation blocks (102-5 to 102-9); and predictive analysis module corresponds Data processing block (104-6), Data exploration block (104-8), Regression model set (104-10), Classification model set (104-12), Classification model set (104-12), Hyper-parameter tuning block (104-14), and Model saving and tracking block (104-16).
In response, examiner respectfully disagrees. Various functional blocks described in the Remarks are just introducing additional functional language terms, and descriptions in the specification for these functional blocks recite several different well-known mathematical algorithms which do not qualify as a specific structure used in the claimed invention for performing these functional blocks to set the boundary of the claim. Therefore, 102(b) rejections are maintained.
Applicant further argues on Pages 16-34 of the Remarks that the claims are directed to a specific and concrete technological improvement in machine-learning optimization and therefore recite patent-eligible subject matter.
In response, examiner respectfully disagrees. In order for a claim reciting a judicial exception and not directed to the judicial exception, "additional elements" (i.e., non-abstract idea elements) must be integrated with other "judicial exception" elements (i.e., abstract idea elements) in a meaningful way (i.e., not just "apply it") so that an improvement of a practical technology described in the specification is reflected in the claim as a whole. Although Remarks points out the improvement of machine-learning optimization described in the specification, however, they are NOT reflected in the claim when considering both "additional elements" and "judicial exception" elements as whole. For example, Claim 1 recites "identifying a set of relevant training data samples from the input training data" which can be processed in human mind by observing a plurality of training data samples received in the input training data, and discovering a set of relevant training data samples from the plurality of training data samples; "selecting a suitable type and configuration of the model from a plurality of types and configurations of models for processing the set of relevant training data samples" which also can be processed in human mind by observing a plurality of types and configurations of models available and the set of relevant training data samples identified earlier, determining which type and configuration of models are best for processing the set of relevant training data samples identified, and selecting a suitable type and configuration of the model from a plurality of types and configurations of models for processing the set of relevant training data samples identified; and "validating the set of relevant training data samples" which also can be processed in human mind by observing the set of relevant training data samples identified to make sure that the set of relevant training data samples are correct (e.g., correct labeling) or proper for a task to be performed. The addition element "receiving input training data comprising a plurality of training data samples" is just a well-known routine of collecting data. These four processing steps (i.e., three mentor capable process steps and one well-known routine of collecting data) are processed by generic "optimization device" (i.e., "apply it"). Considering the claim as a whole, there is no way to even see how these four steps without performing any training or learning steps can achieve "optimizing training of a machine learning model" recited in the preamble. Therefore, these four processing steps cannot demonstrate the improvement in machine-learning optimization technology comparing to common machine-learning optimization techniques at all. In other words, the claims are NOT directed to a specific and concrete technological improvement in machine-learning optimization and do not recite patent-eligible subject matter.
Applicant further argues on Pages 35-37 of the Remarks that cited references fail to teach, disclose or suggest "… identifying, by the optimization device, a set of relevant training data samples from the input training data …".
In response, examiner respectfully disagrees. HIGGINS discloses in ABSTRACT; ¶¶ [0003]-[0005]; ¶¶ [0071]-[0076] and [0083]-[0085] with 410-460 in FIG. 4; ¶¶ [0089]-[0093] with 510-550 in FIG. 5; and ¶¶ [0108]-[0112] with 610-650 in FIG. 6 that (1) obtaining first data relating to a machine learning model, pre-processing the first data to alter the first data to generate second data, processing the second data to select a set of features from the second data, analyzing the set of features to evaluate a plurality of types of machine learning models with respect to the set of features (includes automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models), selecting a particular type of machine learning model, of the plurality of types of machine learning models, for the set of features based on analyzing the set of features to evaluate the plurality of types of machine learning models, tuning a set of parameters of the particular type of machine learning model to train the machine learning model (includes optimizing hyper parameters of the particular type of machine learning model, and retraining the machine learning model based on optimizing the hyper parameters of the machine learning model); (2) receive input data from a data source, preprocess and filter the input data to generate intermediate data based on receiving the input data, label one or more missing labels in the intermediate data to generate output data based on generating the intermediate data, select, for the output data, a machine learning model, of a plurality of types of machine learning models, to apply to the output data, tune a set of hyper-parameters for the machine learning model based on selecting the machine learning model, establish a model pipeline for the machine learning model based on tuning the set of hyper-parameters; and (3) obtain first data relating to a natural language processing or image processing task, pre-process and filter the first data to generate second data, process the second data to select a set of features from a plurality of features of the second data, apply the set of features to a plurality of types of machine learning models to evaluate the plurality of types of machine learning models with respect to completing the natural language processing or image processing task, select a particular type of machine learning model, from the plurality of types of machine learning models, for the set of features based on applying the set of features to the plurality of types of machine learning models. HIGGINS further discloses in ¶¶ [0015]-[0030] with FIG. 1A that (1) identifying characteristics of the training data; (2) obtain contextual information relating to the training data, such as information identifying input fields, output fields, a clustering parameter for clustering-based pre-processing, and/or the like; (3) apply one or more training data modification filtering rules to perform rule-based pre-processing; e.g., apply one or more rules regarding a type of data that is to be received as training data; (4) use one or more regular expressions for filtering of the training data; e.g., determine, based on a regular expression, that a data entry is of a particular type that is to be removed from use in training a machine learning model, such as user identification information, confidential information, and/or the like; (5) perform a cluster analysis on the training data to identify outlier data based on a set of clusters generated for automatically removing or altering the outlier data; (6) apply labels to clusters of data entries based on generating a set of clusters of data entries; (7) enable filtering of labeled data, chaining of filtered labeled data, and/or the like to enable data curation; (8) determine that a data set within the training data with a relatively high data purity may be classified as a low risk data set, whereas a data set with a relatively low data purity may be classified as a high risk data set so that enable selection of a low risk portion of the data set for training one or more models, resulting in a higher accuracy of the one or more models; (8) determine a balance metric for a dataset; e.g., determine whether the dataset has a threshold imbalance, a threshold skew of an attribute, and/or the like so that determine a data quality based on the balance metric and for different types of data (e.g., integer data, categorical data, date data, identifier data, text data, and/or the like); (9) automatically identify a data type of a data entry in the training data; e.g., identify categorical data, identifier data, textual data, and/or the like; (9) use an association technique to account for multicollinearity of attributes; e.g., measure a degree of a relationship between linearly related variables; determine an extent of a non-linearity of a relationship between variables; and (10) a similarity score calculated based on training data parameters. In other words, HIGGINS teaches identifying a set of types of training data samples related to machine learning tasks to be performed and optimized, and filtering out any non-relevant data (e.g., outlier data). Therefore, HIGGINS INDEED discloses "… identifying, by the optimization device, a set of relevant training data samples from the input training data …" as recited in the claim.
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., "reduce the dataset to only necessary or representative examples", "preserve variability or diversity in a selected subset", "evaluate sample importance for training efficiency, rank or select data points for representational coverage", and "perform sample extraction driven by how a machine learning model learns") are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Applicant further argues on Pages 37-39 of the Remarks that cited references fail to teach, disclose or suggest "… selecting, by the optimization device, a suitable type and configuration of a machine learning model from a plurality of types and configurations of machine learning models for processing the set of relevant training data samples …".
In response, examiner respectfully disagrees. As discussed above, HIGGINS discloses in ABSTRACT; ¶¶ [0003]-[0005]; ¶¶ [0071]-[0076] and [0083]-[0085] with 410-460 in FIG. 4; ¶¶ [0089]-[0093] with 510-550 in FIG. 5; and ¶¶ [0108]-[0112] with 610-650 in FIG. 6 that analyze the set of features selected in the training data related to machine learning tasks to be performed and optimized for evaluating a plurality of types of machine learning models with respect to the set of features selected (includes automatically optimizing hyper parameters of the plurality of types of machine learning models to attempt to optimize the plurality of types of machine learning models), then select a particular type of machine learning model from the plurality of types of machine learning models for the set of features selected based on analyzing results; and finally tune a set of parameters of the particular type of machine learning model selected to train the machine learning model (includes optimizing hyper parameters of the particular type of machine learning model, and retraining the machine learning model based on optimizing the hyper parameters of the machine learning model). HIGGINS further discloses in ¶¶ [0030]-[0039] with FIGS. 1A-1B and ¶¶ [0117]-[0119] that (1) obtain algorithms for a set of types of machine learning models (156) from model repository 110; (2) automatically select, train, and deploy a particular machine learning model of the set of possible machine learning models (158); e.g., classify a task type, select a machine learning model category for the task type, select a machine learning model type from the model category, tune parameters of the machine learning model, and train the machine learning model; (3) determine one or more parameters relating to deploying a particular type of machine learning model (160); e.g., determine that the training data relates to a classification task, a regression task, and/or the like, and may filter the set of types of machine learning models based on the type of task; (4) define one or more use cases for a machine learning model, define one or more inputs or outputs for the machine learning model (i.e., configuration of the ML model), and/or the like to enable selection of a machine learning model; (5) use a local interpretable model agnostic explanation (LIME) model to identify an influence of features on the prediction instances of multiple types of machine learning models; (6) automatically select the features for one or more types of machine learning model; e.g., based on the data curation, identify low risk data entries and associated variables to use as features for training a machine learning model; (7) tune hyper parameters (i.e., configuration) in connection with selection of a machine learning model; e.g., evaluate a set of types of machine learning models, and select a particular type of machine learning model with a greatest quantity of optimal hyper parameters (i.e., configuration) so that use hyper parameter optimization to select a machine learning model; (8) when a particular type of machine learning model is selected (e.g., using another technique as described herein), automatically tune hyper parameters (i.e., configuration) of the particular machine learning model to enable the particular type of machine learning model to perform predictions using subsequent prediction data; (9) to identify best estimated hyper parameters of multiple types of machine learning models, use a tree-structured Parzen estimator to identify best estimated hyper parameters within a high dimensional search space of parameters of the multiple types of machine learning models; and (10) determine a score (e.g., based on tuning hyper parameters), and may select a particular type of machine learning model based on the score. In other words, HIGGINS teaches selecting a particular type of machine learning model proper for training data relates to a particular type of task to be trained and preformed, and automatically tune hyper parameters and identify best hyper parameters of the particular machine learning model for the particular type of task. Therefore, HIGGINS INDEED teaches . "… selecting, by the optimization device, a suitable type and configuration of a machine learning model from a plurality of types and configurations of machine learning models for processing the set of relevant training data samples …" as recited in the claim.
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., "selects a model to use curated data efficiently", "reduce dataset to only the most informative sample", "perform selection driven by cognitive feedback of errors", "ensure new model replaces old only upon superior validation performance", "selects a model to use less data more efficiently", "selecting samples that offer maximum variability with faster and better accuracy", "trains and tests each type and configuration of ML model type and configuration pair by executing it against the identified relevant samples") are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Applicant further argues on Pages 39-41 of the Remarks that cited references fail to teach, disclose or suggest "… validating the set of relevant training data samples …".
In response, examiner respectfully disagrees. As discussed above, HIGGINS discloses "identifying a set of relevant training data samples" for training data set and only failed to disclose validating the set of relevant training data samples identified/obtained. Karumanchi teaches this missing limitation in ¶¶ [0048]-[0049] and [0071]-[0072] that (1) train an ML model and obtain a classification accuracy for the ML model on a held-out test data set or using k-fold cross validation on the obtained training data set, and compares the classification accuracy with an accuracy from a previous sampling of the data; and (2) predict data relevance, such as business data relevance (e.g., classified/unclassified, secure/unsecure, sensitive/non-sensitive, etc.) by learning from the sampled training data to predict different categories (e.g., classified, unclassified; private, public, etc.). Therefore, the combination of HIGGINS and Karumanchi teaches "… validating the set of relevant training data samples …" as recited in the claim.
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., "verify that each sample is genuinely relevant to the intended learning task prior to its inclusion in the training dataset", "validate the relevance of training samples", "cognitive reasoning") are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HWEI-MIN LU whose telephone number is (313)446-4913. The examiner can normally be reached Mon - Fri: 9:00 AM - 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D. Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HWEI-MIN LU/Primary Examiner, Art Unit 2142
Prosecution Insights