DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, 3, 5, 6, 8, 10, 12, 13, 15, 17, and 19-21 were amended.
Claims 2, 9, 16, and 22 were cancelled.
Claim 23 is new.
Claims 1, 3-8, 10-15, 17-21, and 23 are pending and examined herein.
Claims 1, 3-8, 10-15, 17-21, and 23 are rejected under 35 U.S.C. 103.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/27/2026 has been entered.
Response to Arguments
Applicant’s arguments, see page 7, filed 02/27/2026, with respect to the objection to claims 3, 9, and 17 have been fully considered and are persuasive. The objection to claims 3, 9, and 17 has been withdrawn.
Applicant’s arguments, see page 7, filed 02/27/2026, with respect to the 35 U.S.C. 112(b) rejections of claims 21 and 22 have been fully considered and are persuasive. The 35 U.S.C. 112(b) rejection of claims 21 and 22 has been withdrawn.
Applicant’s arguments, see pages 7-9, filed 02/27/2026, with respect to the 35 U.S.C. 103 rejection of claims 1, 3-8, 10-15, and 17-21 have been fully considered and are persuasive. The 35 U.S.C. 103 rejection of claims 1, 3-8, 10-15, and 17-21 has been withdrawn. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and Bhati (“Understanding Spark Partitioning”, 2015).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 3, 8, 10, 15 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and Bhati (“Understanding Spark Partitioning”, 2015).
Regarding claim 1, Campos teaches
A computer-implemented method comprising: (The abstract states "Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating and comparing multiple trained machine learning models." [0098] states "FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input/output device 440.")
receiving, by one or more processors, a set of training data samples, a set of labels, and … for training a classification-based machine learning model; and ([0071] states "The evaluation apparatus 160 obtains multiple training data samples (210). For example, the validation system 140 obtains a training dataset to train multiple machine learning models to generate a predictive value depending on the problem being solved using the machine learning model. In the example of the ecommerce webpage 135, the validation system 140 obtains a training dataset 142 that includes multiple training samples where each sample includes a search query, a textual description of an item and a label indicating whether the item is relevant to the search query." The search queries/textual descriptions of items are interpreted as the training data samples, and the labels are interpreted as the set of labels. [0029] – [0030] state "Such a classification model can be selected after evaluation of multiple machine learning models that are trained to perform the same task and subsequent selection of the model that performs better relative to the other evaluated models. To evaluate the multiple machine learning models, the techniques described in this specification generate, using the models under evaluations, predictions based on a subset of samples from a training dataset." Therefore, the training data and labels are used to train a classification-based machine learning model. [0108] states "The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output." Hereinafter, this is considered to be the explanation for the “by the one or more processors” limitations.)
deploying, by the one or more processors, a trained classification-based machine learning model of a plurality of classification-based machine learning models based at least in part on a performance definition set by: ([0067] states "In some implementations, after selecting a particular machine learning model by the evaluation apparatus 160, the validation system 140 can deploy the selected machine learning model for the specific task for which the particular machine learning model (as well as other multiple machine learning models that were evaluated by the evaluation apparatus 160) was trained. For example, the validation system 140 deploys the first machine learning model 146A to classify each item in the list of items selected by the search system 150 in response to the buyer submitting a search query as being ‘relevant’ or ‘irrelevant’." Therefore, the machine learning models are classification-based machine learning models. [0088] states "After determining the performance of the multiple machine learning models based on the difference in the model performance metrics such as precision, recall and FPR, the evaluation apparatus 160 can select based on a pre-specified threshold, one or more machine learning models from multiple machine learning models generated by the validation system" The performance metrics are interpreted as the performance definition set. [0089] states "After selecting a machine learning model by the evaluation apparatus 160, the validation system 140 can deploy the selected machine learning model for the specific problem for which the multiple machine learning models were trained.")
(i) storing the set of training data samples as a plurality of training data partitions, ([0053] states "In some implementations, to evaluate the multiple machine learning models, the evaluation apparatus 160 can select a proper subset of the training dataset 142" The subset is interpreted as the data partition, which, when identified, is stored as a plurality of training data partitions (the selected subset and the unselected subset).) wherein (a) the plurality of training data partitions have a uniform distribution of the set of labels associated with the plurality of classification-based machine learning models, ([0053] states "The proper subset of the training dataset can be selected, for example, using random sampling, stratified sampling, etc., of the training dataset. In some implementations, after selecting the proper subset of the training dataset, the subset can be evaluated to ensure that the subset is representative of the training dataset. Representativeness can be assessed in various ways such as, e.g., the ratio of labels or other attributes in the training dataset. In some implementations, a proper subset of training samples is representative of the training dataset 142 when the proportional distribution of samples across labels in the proper subset is the same as the proportion distribution of samples across labels in the entire training dataset 142.")
(ii) training one or more of the plurality of classification-based machine learning models based at least in part on the plurality of training data partition, and ([0083] states "The evaluation apparatus 160 generates a predicted value for the target attribute using a first machine learning model (230). For example, to evaluate the first machine learning model 146A and the second machine learning model 146B generated by the validation system 140, the evaluation apparatus 160 can use the first machine learning model 146A to generate predicted labels for each data sample in the subset of the training dataset. Similarly, the evaluation apparatus 160 can use the second machine learning model 146B to generate predicted labels for each sample in the subset of the training dataset (240)." Therefore, the two models are trained using the partitioned data.)
(iii) deploying the trained classification-based machine learning model based at least in part on a performance of the trained classification-based machine learning model relative to the plurality of classification-based machine learning models with respect to the performance definition set. ([0088} states "After determining the performance of the multiple machine learning models based on the difference in the model performance metrics such as precision, recall and FPR, the evaluation apparatus 160 can select based on a pre-specified threshold, one or more machine learning models from multiple machine learning models generated by the validation system." The model performance metrics are interpreted as the performance definition set. As multiple models are compared, the performance is relative. [0089] states "After selecting a machine learning model by the evaluation apparatus 160, the validation system 140 can deploy the selected machine learning model for the specific problem for which the multiple machine learning models were trained.")
Campos does not appear to explicitly teach
[receiving] a classification type
(b) a number of the plurality of training data partitions is based at least in part on a minimal number of allowed partitions, a maximal number of allowed partitions, and a maximum sequence length for the plurality of classification-based machine learning models, (c) the minimal number of allowed partitions and the maximal number of allowed partitions are user configurable parameters that are selected based at least in part on a total number of the set of training data samples,
However, TransmogrifAI—directed to analogous art—teaches
[receiving] a classification type (Page 3 states "TransmogrifAI will select the best model and hyper-parameters for you based on the class of modeling you are doing (eg. Classification, Regression etc.)." Page 3 further states "The ModelSelector is an Estimator that uses data to find the best model. BinaryClassificationModelSelector is for binary classification tasks, multi classification tasks can be done using MultiClassificationModelSelector. Best Regression model are done through RegressionModelSelector.” As the user inputs the class of modeling (classification type), the system receives the classification type.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos with the teachings of TransmogrifAI because as TransmogrifAI states on page 3, "Smart model selection and comparison gives the next layer of improvements over traditional ML workflows."
The combination of Campos and TransmogrifAI does not appear to explicitly teach
(b) a number of the plurality of training data partitions is based at least in part on a minimal number of allowed partitions, a maximal number of allowed partitions, and a maximum sequence length for the plurality of classification-based machine learning models, (c) the minimal number of allowed partitions and the maximal number of allowed partitions are user configurable parameters that are selected based at least in part on a total number of the set of training data samples,
However, Bhati—directed to analogous art—teaches
(b) a number of the plurality of training data partitions is based at least in part on a minimal number of allowed partitions, a maximal number of allowed partition … (c) the minimal number of allowed partitions and the maximal number of allowed partitions are user configurable parameters (Column 10, lines 27-32 state "In some embodiments, the configuration interface 154 may also be used to configure the behavior of the repartitioner 156. For example, in some embodiments, the configuration interface may be used to specify the minimum and maximum number of partitions that can be created for each user-specified partition." Column 9, lines 57-64 state "As shown, in some embodiments, the data store 130 may implement a configuration interface 154 for the partition manager 150. Depending on the embodiment, the configuration interface 154 may be a graphical user interface (GUI), 60 a web interface, an application programming interface (API) such as a web services interface, or a combination thereof. In some embodiments, the configuration interface 154 may be exposed as part of a set of administrator interfaces to the data store 130." Therefore, the minimum and maximum number of partitions are user configurable.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos and TransmogrifAI with the teachings of Golding because, as stated by Golding in columns 9 and 10, "In some embodiments, the configuration 65 interface 154 may be provided to individual clients, which may allow the client to configure their own partitioning requirements, policies, etc."
The combination of Campos, TransmogrifAI, and Golding does not appear to explicitly teach
[a number of data partitions is based on] a maximum sequence length for the plurality of classification-based machine learning models
[a minimum and maximum number of partitions] that are selected based at least in part on a total number of the set of training data samples
However, Silfa—directed to analogous art—teaches
[a number of data partitions is based on] a maximum sequence length for the plurality of classification-based machine learning models (Page 2 states "Furthermore, to meet Service-Level-Agreement (SLA), we limit the maximum number of time-steps in a given processing lane." The time-steps are interpreted as the sequence length, as Page 2 states "In an RNN, an input sequence (e.g. an audio frame) is composed of N time-steps, i.e.,
X
=
[
x
1
,
x
2
,
.
.
.
,
x
N
]
. In an LSTM or GRU network, time-steps are processed sequentially, from
x
1
to
x
n
." Page 7 states " In E-Batch, the number of time-steps processed by a given lane is limited to a threshold (i.e., N). " Therefore, as the requests are limited by the maximum sequence length, the number of partitions will be affected based on the maximum sequence length.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, and Golding with the sequence length taught by Silfa because, as Silfa states on page 7, "By using a threshold, we can trade latency for energy consumption. Batching with a small number of time-steps per lane decreases latency but incurs in a large amount of weight swaps, which decreases weight locality. On the contrary, batching with a large number of time-steps per lane increases weight reuse, which significantly reduces energy consumption at the expense of an increase in latency."
The combination of Campos, TransmogrifAI, Golding, and Silfa does not appear to explicitly teach
[a minimum and maximum number of partitions] that are selected based at least in part on a total number of the set of training data samples
However, Bhati—directed to analogous art—teaches
[a minimum and maximum number of partitions] that are selected based at least in part on a total number of the set of training data samples (Page 2, ‘Factors affecting partitioning’ states "External Data Sources-Size of Local collections, Cassandra table or HDFS file determine number of partitions." The external data source is interpreted as the set of training data samples. As the size of the data is determined by the number of samples, the number of partitions is selected based on a total number of the set of training data samples. The recommended guideline on page 4 states "Usually between 100 and lOK partitions depending upon cluster size and data. Lower bound-2 X number of cores in cluster available to application Upper bound-task should take 100+ ms time to execute. If it is taking less time than your partitioned data is too small and your application might be spending more time in scheduling the tasks." As the size also determines the number of partitions, the decision of the maximum (upper bound) and minimum (lower bound) is also based on the number of training data samples.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, and Silfa with the teachings of Bhati because, as Bhati states on page 3, "Having too few and too large number of partitions has certain advantages and disadvantages. So it is recommended to partition judiciously depending upon your cluster configuration and requirements."
Regarding claim 3, the rejection of claim 1 is incorporated herein. Campos does not appear to explicitly teach
wherein the set of training data samples are preprocessed using a rule-based framework.
However, TransmogrifAI—directed to analogous art—teaches
wherein the set of training data samples are preprocessed using a rule-based framework. (Page 1 states “The TransmogifAI tranmogrifier (shortcut .transmogrify()) tales in a sequence of features, automatically applies default transformations to them based on feature types (e.g. imputation, null value tracking, one hot encoding, tokenization, split Emails and pivot out the top K domains) and combines them into a single vector.” The transformations are interpreted as the rule-based framework.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos and the transformations of TransmogrifAI because, as stated by TransmogrifAI on page 1, “This is the Stage that automates the feature engineering step in the machine learning pipeline.”
Regarding claim 8, Campos teaches
A system comprising: ([0098] states "FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above.”)
one or more processors; and ([0098] states "FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input/output device 440.”)
one or more memories storing processor executable instructions that, when executed by the one or more processors to perform operations comprising: ([0098] states "FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input/output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In one implementation, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi-threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430.")
The remainder of claim 8 recites substantially similar subject matter to claim 1 and is rejected with the same rationale, mutatis mutandis.
Claim 10 recites substantially similar subject matter to claim 3 and is rejected with the same rationale, mutatis mutandis.
Regarding claim 15, Campos teaches
One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: ([0098] states "FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input/output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In one implementation, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi-threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430.")
The remainder of claim 15 recites substantially similar subject matter to claim 1 and is rejected with the same rationale, mutatis mutandis.
Claim 17 recites substantially similar subject matter to claim 3 and is rejected with the same rationale, mutatis mutandis.
Regarding claim 23, the rejection of claim 1 is incorporated herein. Campos teaches
train the one or more of the plurality of classification-based machine learning models. ([0083] states "The evaluation apparatus 160 generates a predicted value for the target attribute using a first machine learning model (230). For example, to evaluate the first machine learning model 146A and the second machine learning model 146B generated by the validation system 140, the evaluation apparatus 160 can use the first machine learning model 146A to generate predicted labels for each data sample in the subset of the training dataset. Similarly, the evaluation apparatus 160 can use the second machine learning model 146B to generate predicted labels for each sample in the subset of the training dataset (240)." Therefore, the two models are trained using the partitioned data.)
The combination of Campos, TransmogrifAI, and Golding does not appear to explicitly teach
wherein the minimal number of allowed partitions and the maximal number of allowed partitions are based at least in part on a hardware used to [execute an application]
However, Golding—directed to analogous art—teaches
wherein the minimal number of allowed partitions and the maximal number of allowed partitions are based at least in part on a hardware used to [execute an application] (The guidelines for choosing the number of partitions on page 4 state "Lower bound-2X number of cores in cluster available to application Upper bound-task should take 100+ ms time to execute. If it is taking less time than your partitioned data is too small and your application might be spending more time in scheduling the tasks." The number of cores is hardware for executing the application, and the execution time is determined by the hardware for executing the application. Therefore, the minimum (lower bound) and maximum (upper bound) is based on the hardware used to execute the application.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, and Silfa with the teachings of Bhati for the reasons given above in regards to claim 1.
Claim(s) 4, 11, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and Bhati (“Understanding Spark Partitioning”, 2015) as applied to claim 1 above, further in view of D2L-AI (“Training on Multiple GPUs”, April 2022).
Regarding claim 4, the rejection of claim 1 is incorporated herein. Campos teaches
wherein each of the plurality of the classification-based machine learning models … and (ii) comprises a different model architecture ([0067] states "In some implementations, after selecting a particular machine learning model by the evaluation apparatus 160, the validation system 140 can deploy the selected machine learning model for the specific task for which the particular machine learning model (as well as other multiple machine learning models that were evaluated by the evaluation apparatus 160) was trained. For example, the validation system 140 deploys the first machine learning model 146A to classify each item in the list of items selected by the search system 150 in response to the buyer submitting a search query as being ‘relevant’ or ‘irrelevant’." Therefore, the machine learning models are classification-based machine learning models. [0049] states "The validation system 140 can also include a model configurator 148 that can generate and configure various machine learning models based on the machine learning model properties such as the type of machine learning model, the number of parameters of the machine learning model, the optimization techniques etc. For example, if the machine learning model is a neural network, the model configurator 148 can set the number of neural network layers, the number of neurons per layer, activation function, the number of training iterations of the training process, etc.")
The combination of Campos, TransmogrifAI, Golding, Silfa, and Bhati does not appear to explicitly teach
(i) is trained using a separate graphics processing unit (GPU)
However, D2L-AI—directed to analogous art—teaches
wherein each of the plurality of [machine learning models] is training using a separate graphics processing unit (GPU). (Page 2 states “Last, we could partition data across multiple GPUs. This way all GPUs perform the same type of work, albeit on different observations.” As can be seen in the figure on pages 3, the data parallel method copies the model, meaning that there are a plurality of models, onto each GPU, and they are trained using a subset of the data.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, Silfa, and Bhati with the teachings of D2L-AI because, as D2L teaches on page 3, “By and large, data parallelism is the most convenient way to proceed, provided we have access to GPUs with sufficiently large memory.”
Claims 11 and 18 recite substantially similar subject matter to claim 4 and are rejected with the same rationale, mutatis mutandis.
Claim(s) 5, 12, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and Bhati (“Understanding Spark Partitioning”, 2015) as applied to claim 1 above, further in view of Inamdar (US 2021/0097472 A1).
Regarding claim 5, the rejection of claim 1 is incorporated herein. The combination of Campos, TransmogrifAI, Golding, Silfa, and Bhati does not appear to explicitly teach
wherein each training data sample of the set of training data samples comprises an input-vector based representation of an input document.
However, Inamdar—directed to analogous art—teaches
wherein each training data sample of the set of training data samples comprises an input-vector based representation of an input document. (Figure 6, Step 602 is “generate candidate vectors from a plurality of candidate documents”, Step 608 is “generate a feature vector for each of the subset of candidate documents”, and Step 610 is “apply a feature vector to generate a score for each of the subset of the plurality of candidate documents”. [0056] states "Specifically, the feature extractor 145 can be programmed to determine, for each of a plurality of candidate documents, a feature vector representing the candidate document and an objective document, for example, a candidate search parameter vector as described in FIG. 1." Additionally, [0074] states, in reference to training an artificial neural network, "The values from the feature vector are provided to a plurality of input nodes." As the feature value vector is provided as input, it is interpreted as input-vector based, and the feature vector is interpreted as the training sample.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, Silfa, and Bhati with the teachings of Inamdar because, as stated by Inamdar in [0019], "Systems and methods are described herein that overcome existing cold start problems associated with candidate recommendation systems to provide a technical solution that enables candidate recommendation systems to predict or identify the most qualified candidates ( e.g., resumes, jobs, etc. ) upon these systems been deployed at an organization or until an ML model has been sufficiently trained based on industry relevant training data. Examples are provided herein in context of resume-to-job recommendation, however, this disclosure should not be construed or limited to only encompass candidate matching, and can include, in other examples, different recommendation applications, such as employee matching (e.g., recommending employees for projects, functions, etc.), document matching, item matching, movie matching, song matching, consumer matching, etc."
Claims 12 and 19 recite substantially similar subject matter to claim 5 and are rejected with the same rationale, mutatis mutandis.
Claim(s) 6, 7, 13, 14, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and Bhati (“Understanding Spark Partitioning”, 2015) as applied to claim 1 above, further in view of Plumbley (US 2021/0117869 A1).
Regarding claim 6, the rejection of claim 1 is incorporated herein. Campos teaches
the classification-based machine learning models ([0067] states "In some implementations, after selecting a particular machine learning model by the evaluation apparatus 160, the validation system 140 can deploy the selected machine learning model for the specific task for which the particular machine learning model (as well as other multiple machine learning models that were evaluated by the evaluation apparatus 160) was trained. For example, the validation system 140 deploys the first machine learning model 146A to classify each item in the list of items selected by the search system 150 in response to the buyer submitting a search query as being ‘relevant’ or ‘irrelevant’." Therefore, the machine learning models are classification-based machine learning models.)
The combination of Campos, TransmogrifAI, Golding, Silfa, and Bhati does not appear to explicitly teach
However, Plumbly—directed to analogous art—teaches
determining an ordered sequence of the plurality of [machine learning models] according to the performance definition set. ([0114] states "In another example, all models may be ranked according to their MPSs [model performance statistics] in which the best performing K models or topmost performing K models are selected for inclusion into the set of optimal trained models. The set of optimal trained models may be stored in a model database 232 for use in forming one or more ensemble models." The model performance statistics is interpreted as the performance definition set. One of ordinary skill in the art would realize that an ordered sequence is a ranking.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, Silfa, and Bhati with the performance evaluation taught by Plumbly because, as stated by Plumbly in [0077] "The inventors have advantageously developed a system for generating and selecting from a large number of trained models, or a plurality of sets of trained models, with the same or similar objectives a subset of the best performing trained models that can be used to create one or more ensemble model(s) that have been optimised for modelling a process or problem associated with one or more compounds."
Regarding claim 7, the rejection of claim 6 is incorporated herein. Campos teaches
wherein the performance definition set include a relative cost of precision and recall. ([0066] states "In some implementations, after determining the performance of the multiple machine learning models based on the difference in one or more model performance metrics, such as precision, recall and FPR, using CI and p-value, the evaluation apparatus 160 can select based on a pre-specified threshold, a machine learning models from multiple machine learning models generated by the validation system." The model performance metrics are interpreted as the performance definition set. As the difference between the metrics of the models is calculated, the precision and recall are relative.)
Claims 13 and 14 recite substantially similar subject matter to claims 6 and 7 respectively and are rejected with the same rationale, mutatis mutandis.
Claim 20 recites substantially similar subject matter to claim 6 and is rejected with the same rationale, mutatis mutandis.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and Bhati (“Understanding Spark Partitioning”, 2015) as applied to claim 1 above, further in view of Crankshaw (“Clipper: A Low-Latency Online Prediction Serving System”, 2017).
Regarding claim 21, the rejection of claim 1 is incorporated herein. Campos teaches
the labeled data and ([0071] states "The evaluation apparatus 160 obtains multiple training data samples (210). For example, the validation system 140 obtains a training dataset to train multiple machine learning models to generate a predictive value depending on the problem being solved using the machine learning model. In the example of the ecommerce webpage 135, the validation system 140 obtains a training dataset 142 that includes multiple training samples where each sample includes a search query, a textual description of an item and a label indicating whether the item is relevant to the search query." The search queries/textual descriptions of items are interpreted as the training data samples, and the labels are interpreted as the set of labels.)
Campos does not appear to explicitly teach
wherein … the classification type are received via a public application programming interface (API) and the particular machine learning model is deployed via a private API.
However, TransmogrifAI—directed to analogous art—teaches
the classification type (Page 3 states "TransmogrifAI will select the best model and hyper-parameters for you based on the class of modeling you are doing (eg. Classification, Regression etc.)." Page 3 further states "The ModelSelector is an Estimator that uses data to find the best model. BinaryClassificationModelSelector is for binary classification tasks, multi classification tasks can be done using MultiClassificationModelSelector. Best Regression model are done through RegressionModelSelector.” As the user inputs the class of modeling (classification type), the system receives the classification type.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos with the teachings of TransmogrifAI because as TransmogrifAI states on page 3, "Smart model selection and comparison gives the next layer of improvements over traditional ML workflows."
The combination of Campos, TransmogrifAI, Golding, Silfa, and Bhati does not appear to explicitly teach
wherein the [queries] are received via a public application programming interface (API) and the particular machine learning model is deployed via a private API.
However, Crankshaw—directed to analogous art—teaches
wherein the [queries] are received via a public application programming interface (API) and the particular machine learning model is deployed via a private API. (Fig. 1 shows the application facing API (interpreted as the public API) and the RPC (interpreted as the private API, as it is not available to the applications) between the model containers and Clipper. Page 614 states "Applications issue prediction requests to Clipper through application facing REST or RPC APIs. Prediction requests are first processed by the model selection layer. Based on properties of the prediction request and recent feedback, the model selection layer dispatches the prediction request to one or more of the models through the model abstraction layer." Therefore, the application facing API receives the requests. Page 617 states "Models deployed in Clipper are each encapsulated within their own lightweight container (§4.4), communicating with Clipper through an RPC mechanism that provides a uniform interface to Clipper and simplifies the deployment of new models." Therefore, the machine learning models are deployed via the private API.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, Silfa, and Bhati with the APIs of Crankshaw because, as Crankshaw states on page 617, "The lightweight RPC system minimizes the overhead of the container-based architecture and simplifies cross-language integration." Crankshaw also states on page 620, "By implementing model selection policies that provide different cost-accuracy tradeoffs, as well as an API for users to implement their own policies, Clipper provides a mechanism to easily navigate the tradeoffs between accuracy and computational cost on a per-application basis."
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JESSICA THUY PHAM whose telephone number is (571)272-2605. The examiner can normally be reached Monday - Friday, 9 A.M. - 5:00 P.M..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.P./Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights