DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, 8, 15 and 23 were amended.
Claim 24 is new.
Claims 1, 3-8, 10-15, 17-21, and 23-24 are pending and examined herein.
Claims 1, 3-8, 10-15, 17-21, and 23-24 are rejected under 35 U.S.C. 101.
Claims 1, 3-8, 10-15, 17-21, and 23 are rejected under 35 U.S.C. 103.
Response to Arguments
Applicant’s arguments, see pages 7-9, filed 02/27/2026, with respect to the 35 U.S.C. 103 rejection of claims 1, 3-8, 10-15, and 17-21 have been fully considered and are persuasive. The 35 U.S.C. 103 rejection of claims 1, 3-8, 10-15, and 17-21 has been withdrawn. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and MicroStrategy (“MicroStrategy 10 In-memory Analytics Guide,” 2017).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-8, 10-15, 17-21, and 23-24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
MPEP § 2109(III) sets out steps for evaluating whether a claim is drawn to patent-eligible subject
matter. The analysis of claims 1, 3-8, 10-15, 17-21, and 23-24, in accordance with these steps, follows.
Step 1 Analysis:
Step 1 is to determine whether the claim is directed to a statutory category (process, machine,
manufacture, or composition of matter. Claims 1, 3-7, 21, and 23-24 are directed to a process, claims 8 and 10-14 are directed to a machine, and claims 15 and 17-20 are directed to an article of manufacture. All claims are directed to statutory categories and analysis proceeds.
Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis:
Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101.
None of the claims represent an improvement to technology.
Regarding claim 1, the following are abstract ideas:
(i) selecting a minimal number of allowed training data partitions and a maximal number of allowed training data partitions based at least in part on a total number of the set of training data samples, wherein (a) the minimal number of allowed training data partitions and the maximal number of allowed training data partitions are user configurable parameters and (b) the maximal number of allowed training data partitions is equal to a number of available cores of the one or more processors used to train the plurality of classification-based machine learning models, (Selecting a minimal and maximal number of allowed training data partitions can be practically performed in the human mind. This is a mental process.)
(ii) determining a number of training data partitions based at least in part on the minimal number of allowed training data partitions, the maximal number of allowed training data partitions, and a maximum sequence length for the plurality of classification-based machine learning models, (Selecting a number of allowed training data partitions can be practically performed in the human mind. This is a mental process.)
The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
A computer-implemented method comprising: (This recites a generic computer performing the method; this amounts to mere instructions to apply an exception.)
receiving, by one or more processors, a set of training data samples, a set of labels, and a classification type for training a classification-based machine learning model; and (Receiving data is an existing process on computers; this amounts to mere instructions to apply an exception.)
deploying, by the one or more processors, a trained classification-based machine learning model of a plurality of classification-based machine learning models based at least in part on a performance definition set by: (Deploying machine learning models is an existing process in machine learning process, and the process is recited at a high-level of generality; this amounts to mere instructions to apply an exception.)
(iii) storing the set of training data samples as a plurality of training data partitions having the determined number of training data partitions, wherein the plurality of training data partitions have a uniform distribution of the set of labels associated with the plurality of classification-based machine learning models, (Storing partitioned data is a existing and generic process in computers. This amounts to mere instructions to apply an exception.)
(iv) training one or more of the plurality of classification-based machine learning models based at least in part on the plurality of training data partitions, and (Training machine learning models is an existing process in machine learning process, and the process is recited at a high-level of generality; this amounts to mere instructions to apply an exception.)
(v) deploying the trained classification-based machine learning model based at least in part on a performance of the trained classification-based machine learning models with respect to the performance definition set. (Deploying machine learning models is an existing process in machine learning process, and the process is recited at a high-level of generality; this amounts to mere instructions to apply an exception.)
Regarding claim 3, the rejection of claim 1 is incorporated herein. The following is an abstract idea:
wherein the set of training data samples is pre-processed using a rule-based framework. (Pre-processing training data samples using a rule-based framework can be practically performed in the human mind, i.e. removing duplicates.)
Claim 3 does not recite any further additional elements.
Regarding claim 4, the rejection of claim 1 is incorporated herein. The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein each of the plurality of classification-based machine learning models (i) is trained using a separate graphics processing unit (GPU) and (ii) comprises a different model architecture. (Training using separate GPUs is the generic machine learning process of data-parallel training. Using multiple different architectures for machine learning models is the generic machine learning process of ensemble learning. This amounts to mere instructions to apply an exception.)
Regarding claim 5, the rejection of claim 1 is incorporated herein. The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein a training data sample of the set of training data samples comprises an input-vector based representation of an input document. (This is the insignificant extra-solution activity of selecting a particular data source or type of data to be manipulated. See MPEP § 2106.05(g), ‘Selecting a particular data source or type of data to be manipulated’, ex. i. – iv.)
Regarding claim 6, the rejection of claim 1 is incorporated herein. The following is an abstract idea:
determining an ordered sequence of the plurality of classification-based machine learning models according to the performance definition set. (Determining an ordered sequence of models based on performance metrics can be practically performed in the human mind. This is a mental process.)
Claim 6 does not recite any further additional elements.
Regarding claim 7, the rejection of claim 6 is incorporated herein. The following is a continuation of an abstract idea:
wherein the performance definition set comprises a relative cost of precision and recall. (As used above, the performance definition set is used to determine an ordered sequence of the machine learning models. Determining an ordered sequence of models using a relative cost of precision and recall as the performance definition set can be practically performed in the human mind. This is a continuation of the above mental process.)
Claim 7 does not recite any further additional elements.
Regarding claim 8, the following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
A system comprising: one or more processors; and one or more memories storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: (This recites a generic computer at a high level of generality. This amounts to mere instructions to apply an exception.)
The remainder of claim 8 recites substantially similar subject matter to claim 1, and is rejected with the same rationale, mutatis mutandis.
Claims 10-14 recite substantially similar subject matter to claims 3-7 respectively, and are rejected with the same rationales, mutatis mutandis.
Regarding claim 15, the following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: (This recites a generic computer components and processes at a high level of generality. This amounts to mere instructions to apply an exception.)
The remainder of claim 15 recites substantially similar subject matter to claim 1, and is rejected with the same rationale, mutatis mutandis.
Claims 17-20 recite substantially similar subject matter to claims 3-6 respectively, and are rejected with the same rationales, mutatis mutandis.
Regarding claim 21, the rejection of claim 1 is incorporated herein. The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the set of training data samples and the classification type are received via a public application programming interface (API), and the trained classification-based machine learning model is deployed via a private API. (This describes the generic process of using APIs to communicate, which is an existing process in computing. This amounts to mere instructions to apply an exception.)
Regarding claim 23, the rejection of claim 1 is incorporated herein. The following is a continuation of an abstract idea:
wherein the minimal number of allowed training data partitions and the maximal number of allowed training data partitions are based at least in part on hardware used to train the one or more of the plurality of classification-based machine learning models. (In claim 1, the minimal and maximal numbers of training data partitions are selected. Selecting a minimal and maximal numbers of training data partitions based at least in part on hardware used to train the models can be practically performed in the human mind. This is a continuation of the mental process in claim 1.
Claim 23 does not recite any further additional elements.
Regarding claim 24, the following is an abstract idea:
wherein determining the number of training data partitions comprises determining the number of training data partitions N according to: N= max(L, (M x S) / P), wherein L is the minimal number of allowed training data partitions, M is the maximum sequence length for the plurality of classification-based machine learning models, P is the maximal number of allowed training data partitions, and S is the total number of the set of training data samples (Determining a number based on an equation is a mathematical calculation, which is a mathematical concept. Additionally, the equation is a mathematical equation, which is also a mathematical concept.)
Claim 24 does not recite any further additional elements.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 3, 8, 10, 15 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and MicroStrategy (“MicroStrategy 10 In-memory Analytics Guide,” 2017).
Regarding claim 1, Campos teaches
A computer-implemented method comprising: (The abstract states "Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating and comparing multiple trained machine learning models." [0098] states "FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input/output device 440.")
receiving, by one or more processors, a set of training data samples, a set of labels, and … for training a classification-based machine learning model; and ([0071] states "The evaluation apparatus 160 obtains multiple training data samples (210). For example, the validation system 140 obtains a training dataset to train multiple machine learning models to generate a predictive value depending on the problem being solved using the machine learning model. In the example of the ecommerce webpage 135, the validation system 140 obtains a training dataset 142 that includes multiple training samples where each sample includes a search query, a textual description of an item and a label indicating whether the item is relevant to the search query." The search queries/textual descriptions of items are interpreted as the training data samples, and the labels are interpreted as the set of labels. [0029] – [0030] state "Such a classification model can be selected after evaluation of multiple machine learning models that are trained to perform the same task and subsequent selection of the model that performs better relative to the other evaluated models. To evaluate the multiple machine learning models, the techniques described in this specification generate, using the models under evaluations, predictions based on a subset of samples from a training dataset." Therefore, the training data and labels are used to train a classification-based machine learning model. [0108] states "The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output." Hereinafter, this is considered to be the explanation for the “by the one or more processors” limitations.)
deploying, by the one or more processors, a trained classification-based machine learning model of a plurality of classification-based machine learning models based at least in part on a performance definition set by: ([0067] states "In some implementations, after selecting a particular machine learning model by the evaluation apparatus 160, the validation system 140 can deploy the selected machine learning model for the specific task for which the particular machine learning model (as well as other multiple machine learning models that were evaluated by the evaluation apparatus 160) was trained. For example, the validation system 140 deploys the first machine learning model 146A to classify each item in the list of items selected by the search system 150 in response to the buyer submitting a search query as being ‘relevant’ or ‘irrelevant’." Therefore, the machine learning models are classification-based machine learning models. [0088] states "After determining the performance of the multiple machine learning models based on the difference in the model performance metrics such as precision, recall and FPR, the evaluation apparatus 160 can select based on a pre-specified threshold, one or more machine learning models from multiple machine learning models generated by the validation system" The performance metrics are interpreted as the performance definition set. [0089] states "After selecting a machine learning model by the evaluation apparatus 160, the validation system 140 can deploy the selected machine learning model for the specific problem for which the multiple machine learning models were trained.")
to train the plurality of classification-based machine learning models, ([0029] – [0030] state "Such a classification model can be selected after evaluation of multiple machine learning models that are trained to perform the same task and subsequent selection of the model that performs better relative to the other evaluated models. To evaluate the multiple machine learning models, the techniques described in this specification generate, using the models under evaluations, predictions based on a subset of samples from a training dataset.")
(iii) storing the set of training data samples as a plurality of training data partitions, ([0053] states "In some implementations, to evaluate the multiple machine learning models, the evaluation apparatus 160 can select a proper subset of the training dataset 142" The subset is interpreted as the data partition, which, when identified, is stored as a plurality of training data partitions (the selected subset and the unselected subset).) wherein the plurality of training data partitions have a uniform distribution of the set of labels associated with the plurality of classification-based machine learning models, ([0053] states "The proper subset of the training dataset can be selected, for example, using random sampling, stratified sampling, etc., of the training dataset. In some implementations, after selecting the proper subset of the training dataset, the subset can be evaluated to ensure that the subset is representative of the training dataset. Representativeness can be assessed in various ways such as, e.g., the ratio of labels or other attributes in the training dataset. In some implementations, a proper subset of training samples is representative of the training dataset 142 when the proportional distribution of samples across labels in the proper subset is the same as the proportion distribution of samples across labels in the entire training dataset 142.")
(iv) training one or more of the plurality of classification-based machine learning models based at least in part on the plurality of training data partitions, and ([0083] states "The evaluation apparatus 160 generates a predicted value for the target attribute using a first machine learning model (230). For example, to evaluate the first machine learning model 146A and the second machine learning model 146B generated by the validation system 140, the evaluation apparatus 160 can use the first machine learning model 146A to generate predicted labels for each data sample in the subset of the training dataset. Similarly, the evaluation apparatus 160 can use the second machine learning model 146B to generate predicted labels for each sample in the subset of the training dataset (240)." Therefore, the two models are trained using the partitioned data.)
(v) deploying the trained classification-based machine learning model based at least in part on a performance of the trained classification-based machine learning model relative to the plurality of classification-based machine learning models with respect to the performance definition set. ([0088} states "After determining the performance of the multiple machine learning models based on the difference in the model performance metrics such as precision, recall and FPR, the evaluation apparatus 160 can select based on a pre-specified threshold, one or more machine learning models from multiple machine learning models generated by the validation system." The model performance metrics are interpreted as the performance definition set. As multiple models are compared, the performance is relative. [0089] states "After selecting a machine learning model by the evaluation apparatus 160, the validation system 140 can deploy the selected machine learning model for the specific problem for which the multiple machine learning models were trained.")
Campos does not appear to explicitly teach
[receiving] a classification type
(i) selecting a minimal number of allowed training data partitions and a maximal number of allowed training data partitions based at least in part on a total number of the set of training data samples, wherein (a) the minimal number of allowed training data partitions and the maximal number of allowed training data partitions are user configurable parameters and (b) the maximal number of allowed training data partitions is equal to a number of available cores of the one or more processors used to train the plurality of classification-based machine learning models, (ii) determining a number of training data partitions based at least in part on the minimal number of allowed training data partitions, the maximal number of allowed training data partitions, and a maximum sequence length for the plurality of classification-based machine learning models,
[storing data] having the determined number of training data partitions
However, TransmogrifAI—directed to analogous art—teaches
[receiving] a classification type (Page 3 states "TransmogrifAI will select the best model and hyper-parameters for you based on the class of modeling you are doing (eg. Classification, Regression etc.)." Page 3 further states "The ModelSelector is an Estimator that uses data to find the best model. BinaryClassificationModelSelector is for binary classification tasks, multi classification tasks can be done using MultiClassificationModelSelector. Best Regression model are done through RegressionModelSelector.” As the user inputs the class of modeling (classification type), the system receives the classification type.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos with the teachings of TransmogrifAI because as TransmogrifAI states on page 3, "Smart model selection and comparison gives the next layer of improvements over traditional ML workflows."
The combination of Campos and TransmogrifAI does not appear to explicitly teach
(i) selecting a minimal number of allowed training data partitions and a maximal number of allowed training data partitions based at least in part on a total number of the set of training data samples, wherein (a) the minimal number of allowed training data partitions and the maximal number of allowed training data partitions are user configurable parameters and (b) the maximal number of allowed training data partitions is equal to a number of available cores of the one or more processors used to train the plurality of classification-based machine learning models,
(ii) determining a number of training data partitions based at least in part on the minimal number of allowed training data partitions, the maximal number of allowed training data partitions, and a maximum sequence length for the plurality of classification-based machine learning models,
[storing data] having the determined number of training data partitions
However, Golding—directed to analogous art—teaches
(a) the minimal number of allowed training data partitions and the maximal number of allowed training data partitions are user configurable parameters (Column 10, lines 27-32 state "In some embodiments, the configuration interface 154 may also be used to configure the behavior of the repartitioner 156. For example, in some embodiments, the configuration interface may be used to specify the minimum and maximum number of partitions that can be created for each user-specified partition." Column 9, lines 57-64 state "As shown, in some embodiments, the data store 130 may implement a configuration interface 154 for the partition manager 150. Depending on the embodiment, the configuration interface 154 may be a graphical user interface (GUI), 60 a web interface, an application programming interface (API) such as a web services interface, or a combination thereof. In some embodiments, the configuration interface 154 may be exposed as part of a set of administrator interfaces to the data store 130." Therefore, the minimum and maximum number of partitions are user configurable.)
determining a number of the plurality of training data partitions based at least in part on a minimal number of allowed partitions, a maximal number of allowed partition … (c) the minimal number of allowed partitions and the maximal number of allowed partitions are user configurable parameters (Column 10, lines 27-32 state "In some embodiments, the configuration interface 154 may also be used to configure the behavior of the repartitioner 156. For example, in some embodiments, the configuration interface may be used to specify the minimum and maximum number of partitions that can be created for each user-specified partition." Column 9, lines 57-64 state "As shown, in some embodiments, the data store 130 may implement a configuration interface 154 for the partition manager 150. Depending on the embodiment, the configuration interface 154 may be a graphical user interface (GUI), 60 a web interface, an application programming interface (API) such as a web services interface, or a combination thereof. In some embodiments, the configuration interface 154 may be exposed as part of a set of administrator interfaces to the data store 130." Therefore, the minimum and maximum number of partitions are user configurable.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos and TransmogrifAI with the teachings of Golding because, as stated by Golding in columns 9 and 10, "In some embodiments, the configuration 65 interface 154 may be provided to individual clients, which may allow the client to configure their own partitioning requirements, policies, etc."
The combination of Campos, TransmogrifAI, and Golding does not appear to explicitly teach
(i) selecting a minimal number of allowed training data partitions and a maximal number of allowed training data partitions based at least in part on a total number of the set of training data samples, wherein … and (b) the maximal number of allowed training data partitions is equal to a number of available cores of the one or more processors used to train the plurality of classification-based machine learning models,
[determining a number of data partitions based on] a maximum sequence length for the plurality of classification-based machine learning models
[storing data] having the determined number of training data partitions
However, Silfa—directed to analogous art—teaches
[determining a number of data partitions based on] a maximum sequence length for the plurality of classification-based machine learning models (Page 2 states "Furthermore, to meet Service-Level-Agreement (SLA), we limit the maximum number of time-steps in a given processing lane." The time-steps are interpreted as the sequence length, as Page 2 states "In an RNN, an input sequence (e.g. an audio frame) is composed of N time-steps, i.e.,
X
=
[
x
1
,
x
2
,
.
.
.
,
x
N
]
. In an LSTM or GRU network, time-steps are processed sequentially, from
x
1
to
x
n
." Page 7 states " In E-Batch, the number of time-steps processed by a given lane is limited to a threshold (i.e., N). " Therefore, as the requests are limited by the maximum sequence length, the number of partitions will be affected based on the maximum sequence length.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, and Golding with the sequence length taught by Silfa because, as Silfa states on page 7, "By using a threshold, we can trade latency for energy consumption. Batching with a small number of time-steps per lane decreases latency but incurs in a large amount of weight swaps, which decreases weight locality. On the contrary, batching with a large number of time-steps per lane increases weight reuse, which significantly reduces energy consumption at the expense of an increase in latency."
The combination of Campos, TransmogrifAI, Golding, and Silfa does not appear to explicitly teach
(i) selecting a minimal number of allowed training data partitions and a maximal number of allowed training data partitions based at least in part on a total number of the set of training data samples, wherein … and (b) the maximal number of allowed training data partitions is equal to a number of available cores of the one or more processors used to train the plurality of classification-based machine learning models,
[storing data] having the determined number of training data partitions
However, MicroStrategy—directed to analogous art—teaches
(i) selecting a minimal number of allowed training data partitions and a maximal number of allowed training data partitions based at least in part on a total number of the set of training data samples, wherein … (Page 276 states "The minimum number of partitions is dictated by the number of rows in the largest table divided by 2 billion, since each partition can hold up to 2 billion records. The maximum number of partitions is dictated by the number of cores on the box. The number of partitions should typically be between the minimum and maximum, and closer to half the number of logical cores." The number of rows is interpreted as the total number of the set of training data samples.) and (b) the maximal number of allowed training data partitions is equal to a number of available cores of the one or more processors used (Page 276 states " The maximum number of partitions is dictated by the number of cores on the box.”)
[storing data] having the determined number of training data partitions (Page 275 states "A major advantage of MicroStrategy 10.0 cubes is the ability to partition cubes. While OLAP cubes were limited to 2 billion rows, PRIME OLAP cubes can be divided into partitions and each partition can contain up 2 billion rows." One of ordinary skill in the art would reason that the number of partitions determined will be used to store the data. Additionally, page 74 states "In the Number of Partitions field, type the number of partitions to split the dataset into.")
It would have been obvious to a person having ordinary skill in the art before the effective filing date of this application to combine the teachings of Campos, TransmogrifAI, Golding, and Silfa with the teachings of MicroStrategy because, as Microstrategy states on page 275 "A major advantage of MicroStrategy 10.0 cubes is the ability to partition cubes. While OLAP cubes were limited to 2 billion rows, PRIME OLAP cubes can be divided into partitions and each partition can contain up 2 billion rows. This advantage allows PRIME OLAP cubes to increase in capacity and scalability through partitions." Additionally, page 276 states "Some of the largest fact tables in the application are typically good candidates for partitioning and thus influence the choice of the partition attribute. They need to be partitioned to accommodate large data sizes and to take advantage of the PRIME parallel processing architecture." Also, page 70 states "The benefits of storing your data in your Intelligence Server’s memory and processing the data using all the server’s processor cores partitioning your dataset include: • Improving the response time for your reports, documents, and dashboards. • Reducing the load on your data warehouse, because your dataset is stored in Intelligence Server’s memory instead of the data warehouse. • Analyzing large volumes of data in real time, since your datasets can contain up to 2 billion rows of data for every partition on the Intelligence Server. • Using the standard features of the MicroStrategy platform, such derived metrics, custom groups, consolidations, and so on, on your dataset."
Regarding claim 3, the rejection of claim 1 is incorporated herein. Campos does not appear to explicitly teach
wherein the set of training data samples are preprocessed using a rule-based framework.
However, TransmogrifAI—directed to analogous art—teaches
wherein the set of training data samples are preprocessed using a rule-based framework. (Page 1 states “The TransmogifAI tranmogrifier (shortcut .transmogrify()) tales in a sequence of features, automatically applies default transformations to them based on feature types (e.g. imputation, null value tracking, one hot encoding, tokenization, split Emails and pivot out the top K domains) and combines them into a single vector.” The transformations are interpreted as the rule-based framework.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos and the transformations of TransmogrifAI because, as stated by TransmogrifAI on page 1, “This is the Stage that automates the feature engineering step in the machine learning pipeline.”
Regarding claim 8, Campos teaches
A system comprising: ([0098] states "FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above.”)
one or more processors; and ([0098] states "FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input/output device 440.”)
one or more memories storing processor executable instructions that, when executed by the one or more processors to perform operations comprising: ([0098] states "FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input/output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In one implementation, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi-threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430.")
The remainder of claim 8 recites substantially similar subject matter to claim 1 and is rejected with the same rationale, mutatis mutandis.
Claim 10 recites substantially similar subject matter to claim 3 and is rejected with the same rationale, mutatis mutandis.
Regarding claim 15, Campos teaches
One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: ([0098] states "FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input/output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In one implementation, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi-threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430.")
The remainder of claim 15 recites substantially similar subject matter to claim 1 and is rejected with the same rationale, mutatis mutandis.
Claim 17 recites substantially similar subject matter to claim 3 and is rejected with the same rationale, mutatis mutandis.
Regarding claim 23, the rejection of claim 1 is incorporated herein. Campos teaches
train the one or more of the plurality of classification-based machine learning models. ([0083] states "The evaluation apparatus 160 generates a predicted value for the target attribute using a first machine learning model (230). For example, to evaluate the first machine learning model 146A and the second machine learning model 146B generated by the validation system 140, the evaluation apparatus 160 can use the first machine learning model 146A to generate predicted labels for each data sample in the subset of the training dataset. Similarly, the evaluation apparatus 160 can use the second machine learning model 146B to generate predicted labels for each sample in the subset of the training dataset (240)." Therefore, the two models are trained using the partitioned data.)
The combination of Campos, TransmogrifAI, Golding, and Silfa does not appear to explicitly teach
wherein the minimal number of allowed training data partitions and the maximal number of allowed training data partitions are based at least in part on hardware used to [execute an application]
However, MicroStrategy—directed to analogous art—teaches
wherein the minimal number of allowed training data partitions and the maximal number of allowed training data partitions are based at least in part on hardware used to [execute an application] (Page 276 states "The minimum number of partitions is dictated by the number of rows in the largest table divided by 2 billion, since each partition can hold up to 2 billion records. The maximum number of partitions is dictated by the number of cores on the box. The number of partitions should typically be between the minimum and maximum, and closer to half the number of logical cores." The number of rows is interpreted as the total number of the set of training data samples.) and (b) the maximal number of allowed training data partitions is equal to a number of available cores of the one or more processors used (Page 276 states " The maximum number of partitions is dictated by the number of cores on the box.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, and Silfa with the teachings of MicroStrategy for the reasons given above in regards to claim 1.
Claim(s) 4, 11, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and MicroStrategy (“MicroStrategy 10 In-memory Analytics Guide,” 2017) as applied to claim 1 above, further in view of D2L-AI (“Training on Multiple GPUs”, April 2022).
Regarding claim 4, the rejection of claim 1 is incorporated herein. Campos teaches
wherein each of the plurality of the classification-based machine learning models … and (ii) comprises a different model architecture ([0067] states "In some implementations, after selecting a particular machine learning model by the evaluation apparatus 160, the validation system 140 can deploy the selected machine learning model for the specific task for which the particular machine learning model (as well as other multiple machine learning models that were evaluated by the evaluation apparatus 160) was trained. For example, the validation system 140 deploys the first machine learning model 146A to classify each item in the list of items selected by the search system 150 in response to the buyer submitting a search query as being ‘relevant’ or ‘irrelevant’." Therefore, the machine learning models are classification-based machine learning models. [0049] states "The validation system 140 can also include a model configurator 148 that can generate and configure various machine learning models based on the machine learning model properties such as the type of machine learning model, the number of parameters of the machine learning model, the optimization techniques etc. For example, if the machine learning model is a neural network, the model configurator 148 can set the number of neural network layers, the number of neurons per layer, activation function, the number of training iterations of the training process, etc.")
The combination of Campos, TransmogrifAI, Golding, Silfa, and MicroStrategy does not appear to explicitly teach
(i) is trained using a separate graphics processing unit (GPU)
However, D2L-AI—directed to analogous art—teaches
wherein each of the plurality of [machine learning models] is training using a separate graphics processing unit (GPU). (Page 2 states “Last, we could partition data across multiple GPUs. This way all GPUs perform the same type of work, albeit on different observations.” As can be seen in the figure on pages 3, the data parallel method copies the model, meaning that there are a plurality of models, onto each GPU, and they are trained using a subset of the data.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, Silfa, and MicroStrategy with the teachings of D2L-AI because, as D2L teaches on page 3, “By and large, data parallelism is the most convenient way to proceed, provided we have access to GPUs with sufficiently large memory.”
Claims 11 and 18 recite substantially similar subject matter to claim 4 and are rejected with the same rationale, mutatis mutandis.
Claim(s) 5, 12, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and MicroStrategy (“MicroStrategy 10 In-memory Analytics Guide,” 2017) as applied to claim 1 above, further in view of Inamdar (US 2021/0097472 A1).
Regarding claim 5, the rejection of claim 1 is incorporated herein. The combination of Campos, TransmogrifAI, Golding, Silfa, and MicroStrategy does not appear to explicitly teach
wherein each training data sample of the set of training data samples comprises an input-vector based representation of an input document.
However, Inamdar—directed to analogous art—teaches
wherein each training data sample of the set of training data samples comprises an input-vector based representation of an input document. (Figure 6, Step 602 is “generate candidate vectors from a plurality of candidate documents”, Step 608 is “generate a feature vector for each of the subset of candidate documents”, and Step 610 is “apply a feature vector to generate a score for each of the subset of the plurality of candidate documents”. [0056] states "Specifically, the feature extractor 145 can be programmed to determine, for each of a plurality of candidate documents, a feature vector representing the candidate document and an objective document, for example, a candidate search parameter vector as described in FIG. 1." Additionally, [0074] states, in reference to training an artificial neural network, "The values from the feature vector are provided to a plurality of input nodes." As the feature value vector is provided as input, it is interpreted as input-vector based, and the feature vector is interpreted as the training sample.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, Silfa, and MicroStrategy with the teachings of Inamdar because, as stated by Inamdar in [0019], "Systems and methods are described herein that overcome existing cold start problems associated with candidate recommendation systems to provide a technical solution that enables candidate recommendation systems to predict or identify the most qualified candidates ( e.g., resumes, jobs, etc. ) upon these systems been deployed at an organization or until an ML model has been sufficiently trained based on industry relevant training data. Examples are provided herein in context of resume-to-job recommendation, however, this disclosure should not be construed or limited to only encompass candidate matching, and can include, in other examples, different recommendation applications, such as employee matching (e.g., recommending employees for projects, functions, etc.), document matching, item matching, movie matching, song matching, consumer matching, etc."
Claims 12 and 19 recite substantially similar subject matter to claim 5 and are rejected with the same rationale, mutatis mutandis.
Claim(s) 6, 7, 13, 14, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and MicroStrategy (“MicroStrategy 10 In-memory Analytics Guide,” 2017) as applied to claim 1 above, further in view of Plumbley (US 2021/0117869 A1).
Regarding claim 6, the rejection of claim 1 is incorporated herein. Campos teaches
the classification-based machine learning models ([0067] states "In some implementations, after selecting a particular machine learning model by the evaluation apparatus 160, the validation system 140 can deploy the selected machine learning model for the specific task for which the particular machine learning model (as well as other multiple machine learning models that were evaluated by the evaluation apparatus 160) was trained. For example, the validation system 140 deploys the first machine learning model 146A to classify each item in the list of items selected by the search system 150 in response to the buyer submitting a search query as being ‘relevant’ or ‘irrelevant’." Therefore, the machine learning models are classification-based machine learning models.)
The combination of Campos, TransmogrifAI, Golding, Silfa, and MicroStrategy does not appear to explicitly teach
determining an ordered sequence of the plurality of [machine learning models] according to the performance definition set.
However, Plumbly—directed to analogous art—teaches
determining an ordered sequence of the plurality of [machine learning models] according to the performance definition set. ([0114] states "In another example, all models may be ranked according to their MPSs [model performance statistics] in which the best performing K models or topmost performing K models are selected for inclusion into the set of optimal trained models. The set of optimal trained models may be stored in a model database 232 for use in forming one or more ensemble models." The model performance statistics is interpreted as the performance definition set. One of ordinary skill in the art would realize that an ordered sequence is a ranking.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, Silfa, and MicroStrategy with the performance evaluation taught by Plumbly because, as stated by Plumbly in [0077] "The inventors have advantageously developed a system for generating and selecting from a large number of trained models, or a plurality of sets of trained models, with the same or similar objectives a subset of the best performing trained models that can be used to create one or more ensemble model(s) that have been optimised for modelling a process or problem associated with one or more compounds."
Regarding claim 7, the rejection of claim 6 is incorporated herein. Campos teaches
wherein the performance definition set include a relative cost of precision and recall. ([0066] states "In some implementations, after determining the performance of the multiple machine learning models based on the difference in one or more model performance metrics, such as precision, recall and FPR, using CI and p-value, the evaluation apparatus 160 can select based on a pre-specified threshold, a machine learning models from multiple machine learning models generated by the validation system." The model performance metrics are interpreted as the performance definition set. As the difference between the metrics of the models is calculated, the precision and recall are relative.)
Claims 13 and 14 recite substantially similar subject matter to claims 6 and 7 respectively and are rejected with the same rationale, mutatis mutandis.
Claim 20 recites substantially similar subject matter to claim 6 and is rejected with the same rationale, mutatis mutandis.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Campos (US 2023/0097897 A1), TransmogrifAI (“AutoML Capabilities”, 2020), Golding (US 11,061,930 B1), Silfa (“E-BATCH: Energy-Efficient and High-Throughput RNN Batching”, 2020), and MicroStrategy (“MicroStrategy 10 In-memory Analytics Guide,” 2017) as applied to claim 1 above, further in view of Crankshaw (“Clipper: A Low-Latency Online Prediction Serving System”, 2017).
Regarding claim 21, the rejection of claim 1 is incorporated herein. Campos teaches
the labeled data and ([0071] states "The evaluation apparatus 160 obtains multiple training data samples (210). For example, the validation system 140 obtains a training dataset to train multiple machine learning models to generate a predictive value depending on the problem being solved using the machine learning model. In the example of the ecommerce webpage 135, the validation system 140 obtains a training dataset 142 that includes multiple training samples where each sample includes a search query, a textual description of an item and a label indicating whether the item is relevant to the search query." The search queries/textual descriptions of items are interpreted as the training data samples, and the labels are interpreted as the set of labels.)
Campos does not appear to explicitly teach
wherein … the classification type are received via a public application programming interface (API) and the particular machine learning model is deployed via a private API.
However, TransmogrifAI—directed to analogous art—teaches
the classification type (Page 3 states "TransmogrifAI will select the best model and hyper-parameters for you based on the class of modeling you are doing (eg. Classification, Regression etc.)." Page 3 further states "The ModelSelector is an Estimator that uses data to find the best model. BinaryClassificationModelSelector is for binary classification tasks, multi classification tasks can be done using MultiClassificationModelSelector. Best Regression model are done through RegressionModelSelector.” As the user inputs the class of modeling (classification type), the system receives the classification type.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos with the teachings of TransmogrifAI because as TransmogrifAI states on page 3, "Smart model selection and comparison gives the next layer of improvements over traditional ML workflows."
The combination of Campos, TransmogrifAI, Golding, Silfa, and MicroStrategy does not appear to explicitly teach
wherein the [queries] are received via a public application programming interface (API) and the particular machine learning model is deployed via a private API.
However, Crankshaw—directed to analogous art—teaches
wherein the [queries] are received via a public application programming interface (API) and the particular machine learning model is deployed via a private API. (Fig. 1 shows the application facing API (interpreted as the public API) and the RPC (interpreted as the private API, as it is not available to the applications) between the model containers and Clipper. Page 614 states "Applications issue prediction requests to Clipper through application facing REST or RPC APIs. Prediction requests are first processed by the model selection layer. Based on properties of the prediction request and recent feedback, the model selection layer dispatches the prediction request to one or more of the models through the model abstraction layer." Therefore, the application facing API receives the requests. Page 617 states "Models deployed in Clipper are each encapsulated within their own lightweight container (§4.4), communicating with Clipper through an RPC mechanism that provides a uniform interface to Clipper and simplifies the deployment of new models." Therefore, the machine learning models are deployed via the private API.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Campos, TransmogrifAI, Golding, Silfa, and MicroStrategy with the APIs of Crankshaw because, as Crankshaw states on page 617, "The lightweight RPC system minimizes the overhead of the container-based architecture and simplifies cross-language integration." Crankshaw also states on page 620, "By implementing model selection policies that provide different cost-accuracy tradeoffs, as well as an API for users to implement their own policies, Clipper provides a mechanism to easily navigate the tradeoffs between accuracy and computational cost on a per-application basis."
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JESSICA THUY PHAM whose telephone number is (571)272-2605. The examiner can normally be reached Monday - Friday, 9 A.M. - 5:00 P.M..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.P./Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
