DETAILED ACTION
This action is responsive to the following communications: Original Application filed on August 31, 2022. All references to this application refer to the U.S. Patent Application Publication No. 2024/0070528 A1.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending in this case. Claims 1, 7, and 14 are the independent claims. Claims 1-20 are rejected.
Specification
The disclosure is objected to because of the following informalities:
In paragraph 0038, there is an extra space in “application/j son” that should be removed.
Appropriate correction is required.
The use of trademarks has been noted in this application. The term should be accompanied by the generic terminology, if appropriate; furthermore the term should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM , or ® following the term.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
The following were not properly marked:
JAVASCRIPT (paragraph 0036
BLUETOOTH (paragraphs 0042, 0045)
BLU-RAY (paragraph 0049)
Appropriate corrections are required.
Claim Objections
Claims 4, 10, and 17 are objected to because of the following informalities:
Each of claims 4, 10, and 17 recite “wherein the selecting the most effective ensemble is performed…” For consistency, each of these claims should be amended to recite “wherein the selecting the most effective ensemble of models is performed…”
Appropriate corrections are required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Each of the independent claims (1, 7, and 14) recite the configuration file specifying “a sub-group of base models selected from available models” and “base estimators” (as well as further recitations concerning hyperparameters associated with “base estimators.” However, paragraph 0021 recites “[t]hroughout this disclosure, the terms "estimator" and "model" may be used interchangeably unless otherwise specified.” Therefore, the use of both phrases within a single claim introduces indefiniteness because it is unclear if the “available models” are the same as “base estimators,” such that the sub-group of base models is equivalent to a sub-group of base estimators, or if the base estimators are different than the recited “available models” such that the sub-group of base models is not equivalent to a sub-group of base estimators.
Accordingly, independent claims 1, 7, and 14 are rendered indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In view of the Specification and the use of “models” throughout the independent and dependent claims (which do not use “base estimators”), for the purposes of examination, the claims are interpreted as reciting “base models” in each instance where “base estimators” appears. To overcome these rejections, the Examiner recommends amending all instances of “base estimators” to recite “base models.”
Dependent claims 2-6, 8-13, and 15-20 are rejected solely due to dependence from a rejected parent claim.
To expedite a complete examination of the instant application, the claims rejected above under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention, are further rejected as set forth below in anticipation of amendments to these claims to correct the failure.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
With regard to claim 1,
Step 2A, Prong 1
This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim “recites” a judicial exception when the judicial exception is “set forth” or “described” in the claim.
Claim 1 recites:
A method of evaluating and selecting an ensemble of machine language models using extremely randomized bootstrap aggregation with replacement, the method comprising:
receiving:
a configuration file, the configuration file specifying:
a sub-group of base models selected from a group of available models;
base estimators; and
available hyperparameter values associated with each of the base estimators;
an input training dataset;
a validation dataset;
an integer specifying a number of trials to build and train an extremely randomized ensemble;
a range specifying a minimum and maximum number of the base estimators to be contained in a single randomized ensemble; and
one or more evaluation metrics to evaluate a trained ensemble on the validation dataset;
selecting, with replacement and from the received configuration file, a random number of base estimators within the range specifying the minimum and maximum number of base estimators;
training models of an ensemble of models as specified by the sub-group of base models using a random sample of the input training dataset and using a random set of the available hyperparameter values;
generating, by the trained models using the validation dataset, individual predictions for each record in the validation dataset;
generating, using the evaluation metric, a score of each ensemble of models based on a comparison of the generated individual predictions and known outputs of the validation dataset;
selecting, based on the score, an ensemble of models; and
outputting the selected ensemble of models including associated base estimators and hyperparameters.
The broadest reasonable interpretation of the bolded limitations above are directed to a mental process able to be performed in the human mind or by a human using pen and paper. A human can determine and make observations and comparisons, track metrics against thresholds or baselines, generate inferences and compare against validation sets, and generate scores mentally or with pen and paper.
Step 2A, Prong 1 (Yes).
Step 2A, Prong 2
This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception or whether the claim is “directed to” the judicial exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d).
As this is a method claim, there are no additional elements in this claim, as all limitations form parts of the judicial exception.
Step 2A, Prong 2 (No).
Step 2B
This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP 2106.05.
As explained with respect to Step 2A, there are no additional elements. See MPEP 2106.05(f).
Step 2B (No).
Claim 1 is ineligible.
With respect to independent claims 7 and 14,
These claims are similar in scope to Claim 1 and are rejected under a similar rationale. The processors, memory, and non-transitory computer-readable medium recited in these claims are also generic computing components.
Claims 7 and 14 are ineligible.
Dependent Claims:
Claims 2-5, 8-11, and 15-18: These claims only recite further abstract ideas (mental processes: conditions or observations for determining termination or methodology to select a preferred ensemble) and thus are ineligible.
Claims 6, 12, and 19: These claims recite further generic computer components (“processors”) performing typical processing activity (“parallel processing”) and do not provide a practical application or inventive concept and thus are ineligible.
Claims 13 and 20: These claims recite a limitation that describing the type of model (not limited to decision trees”). This is a recitation of a particular data structure, and therefore does not provide a practical application or inventive concept and thus are ineligible.
To expedite a complete examination of the instant application, the claims rejected above under 35 U.S.C. 101, as relating to judicial exceptions without significantly more, are further rejected as set forth below in anticipation of amendments to these claims to place them within the four statutory categories of invention.
Examiner’s Note
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-12 and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. 2022/0198222 A1, filed by Rawat et al., on December 17, 2020, and published on June 23, 2022 (hereinafter Rawat), in view of U.S. Patent Application Publication No. 2021/0248517 A1, filed by Soppin et al., on March 26, 2020, and published on August 12, 2021 (hereinafter Soppin).
With respect to independent claim 1, Rawat discloses a method of evaluating and selecting an ensemble of machine language models using extremely randomized bootstrap aggregation with replacement, the method comprising: Rawat discloses a method of evaluating and selecting ML models, including ensemble models, using bagging techniques (see Rawat, abstract [geniting a preferred ensemble of ML models based on received inputs, hyperparameters, and various metrics and validation data]).
Receiving:
A configuration file, the configuration file specifying: Rawat discloses that the information can be received via a user from a configuration file (see Rawat, paragraph 0017 [data may be provided by a user through a file or UI]).
A sub-group of base models selected from a group of available models; Rawat discloses the dataset including a sub-group of ML pipelines (e.g., sub-group of models) (see Rawat, paragraph 0019 [receives a dataset and ML pipeline components]; see also, Rawat, paragraph 0017, described supra).
Base estimators; Rawat discloses the dataset including base pipelines (see Rawat, paragraphs 0017 and 0019, described supra).
Available hyperparameter values associated with each of the base estimators; an input training dataset; Rawat discloses the dataset including hyperparameters associated with each of the ML pipelines and an input training set (see Rawat, Fig. 1; see also, Rawat, paragraph 0020 [receives learning algorithms, preprocessing routines, hyperparameters]; see also, Rawat, paragraphs 0017 and 0019, described supra).
A validation dataset; Rawat discloses the dataset including validation dataset (see Rawat, paragraph 0017, described supra).
An integer specifying a number of trials to build and train an extremely randomized ensemble; Rawat discloses a number of cycles (e.g., trials) to train an ensemble (see Rawat, paragraph 0031 [describing termination conditions, such as number of cycles]).
Rawat fails to expressly disclose the dataset including a range specifying a minimum and maximum number of the base estimators to be contained in a single randomized ensemble.
However, Soppin teaches defining the number of models in ensemble clusters as a hyperparameter (see Soppin, paragraph 0041 [describing the size of clusters based on the number “n” of the total (such as n/2)]).
Accordingly, it would have been obvious to one of ordinary skill in the art, having the teachings of Rawat and Soppin before him before the effective filing date of the claimed invention, to modify the method of Rawat to incorporate using size ranges to determine ensemble sizes as taught by Soppin. One would have been motivated to make such a combination because this provides more efficiency in determining optimal hyperparameters for training ML models, as taught by Soppin (see Soppin, paragraph 0004 [“Further, in existing techniques, all the models in an ensemble model are typically developed independent of each other and, then, combined optimally during deployment. This results in wastage of computational resources, redundancies, and conflict in decision-making. For example, each model can learn certain features in a better way and no single model can learn all the good features. Further, for example, each model is accurate for different types of field data and for the given training data. Thus, apart from wastage of computational resources, there is confusion and deployment errors. Further, in some of the existing techniques, there is no learning of features after the ensemble model are developed. Further, existing techniques do not disclose competitive learning that enables the ensemble model to learn many relevant features. Moreover, existing techniques perform multiple iterations resulting in wastage of computational resources, over training, and inaccuracies.”]).
Rawat, as modified by Soppin, further teaches:
One or more evaluation metrics to evaluate a trained ensemble on the validation dataset; Rawat further teaches the dataset providing one or more evaluation metrics to evaluation trained ensembles (see Rawat, paragraphs 0018 [describing how the pipelines are evaluated and validated based on preselected performance metrics, such as validation loss], 0021 [performance values with respect to each pipeline is determined and used for validation], and 0032 [preferred ensembles are selected based on performance value ranking]).
Selecting, with replacement and from the received configuration file, a random number of base estimators within the range specifying the minimum and maximum number of base estimators; Rawat further teaches selecting with replacement (e.g., bagging) a random number of ML pipelines within the range (see Rawat, paragraphs 0017 and 0019, described supra).
Training models of an ensemble of models as specified by the sub-group of base models using a random sample of the input training dataset and using a random set of the available hyperparameter values; Rawat further teaches training he ML pipelines as specified using random samples of the training dataset (see Rawat, paragraph 0017, described supra).
Generating, by the trained models using the validation dataset, individual predictions for each record in the validation dataset; Rawat further teaches generating inferences for each item in the training data by the ML pipelines (see Rawat, paragraphs 0017 and 0019, described supra).
Generating, using the evaluation metric, a score of each ensemble of models based on a comparison of the generated individual predictions and known outputs of the validation dataset; Rawat further teaches generating scores for each ML pipeline and cluster of ML pipelines using the performance metric and the validation set (see Rawat, paragraphs 0018, 0021, and 0032, described supra).
Selecting, based on the score, an ensemble of models; Rawat further teaches selecting a preferred ensemble based on the score (see Rawat, paragraph 0032, described supra).
Outputting the selected ensemble of models including associated base estimators and hyperparameters; Rawat further teaches outputting/saving the preferred ensemble including the components and hyperparameters (see Rawat, paragraph 0032, described supra).
With respect to dependent claim 2, Rawat, as modified by Soppin, teaches the method of claim 1, as described above.
Rawat further teaches the method wherein selecting the most effective ensemble of models based on the scoring comprises averaging the individual predictions for each record and selecting the ensemble of models with the highest average of individual predictions.
Rawat further teaches using the performance metric and averaging the values to determine best performers (see Rawat, paragraphs 0019 and 0032, described supra, claim 1).
With respect to dependent claim 3, Rawat, as modified by Soppin, teaches the method of claim 1, as described above.
Rawat further teaches the method wherein selecting the most effective ensemble of models is based on determining a majority rule of correct individual predictions for each record.
Rawat further teaches using majority vote to determine preferred ensembles (see Rawat, paragraphs 0017, 0019, and 0032, described supra, claim 1).
With respect to dependent claim 4, Rawat, as modified by Soppin, teaches the method of claim 1, as described above.
Rawat further teaches the method wherein the selecting the most effective ensemble is performed after training a number of ensembles equal to the integer specifying the number of trials to build and train the extremely randomized ensemble.
Rawat further teaches the selection of preferred ensembles is made after the minimum number of training cycles has been completed (see Rawat, paragraph 0031, described supra, claim 1).
With respect to dependent claim 5, Rawat, as modified by Soppin, teaches the method of claim 1, as described above.
Rawat further teaches the method, further comprising logging the trained models, hyperparameters used in the training, and an evaluation metric used in the scoring.
Rawat further teaches outputting/saving the preferred ensemble including the components and hyperparameters (see Rawat, paragraph 0032, described supra, claim 1).
With respect to dependent claim 6, Rawat, as modified by Soppin, teaches the method of claim 1, as described above.
Soppin further teaches the method wherein a cluster of computer processors are configured to perform the training of the models of the ensemble of models in parallel.
Soppin further teaches that training of models can be performed in parallel (see Soppin, paragraphs 0025 [models and clusters may be trained in parallel] and 0089 [each set of models may be processes (e.g., trained) in parallel]).
Independent claim 7, and its respective dependent claims 8-12, recite a system, comprising: a processor and memory comprising instructions that when executed by the processor cause the processor to perform the method of independent claim 1, and its respective dependent claims 2-6. Accordingly, independent claim 7, and its respective dependent claims 8-12 are rejected under the same rationales used to reject independent claim 1, and its respective dependent claims 2-6, which are incorporated herein.
Independent claim 14, and its respective dependent claims 15-19, recite a non-transitory computer-readable medium comprising a set of instructions that, in response to being executed by a processor circuit, cause the processor circuit to perform a method of independent claim 1, and its respective dependent claims 2-6. Accordingly, independent claim 14, and its respective dependent claims 15-19 are rejected under the same rationales used to reject independent claim 1, and its respective dependent claims 2-6, which are incorporated herein.
Claims 13 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Rawat, in view of Soppin, further in view of U.S. Patent Application Publication No. 2019/0244139 A1, filed by Varadarajan et al., on March 7, 2018, and published on August 8, 2019 (hereinafter Varadarajan).
With respect to dependent claim 13, Rawat, as modified by Soppin, teaches the system of claim 7, as described above.
Rawat and Soppin fail to further teach the system wherein the models are not limited to decision trees.
However, Varadarajan teaches application of ensemble selection and training can be performed on multiple types of models (see Varadarajan, paragraph 0061 [models include support vector machines, artificial neural networks, decision trees, or random forest]).
Accordingly, it would have been obvious to one of ordinary skill in the art, having the teachings of Rawat, Soppin, and Varadarajan before him before the effective filing date of the claimed invention, to modify the method of Rawat, as modified by Soppin, to apply the technique to models other than decision trees as taught by Soppin. One would have been motivated to make such a combination because these are the typical types of ML models known to one of ordinary skill in the art, as taught by Varadarajan (see Varadarajan, paragraph 0061, described supra).
Dependent claim 20 recites a non-transitory computer-readable medium comprising a set of instructions that, in response to being executed by a processor circuit, cause the processor circuit to perform as the system of dependent claim 13. Accordingly, dependent claim 20 is rejected under the same rationales used to reject dependent claim 13, which are incorporated herein.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure. See PTO-892.
It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
Any inquiry concerning this communication or earlier communications from the Examiner should be directed to ERIC J. BYCER whose telephone number is (571) 270-3741. The Examiner can normally be reached Monday - Thursday 9am-6pm, and alternate Fridays 9am-5pm.
Examiner interviews are available via a variety of formats. See MPEP § 713.01. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/InterviewPractice.
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, MATT ELL can be reached on (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center to authorized users only. Should you have questions about access to the USPTO patent electronic filing system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/ERIC J. BYCER/
Primary Examiner
Art Unit 2141
Prosecution Insights