Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-15 are pending for examination. Claims 1 and 15 are independent.
Drawings
The drawings are objected to because Figure 7 on the left side in line 3 shows “our2” which does not correspond to the details in corresponding sections of the specification and is unclear. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 1 and 15 objected to because of the following informalities: Claim 1 line 1 recites "a ML Model" without reciting what the abbreviation "ML" stands for. Claim 15 recites a similar limitation. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4-7 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 4 recites the limitation "the minimum of the cost function" in line 2. There is insufficient antecedent basis for this limitation in the claim.
Claim 5 recites the limitation "the input data" in line 3. There is insufficient antecedent basis for this limitation in the claim.
Claims 6-7 depend on claim 5 and do not resolve the 112(b) rejection and are also rejected under 112(b).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1
According to the first part of the analysis, in the instant case, claims 1-14 are directed to a method, and claim 15 is directed to a system. Thus, each of the claims falls within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Regarding Claim 1:
2A Prong 1:
extracting a plurality of global features from the sample dataset; (This step for extracting features is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
applying a feature selection process for selecting a first subset of the plurality of global features; (This step for selecting a subset of features is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
analyzing a classification performance of the ML model fed the first subset, to identify an error in classification by the ML model; (This step for analyzing performance to identify error is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
identifying a subset of the sample dataset related to the error; (This step for identifying a subset related to error is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
extracting a plurality of second features from the subset of the sample data; (This step for extracting features is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
applying the feature selection process for selecting a second subset of the plurality of second features; (This step for selecting a subset of features is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
A computer implemented method of generating a reduced version of a ML model, comprising: (The computer implemented method is understood to be generic computer elements – See MPEP 2106.05(f).)
obtaining a sample dataset; (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and data gathering. See MPEP 2106.05(g).)
creating a reduced version of the ML model, comprising an ensemble of: a first ML model component trained by applying the first subset of the plurality of global features to the sample dataset, and a second ML model component trained by applying the second subset of the plurality of features to the subset of the sample data related to the error. (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying machine learning as a tool to perform the abstract idea - see MPEP 2106.05(f).)
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are insignificant extra solution activity in combination of generic computer functions that are implemented to perform the disclosed abstract idea above.
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
A computer implemented method of generating a reduced version of a ML model, comprising: (The computer implemented method is understood to be generic computer elements – See MPEP 2106.05(f).)
obtaining a sample dataset; (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and is well understood, routine and conventional activity of transmitting and receiving data as identified by the court (MPEP2106.05(d)(ll)(i))))
creating a reduced version of the ML model, comprising an ensemble of: a first ML model component trained by applying the first subset of the plurality of global features to the sample dataset, and a second ML model component trained by applying the second subset of the plurality of features to the subset of the sample data related to the error. (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying machine learning as a tool to perform the abstract idea - see MPEP 2106.05(f).)
The additional elements as disclosed above in combination of the abstract idea
are not sufficient to amount to significantly more than the judicial exception as they are
well, understood, routine and conventional activity as disclosed in combination
of generic computer functions that are implemented to perform the disclosed abstract idea above.
Regarding Claim 15: see the rejection of claim 1 above. Same rationale applies.
2A Prong 2 & 2B: The claim recites another additional element “A system for generating a reduced version of a ML model, comprising: at least one processor executing a code for:” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claim 2
2A Prong 1: The claim does not recite an Abstract idea.
2A Prong 2 & 2B:
wherein during inference, input data is first fed into the first ML model component to obtain a classification outcome, and in response to the classification outcome denoting the error in classification comprising ambiguity in the classification outcome, the input data is fed into the second ML model component to obtain a resolution to the classification outcome. (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying machine learning as a tool to perform the abstract idea - see MPEP 2106.05(f).)
Regarding Claim 3
2A Prong 1:
wherein the feature selection process comprises defining a target function according to the plurality of global features or second features and a correlation with an expected outcome of the ML model being fed the plurality of global features or second features, and finding a minimum of the target function, the minimum representing the first subset or the second subset. (This step is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
2A Prong 2 & 2B: The claim does not recite any additional elements.
Regarding Claim 4
2A Prong 1: The claim does not recite an Abstract idea.
2A Prong 2 & 2B:
further comprising applying a quantum annealer based process for finding the minimum of the cost function. (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying quantum annealer as a tool to perform the abstract idea - see MPEP 2106.05(f).)
Regarding Claim 5
2A Prong 1: The claim does not recite an Abstract idea.
2A Prong 2:
wherein the error comprises at least two classification categories of a plurality of classification categories for which the ML model performance incorrect classification at a rate above a threshold (The specification of data to be stored is understood to be a field of use limitation. The limitation further specifies the error - See MPEP 2106.05(h).), wherein the input data is fed into the second ML model component when the first ML model component classifies input data into the at least two classification categories (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and data gathering. See MPEP 2106.05(g).), wherein the second ML model component classifies the input data into one of the at least two classification categories. (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying machine learning as a tool to perform the abstract idea - see MPEP 2106.05(f).)
2B:
wherein the error comprises at least two classification categories of a plurality of classification categories for which the ML model performance incorrect classification at a rate above a threshold (The specification of data to be stored is understood to be a field of use limitation. The limitation further specifies the error - See MPEP 2106.05(h).), wherein the input data is fed into the second ML model component when the first ML model component classifies input data into the at least two classification categories (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and is well understood, routine and conventional activity of transmitting and receiving data as identified by the court (MPEP2106.05(d)(ll)(i)))), wherein the second ML model component classifies the input data into one of the at least two classification categories. (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying machine learning as a tool to perform the abstract idea - see MPEP 2106.05(f).)
Regarding Claim 6
2A Prong 1:
wherein the at least two classification categories are merged into a single classification category, (This step is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
2A Prong 2 & 2B:
and the first ML model component is trained to classify the input data into the single classification category or other classification categories of the plurality of classification categories. (Training a machine learning model is understood as mere instructions to implement an abstract idea (e.g., generate inferences) on a computer - see MPEP 2106.05(f).))
Regarding Claim 7
2A Prong 1: The claim does not recite an Abstract idea.
2A Prong 2 & 2B:
wherein the second ML model component resolves ambiguity of the single classification category by classifying the input data into one of the at least two classification categories merged into the single classification category. (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying machine learning as a tool to perform the abstract idea - see MPEP 2106.05(f).)
Regarding Claim 8
2A Prong 1:
further comprising computing a confusion matrix of the ML model fed the first subset to identify the error. (This step is practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
2A Prong 2 & 2B: The claim does not recite any additional elements.
Regarding Claim 9
2A Prong 1:
further comprising: measuring a baseline classification performance of the ML model fed the plurality of global features; measuring the classification performance of the ML model fed the first subset; evaluating the classification performance of the ML model fed the first subset relative to the baseline classification performance to determine significant degradation in performance; and wherein the identification of the error in classification is in response to the determination of significant degradation in performance. (These steps are practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
2A Prong 2 & 2B: The claim does not recite any additional elements.
Regarding Claim 10
2A Prong 1:
further comprising: analyzing the classification performance of the second ML model component fed the second subset, to identify a second error in classification by the second ML model component; identifying a second subset of the sample dataset related to the second error; extracting a plurality of third features from the second subset of the sample data; applying the feature selection process for selecting a third subset of the plurality of third features; (These steps are practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
2A Prong 2 & 2B:
creating a third ML model component for inclusion in the reduced version of the ML model, the third ML model component trained by applying the third subset of features to the sample dataset, wherein input data is fed into the third ML model component when the second ML model component performs the second error in classification. (This step is adding the words “apply it” (or an equivalent) with the judicial exception, or merely applying machine learning as a tool to perform the abstract idea - see MPEP 2106.05(f).)
Regarding Claim 11
2A Prong 1:
further comprising: iterating the analyzing the classification performance, the identifying the subset, the extracting, the applying the feature selection process, and the creating the reduced version, for creating a hierarchical tree of ML model components, wherein each lower level ML model component is for resolving classification ambiguity of a higher level ML model component. (These steps are practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
2A Prong 2 & 2B: The claim does not recite any additional elements.
Regarding Claim 12
2A Prong 1: The claim does not recite an Abstract idea.
2A Prong 2 & 2B:
wherein the plurality of second features are the same as the plurality of global features. (The specification of data to be stored is understood to be a field of use limitation. The limitation further specifies the second features - See MPEP 2106.05(h).)
Regarding Claim 13
2A Prong 1:
extracting a first subset of features from the input data;
analyzing the first classification outcome to determine whether an error in classification occurred,
in response to determining the error, extracting a second subset of features from the input data; (These steps are practically performable in the human mind and is understood to be a recitation of a mental process (i.e., judgment/evaluation).)
2A Prong 2:
A computer implemented method of inference by a reduced version of a ML model, comprising: (The computer implemented method is understood to be generic computer elements – See MPEP 2106.05(f).)
obtaining input data; (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and data gathering. See MPEP 2106.05(g).)
feeding the first subset of features into a first ML model component of the reduced version of the ML model; (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and data gathering. See MPEP 2106.05(g).)
obtaining a first classification outcome from the first ML model component; (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and data gathering. See MPEP 2106.05(g).)
feeding the second subset of features into a second ML model component of the reduced version of the ML model; (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and data gathering. See MPEP 2106.05(g).)and
obtaining a second classification outcome comprising a resolution to the error of the first classification outcome from the second ML model, wherein the reduced version of the ML model is created according to claim 1. (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and data gathering. See MPEP 2106.05(g).)
2B:
A computer implemented method of inference by a reduced version of a ML model, comprising: (The computer implemented method is understood to be generic computer elements – See MPEP 2106.05(f).)
obtaining input data; (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and is well understood, routine and conventional activity of transmitting and receiving data as identified by the court (MPEP2106.05(d)(ll)(i))))
feeding the first subset of features into a first ML model component of the reduced version of the ML model; (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and is well understood, routine and conventional activity of transmitting and receiving data as identified by the court (MPEP2106.05(d)(ll)(i))))
obtaining a first classification outcome from the first ML model component; (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and is well understood, routine and conventional activity of transmitting and receiving data as identified by the court (MPEP2106.05(d)(ll)(i))))
feeding the second subset of features into a second ML model component of the reduced version of the ML model; (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and is well understood, routine and conventional activity of transmitting and receiving data as identified by the court (MPEP2106.05(d)(ll)(i)))) and
obtaining a second classification outcome comprising a resolution to the error of the first classification outcome from the second ML model, wherein the reduced version of the ML model is created according to claim 1. (This step is directed to transmitting or receiving information, which is understood to be insignificant extra-solution activity and is well understood, routine and conventional activity of transmitting and receiving data as identified by the court (MPEP2106.05(d)(ll)(i))))
Regarding Claim 14
2A Prong 1: The claim does not recite an Abstract idea.
2A Prong 2 & 2B:
wherein the error in classification occurred when the first classification outcome comprises a single classification category representing a merger of a plurality of different classification categories, wherein the first ML model component is unable to accurately classify into one of the plurality of different classification categories, wherein the second ML model component accurately classifies into one of the plurality of different classification categories. (The specification of data to be stored is understood to be a field of use limitation. The limitation further specifies the error - See MPEP 2106.05(h).)
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-2, 8-9, 12-13, and 15 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Brennan et al. (US 20180075368 A1, hereinafter "Brennan").
Regarding Claim 1
Brennan discloses: A computer implemented method of generating a reduced version of a ML model ([Para 0018, 0032, 0052, and Fig 1-2] discloses a computer and method for removing instances (i.e. reducing ML model).), comprising:
obtaining a sample dataset ([Para 0025-0028, Fig 1 and Fig 4] describes retrieving ground truth data to generate annotated ground truth data as training sets.);
extracting a plurality of global features from the sample dataset ([Para 0027, 0045, and Fig 4] describes the machine-annotated ground truth data which are clustered by feature similarity into cluster feature vectors (i.e. global features).);
applying a feature selection process for selecting a first subset of the plurality of global features ([Para 0044-0045 and Fig 4] describes employing feature selection algorithms as part of the machine analysis to train a model from the machine annotated entity/relationship instances. Also describes generating feature vectors from the training data.);
analyzing a classification performance of the ML model fed the first subset, to identify an error in classification by the ML model ([0046-0047 and Fig 4] describes identifying classes that are confused or misclassified and likely to have high error rates.);
identifying a subset of the sample dataset related to the error ([Para 0029, 0047-0048, and Fig 4] describes identifying misclassified entity/relationship instances (i.e. subset of the sample dataset related to the error).);
extracting a plurality of second features from the subset of the sample data; ([Para 0029, 0049, and Fig 4] describes generating misclassification feature vectors (i.e. extracting second features) from the identified misclassified machine-annotated training set instances.)
applying the feature selection process for selecting a second subset of the plurality of second features ([Para 0029, 0047-0048, and Fig 4] describes employs feature selection algorithms (e.g., sparse coding) on the misclassified entity/relationship instances to learn common features/characteristics of the misclassified examples that can be used to detect suspected misclassification errors on the clusters of entity/relation instances.); and
creating a reduced version of the ML model, comprising an ensemble of: a first ML model component trained by applying the first subset of the plurality of global features to the sample dataset, and a second ML model component trained by applying the second subset of the plurality of features to the subset of the sample data related to the error. ([0002, 0052, 0057, and Fig 4] describes flagging candidate erroneous training examples for possible reclassification or removal (i.e. creating a reduced version ML) with an ensemble comprising a first classifier and second classifier. The first classifier trained on feature from training data and the second classifier trained with misclassification feature from misclassified entity/relationship instances.)
Regarding Claim 15
Brennan discloses: A system for generating a reduced version of a ML model, comprising: at least one processor executing a code for ([Para 0015, 0018, 0032, 0052, and Fig 1-2] discloses a computer system for removing instances (i.e. reducing ML model).): (Claim 15 is a system claim that corresponds to claim 1 and the rest of the limitations are rejected on the same ground)
Regarding Claim 2
Brennan discloses: The computer implemented method of claim 1, wherein during inference, input data is first fed into the first ML model component to obtain a classification outcome, and in response to the classification outcome denoting the error in classification comprising ambiguity in the classification outcome, the input data is fed into the second ML model component to obtain a resolution to the classification outcome. ([Para 0046-0052, 0057, and Fig 4] describe a second classifier trained with misclassification feature vectors from misclassified entity/relationship instances of a first classifier.)
Regarding Claim 8
Brennan discloses: The computer implemented method of claim 1, further comprising computing a confusion matrix of the ML model fed the first subset to identify the error. ([Para 0038, 0046, 0050, and Fig 1-4] describes the confusion matrix to identify confusion/misclassification.)
Regarding Claim 9
Brennan discloses: The computer implemented method of claim 1, further comprising: measuring a baseline classification performance of the ML model fed the plurality of global features; measuring the classification performance of the ML model fed the first subset; evaluating the classification performance of the ML model fed the first subset relative to the baseline classification performance to determine significant degradation in performance; and wherein the identification of the error in classification is in response to the determination of significant degradation in performance. ([Para 0002, 0052, 0057, and Fig 4] describes flagging candidate erroneous training examples for possible reclassification or removal (i.e. creating a reduced version ML) with an ensemble comprising a first classifier and second classifier. The first classifier trained on feature from training data and the second classifier trained with misclassification feature from misclassified entity/relationship instances.)
Regarding Claim 12
Brennan discloses: The computer implemented method of claim 1, wherein the plurality of second features are the same as the plurality of global features. ([Para 0047-0050, and Fig 4] describes misclassification feature vectors are generated from the identified misclassified machine-annotated training set instances.)
Regarding Claim 13
Brennan discloses: A computer implemented method of inference by a reduced version of a ML model, comprising:
obtaining input data; ([Para 0025-0028, Fig 1, and Fig 4] describes retrieving ground truth data.)
extracting a first subset of features from the input data; ([Para 0027, 0045, and Fig 4] describes the machine-annotated ground truth data are clustered by feature similarity into cluster feature vectors (i.e. features).
feeding the first subset of features into a first ML model component of the reduced version of the ML model ([Para 0044-0045, and Fig 4] describes employing feature selection algorithms as part of the machine analysis to train a model from the machine annotated entity/relationship instances. Also describes generating feature vectors from the training data.);
obtaining a first classification outcome from the first ML model component ([Para 0002, 0044, 0052-0053, and Fig 4] describes a first classifier model classifying instances.);
analyzing the first classification outcome to determine whether an error in classification occurred ([0002, 0046-0047, and Fig 4] describes identifying classes that are confused or misclassified and likely to have high error rates.),
in response to determining the error, extracting a second subset of features from the input data ([Para 0029, 0047-0048, and Fig 4] describes identifying misclassified entity/relationship instances (i.e. subset of the sample dataset related to the error).);
feeding the second subset of features into a second ML model component of the reduced version of the ML model ([Para 0046-0050, and Fig 4] describes misclassification features to train the second classifier.); and
obtaining a second classification outcome comprising a resolution to the error of the first classification outcome from the second ML model, wherein the reduced version of the ML model is created according to claim 1. ([Para 0002, 0052-0053, 0057, and Fig 4] describes resolving erroneous training examples for possible reclassification or removal based on the second classifier and repeating steps.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 3-4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Brennan in view of Herbster et al. (US 20200005154 A1, hereinafter "Herbster")
Regarding Claim 3
Brennan discloses: The computer implemented method of claim 1,
Brennan does not explicitly disclose: wherein the feature selection process comprises defining a target function according to the plurality of global features or second features and a correlation with an expected outcome of the ML model being fed the plurality of global features or second features, and finding a minimum of the target function, the minimum representing the first subset or the second subset.
However, Herbster discloses in the same field of endeavor: wherein the feature selection process comprises defining a target function according to the plurality of global features or second features and a correlation with an expected outcome of the ML model being fed the plurality of global features or second features, and finding a minimum of the target function, the minimum representing the first subset or the second subset. ([Para 0068-0073] describes minimizing the value of a loss function thereby optimizing the accuracy of the updated feature vector generated.)
It would have been obvious to a person of ordinary skill in art before the effective filling date of the invention to implement the function of Data encoding and Classification disclosed by Herbster into the method for Confused Class prediction disclosed by Brennan to minimize a target function. The modification would have been obvious because one of the ordinary skills of the art would be motivated to utilize the feature of Data encoding and Classification disclosed by Herbster as all the references are in the field of machine learning. A person of ordinary skill of the art would have been motivated to perform the combination for being able to minimize the value of a loss function thereby optimizing the accuracy of updated feature vectors.
Regarding Claim 4
Brennan in view of Herbster discloses: The computer implemented method of claim 3, further comprising applying a quantum annealer based process for finding the minimum of the cost function. ([Para 0073], Herbster describes minimizing a loss function and quantum annealer.)
Claim(s) 5-7 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Brennan in view of Alon et al. (US 20220156524 A1, hereinafter "Alon").
Regarding Claim 5
Brennan discloses: The computer implemented method of claim 1, wherein the error comprises at least two classification categories of a plurality of classification categories for which the ML model performance incorrect classification ([Para 0042] describes the first classifier may implement a machine annotation process on a given input sentence statement to locate and classify named entities in the training set text into pre-defined categories.), wherein the input data is fed into the second ML model component when the first ML model component classifies input data into the at least two classification categories, wherein the second ML model component classifies the input data into one of the at least two classification categories ([Para 0038, and 0048] describes classifying with a second classifier model.).
Brennan does not explicitly disclose: the ML model performance incorrect classification at a rate above a threshold,
However, Alon discloses in the same field of endeavor: the ML model performance incorrect classification at a rate above a threshold ([Para 0004, 0035, 0063, and Fig 4] describes a sub-threshold correctness metric.), wherein the input data is fed into the second ML model component when the first ML model component classifies input data into the at least two classification categories, wherein the second ML model component classifies the input data into one of the at least two classification categories ([Para 0004, 0035, 0063, and Fig 4] describes applying the second machine learning model to the input to generate a second model output.)
It would have been obvious to a person of ordinary skill in art before the effective filling date of the invention to implement the function of Ensembles and Cascades disclosed by Alon into the method of Confused Class prediction disclosed by Brennan to disclose a classification threshold. The modification would have been obvious because one of the ordinary skills of the art would be motivated to utilize the feature of Ensembles and Cascades disclosed by Alon as all the references are in the field of machine learning. A person of ordinary skill of the art would have been motivated to perform the combination for being able to render an increase in a combined output accuracy for executing additional machine learning model(s).
Regarding Claim 6
Brennan in view of Alon discloses: The computer implemented method of claim 5, wherein the at least two classification categories are merged into a single classification category, and the first ML model component is trained to classify the input data into the single classification category or other classification categories of the plurality of classification categories.([Para 0021-0022, 0069, and 0075], Alon describes combining model outputs.)
Regarding Claim 7
Brennan in view of Alon discloses: The computer implemented method of claim 6, wherein the second ML model component resolves ambiguity of the single classification category by classifying the input data into one of the at least two classification categories merged into the single classification category. ([Para 0021-0022, 0069, and 0075], Alon describes combining model outputs.)
Regarding Claim 14
Brennan in view of Alon discloses: The computer implemented method of claim 13, wherein the error in classification occurred when the first classification outcome comprises a single classification category representing a merger of a plurality of different classification categories ([Para 0021-0022, 0069, and 0075], Alon describes combining model outputs.), wherein the first ML model component is unable to accurately classify into one of the plurality of different classification categories, wherein the second ML model component accurately classifies into one of the plurality of different classification categories. ([Para 0035] Alon describes A second, conditionally-executed model trained only on those examples in the training dataset for which the first model performed poorly.)
Claim(s) 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Brennan in view of Farhadi et al. (US 20220067453 A1, hereinafter "Farhadi").
Regarding Claim 10
Brennan discloses: The computer implemented method of claim 1,
Brennan does not explicitly disclose: further comprising: analyzing the classification performance of the second ML model component fed the second subset, to identify a second error in classification by the second ML model component; identifying a second subset of the sample dataset related to the second error; extracting a plurality of third features from the second subset of the sample data; applying the feature selection process for selecting a third subset of the plurality of third features; and creating a third ML model component for inclusion in the reduced version of the ML model, the third ML model component trained by applying the third subset of features to the sample dataset, wherein input data is fed into the third ML model component when the second ML model component performs the second error in classification.
However, Farhadi discloses in the same field of endeavor: further comprising: analyzing the classification performance of the second ML model component fed the second subset, to identify a second error in classification by the second ML model component; identifying a second subset of the sample dataset related to the second error; extracting a plurality of third features from the second subset of the sample data; applying the feature selection process for selecting a third subset of the plurality of third features; and creating a third ML model component for inclusion in the reduced version of the ML model, the third ML model component trained by applying the third subset of features to the sample dataset, wherein input data is fed into the third ML model component when the second ML model component performs the second error in classification. ([Para 0085, Claims 14-15, and Fig 6] describes a third neural network for deeper feature extractions.)
It would have been obvious to a person of ordinary skill in art before the effective filling date of the invention to implement the function of Hierarchical Neural networks disclosed by Farhadi into the method of Confused Class prediction disclosed by Brennan to “limitation that is rejected using new reference”. The modification would have been obvious because one of the ordinary skills of the art would be motivated to utilize the feature of Hierarchical Neural networks disclosed by Farhadi as all the references are in the field of machine learning. A person of ordinary skill of the art would have been motivated to perform the combination for being able to utilize deeper models to gain a higher accuracy.
Regarding Claim 11
Brennan in view of Farhadi discloses: The computer implemented method of claim 1, further comprising: iterating the analyzing the classification performance, the identifying the subset, the extracting, the applying the feature selection process, and the creating the reduced version, for creating a hierarchical tree of ML model components, wherein each lower level ML model component is for resolving classification ambiguity of a higher level ML model component. ([Para 0006, 0028, 0085, Claims 13-15, and Fig 6] Farhadi describes hierarchical convolutional neural networks cascading down from a first neural network.)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Ghanta et al. (US 2020/0034665 A1) describes determining machine learning model validity. Valpola (US 2020/0151547 A1 ) describes evaluating model uncertainty.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TEWODROS E MENGISTU whose telephone number is (571)270-7714. The examiner can normally be reached Mon-Fri 9:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ABDULLAH KAWSAR can be reached at (571)270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TEWODROS E MENGISTU/ Examiner, Art Unit 2127
Prosecution Insights