Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1–20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Analysis is provided for the claims under the guidelines of MPEP 2106.
Regarding claim 1:
Step 1:
The claim recites “[a] computer-implemented method comprising” the steps that follow. Thus, the claim is to a process, which is a statutory category of invention.
Step 2A prong 1:
The limitation “determining, for each class and each output of the predictions of the multi-output multi-label ML model and based on the truth data, false negative values (FN), false positive values (FP), true negative values (TN), and true positive values (TP)” recites a mental process. A person could determine the false negative values, false positive values, true negative values, and true positive values from predictions and truth values using observation.
The limitation “generating a micro value for a performance metric based on aggregated values associated with the FN, the FP, the TN, and the TP” recites a mathematical concept. A micro value for a performance metric based on aggregated values associated with false negative values, false positive values, true negative values, and true positive values could be generated using mathematical calculations.
The limitation “generating a macro value for the performance metric based on averaged values of the FN, the FP, the TN, and the TP” recites a mathematical concept. A macro value for the performance metric based on averaged values of the false negative values, false positive values, true negative values, and true positive values could be generated using mathematical calculations.
Thus, the claim recites an abstract idea.
Step 2A prong 2:
The further element “receiving data about a multi-output, multi-label machine-learning (ML) model, the data comprising predictions made by the multi-output, multi-label ML model and truth data associated with the predictions, the truth data comprising information about a ground truth of corresponding predictions of the predictions” recites mere data gathering, which is insignificant extra-solution activity (MPEP 2106.05(g)).
The further element “outputting the performance metric for controlling the multi-output, multi-label ML model based at least in part upon the performance metrics” recites mere data output, which is insignificant extra-solution activity (MPEP 2106.05(g)).
Thus, the additional elements merely recite insignificant extra-solution activity. Taken alone, the additional elements do not integrate the abstract idea into a practical application. Considering the elements together as an ordered combination adds nothing that is not present from examining the elements individually. The elements, individually or together, do not describe an improvement in the functioning of technology.
Step 2B:
The claim as a whole does not amount to significantly more than the recited judicial exception.
The element “receiving data about a multi-output, multi-label machine-learning (ML) model, the data comprising predictions made by the multi-output, multi-label ML model and truth data associated with the predictions, the truth data comprising information about a ground truth of corresponding predictions of the predictions” recites mere data gathering, which is recognized as well-understood, routine, and conventional activity in the art (see MPEP § 2106.05(d)(II)(i)).
The element “outputting the performance metric for controlling the multi-output, multi-label ML model based at least in part upon the performance metrics” recites mere data transmission, which is recognized as well-understood, routine, and conventional activity in the art (see MPEP § 2106.05(d)(II)(i)).
Even when considered in combination, the additional elements represent insignificant extra-solution activity, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 2:
For step 2A prong 1, claim 2 further limits claim 1 and the same elements in claim 2 still recite an abstract idea.
The further limitations “wherein the predictions comprise a plurality of outputs” and “wherein the plurality of classes for an output included in the plurality of outputs is different from a plurality of classes associated with one or more other outputs included in the plurality of outputs” further limit the data used in the abstract idea of the claim, but it remains an abstract idea.
For step 2A prong 2, the further element “wherein, for each output included in the plurality of outputs, the multi-output, multi-label ML model selects a class prediction for the output from a plurality of classes associated with the output” recites model prediction at a high level of generality. No particular model or method of prediction is described. The element thus merely recites the use of a computer as a tool to perform the abstract idea, and is equivalent to adding the words “apply it” or the equivalent to the judicial exception (MPEP 2106.05(f)).
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The additional element “wherein, for each output included in the plurality of outputs, the multi-output, multi-label ML model selects a class prediction for the output from a plurality of classes associated with the output” recites mere instructions to apply the abstract idea.
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer or represent insignificant extra-solution activity, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 3:
For step 2A prong 1, claim 3 further limits claim 2 and the same elements in claim 3 still recite an abstract idea.
The limitations “wherein receiving the data about the multi-output, multi-label ML model comprises: receiving data for each of a plurality of datapoints, wherein the data for each datapoint included in the plurality of datapoints comprises information included in the predictions and identifying a class predicted by the multi-output, multi-label ML model for each output included the plurality of outputs and information included in the truth data and identifying a ground truth class for each output included the plurality of outputs” further limits the data received for use in the abstract idea of the claim, but it remains an abstract idea.
The limitation “partitioning the received data” recites a mental process. A person could partition data using observation and judgement, with the aid of pencil and paper. Thus, the limitation adds to the abstract idea.
For step 2A prong 2, and step 2B, no further elements remain to be considered. The claim as a whole does not amount to significantly more than the recited judicial exception and is ineligible under 35 U.S.C. 101.
Regarding claim 4:
For step 2A prong 1, claim 4 further limits claim 3 and the same elements in claim 4 still recite an abstract idea.
The limitation “wherein partitioning the received data comprises: partitioning the data received for the multi-output, multi-label ML model into at least a first partition and a second partition, the first partition comprising data for a first set of datapoints from the plurality of datapoints and the second partition comprising data for a second set of datapoints from the plurality of datapoints” further limits the partitioning mental process of the claim, but it remains a mental process.
The limitations “computing, by a first processing system, a first confusion matrix and a first set of values based upon the first partition received by the first processing system; computing, by a second processing system, a second confusion matrix and a second set of values based upon the second partition received by the second processing system” recite a mental process. A person could compute a confusion matrix using observation and judgement, with the aid of pencil and paper.
The limitation “generating a merged confusion matrix by merging the first confusion matrix and the second confusion matrix” recites a mental process. A person could merge confusion matrices using judgement.
Thus, the limitations add to the abstract idea.
For step 2A prong 2, and step 2B, no further elements remain to be considered. The claim as a whole does not amount to significantly more than the recited judicial exception and is ineligible under 35 U.S.C. 101.
Regarding claim 5:
For step 2A prong 1, claim 5 further limits claim 4 and the same elements in claim 5 still recite an abstract idea.
The limitation “using the merged confusion matrix to determine the FN, the FP, the TN, and the TP for each output and for each class” recites a mental process. A person could determine the false negative values, false positive values, true negative values, and true positive values for each output and class using a confusion matrix, using observation, with the aid of pencil and paper.
Thus, the limitation adds to the abstract idea.
For step 2A prong 2, and step 2B, no further elements remain to be considered. The claim as a whole does not amount to significantly more than the recited judicial exception and is ineligible under 35 U.S.C. 101.
Regarding claim 6:
For step 2A prong 1, claim 6 further limits claim 4 and the same elements in claim 6 still recite an abstract idea.
For step 2A prong 2, the further element “wherein at least a portion of the computing performed by the second processing system is performed substantially contemporaneously with respect to the computing performed by the first processing system” recites mere parallel computation at a high level of generality. No particular method of contemporaneous performance is described. The limitation amounts to insignificant extra-solution activity under 2106.04(d) that fails to integrate the abstract idea into a practical application.
For step 2B, the claim as a whole does not amount to significantly more than the recited judicial exception. The further element “wherein at least a portion of the computing performed by the second processing system is performed substantially contemporaneously with respect to the computing performed by the first processing system” recites mere parallel computation at a high level of generality, which is recognized as well-understood, routine, and conventional activity in the art (see Wikipedia, “Parallel Computing,” revision of 09/29/2023, “Parallel computing is a type of computation in which many calculations or processes are carried out simultaneously.[1] Large problems can often be divided into smaller ones, which can then be solved at the same time. There are several different forms of parallel computing: bit-level, instruction-level, data, and task parallelism. Parallelism has long been employed in high-performance computing, but has gained broader interest due to the physical constraints preventing frequency scaling.[2] As power consumption (and consequently heat generation) by computers has become a concern in recent years,[3] parallel computing has become the dominant paradigm in computer architecture, mainly in the form of multi-core processors.[4]”).
Even when considered in combination, the additional elements represent mere instructions to apply the abstract idea to a computer or represent insignificant extra-solution activity, which do not provide an inventive concept. The claim is not eligible under 35 U.S.C. 101.
Regarding claim 7:
For step 2A prong 1, claim 7 further limits claim 1 and the same elements in claim 7 still recite an abstract idea.
The limitation “generating the micro value comprises calculating the micro value for the metric by aggregating, by class, the FN, the FP, the TN, and the TP” recites a mathematical concept. A micro value can be generated by aggregating, by class, false negative values, false positive values, true negative values, and true positive values using mathematical calculations.
The limitation “macro value comprises (i) calculating metric values for each class of each output using the FN, the FP, the TN, and the TP and (ii) averaging the metric values” recites a mathematical concept. A macro value can be generated using metrics from false negative values, false positive values, true negative values, and true positive values, and averaging those metrics, using mathematical calculations.
Thus, the limitations add to the abstract idea.
For step 2A prong 2, and step 2B, no further elements remain to be considered. The claim as a whole does not amount to significantly more than the recited judicial exception and is ineligible under 35 U.S.C. 101.
Regarding claims 8–14:
These claims are to “[a] system comprising: one or more processors; and a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions comprising instructions executable by the one or more processors to cause the one or more processors to perform operations comprising” the steps that follow. Thus, the claims are to a machine, which is a statutory category of invention. The claims are otherwise analogous to claims 1–7, respectively, and are found ineligible by the same arguments.
Regarding claims 15–18 and 20:
These claims are to “[a] non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, comprising” the steps that follow. Thus, the claims are to a manufacture, which is a statutory category of invention. The claims are otherwise analogous to claims 1–4 and 7, respectively, and are found ineligible by the same arguments.
Regarding claim 19:
Claim 19 is analogous to claim 5 with addition of the limitation of claim 6, and is rejection by analogous arguments.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1–2, 8–9 and 15–16 rejected under 35 U.S.C. 102(a) (1) as being anticipated by Jeong, US Pre-Grant Publication No. 2022/0076081 (hereafter Jeong).
Regarding claim 1 and analogous claims 8 and 15:
Jeong teaches:
“[a] computer-implemented method comprising”: Jeong, paragraph 0006, “In certain example embodiments, a computer system is provided that is designed to handle multi-label classification. The computer system includes multiple processing instances [computer-implemented method] that share a common code base (e.g., a single module may be reused as the basis for the distinct instances), but each processing instance may run a different model that is individually trained to classify one or more labels to an input dataset (such as a group of documents). Each model may be trained to make different classification predictions”; Jeong, paragraph 0095, “Memory devices 604 are examples of non-transitory computer-readable storage media.”
“receiving data about a multi-output, multi-label machine-learning (ML) model, the data comprising predictions made by the multi-output, multi-label ML model and truth data associated with the predictions, the truth data comprising information about a ground truth of corresponding predictions of the predictions”: Jeong, paragraph 0020, "In certain example embodiments, a computer system is provided that is designed to handle multi-label classification. The computer system includes multiple processing instances that may share a common code base (e.g., each of the processing instances may be distinct instantiations of the same software module or software code). The multiple processing instances may each execute or process a different model that has been individually trained to classify one or more labels to an input dataset [a multi-output, multi-label machine-learning (ML) model] (such as a group of documents). The processing instances of the computer system may be arranged in a hierarchical or other structured manner such that the output from one processing instance (and the corresponding model) may be used as input for another. Thus, the classification performed by one model may rely on the results of classification performed by another model. In certain example embodiments, each of the models may use a different threshold value that is used to determine, from the probability output from the classification model, whether a label should be assigned to a given member within a dataset ( e.g., a document or the like)"; Jeong, paragraph 0009, "In certain example embodiments, models may be continually updated as datasets are labeled and then subsequently verified. The verified and labeled data may be used to train (or retrain) future models”; Jeong, paragraph 0047, “Results 306 are then passed to a predication validation process 116A. In certain example embodiments, this may be a manual process where the results are reviewed by a human that validates the confirmed instances of the label (e.g., documents 1-14 from the above table). The validation performed at 116A is used to produce a dataset 310, which are instances where the label has been confirmed to be applied against the corresponding documents [comprising predictions made by the multi-output, multi-label ML model and truth data associated with the predictions, the truth data comprising information about a ground truth of corresponding predictions of the predictions].”
“determining, for each class and each output of the predictions of the multi-output multi-label ML model and based on the truth data, false negative values (FN), false positive values (FP), true negative values (TN), and true positive values (TP)”: Jeong, paragraph 0036, "The following metrics may be calculated form the above confusion matrix, with TP being true positives, TN being true negatives, FN being false negatives, and FP being false positives [false negative values (FN), false positive values (FP), true negative values (TN), and true positive values (TP)]."
“generating a micro value for a performance metric based on aggregated values associated with the FN, the FP, the TN, and the TP”: Jeong, paragraph 0034, "When machine learned models (e.g., that are used to make predications) are trained, the effectiveness of such models may be judged based on the 'recall' metric. Recall is the number of true positives divided by the sum of the true positives and false negatives (TP/(TP+FN))"; Jeong, Table 3,
PNG
media_image1.png
248
476
media_image1.png
Greyscale
[showing performance metrics]; Jeong, Table 4,
PNG
media_image2.png
224
482
media_image2.png
Greyscale
[showing micro value that is based on aggregated values associated with the FN, the FP, the TN, and the TP];
“generating a macro value for the performance metric based on averaged values of the FN, the FP, the TN, and the TP”: Jeong, Table 3,
PNG
media_image1.png
248
476
media_image1.png
Greyscale
[showing performance metrics that are based on averages, including recall, precision, and balanced accuracy]; Jeong, Table 5,
PNG
media_image3.png
228
476
media_image3.png
Greyscale
[showing macro value for the performance metric based on averaged values of the FN, the FP, the TN, and the TP];
“outputting the performance metric for controlling the multi-output, multi-label ML model based at least in part upon the performance metrics”: Jeong, paragraph 0050, "Model 302 may also be archived to storage 102. This may also include any errors or other data generated during processing of a model with new data. Model validation process 114A checks the models and generates an error log 308 for any inconsistencies that may be generated by the model 302. For example, by checking the performance metrics of the model ( e.g., by assessing its recall metric or other metric). If the model does not meet or exceed performance expectations then an error may be thrown and written to a log file for further follow-up [outputting the performance metric for controlling the multi-output, multi-label ML model based at least in part upon the performance metrics]."
Regarding claim 2 and analogous claims 9 and 16:
Jeong teaches “[t]he computer-implemented method of claim 1.”
Jeong further teaches:
“wherein the predictions comprise a plurality of outputs, wherein, for each output included in the plurality of outputs, the multi-output, multi-label ML model selects a class prediction for the output from a plurality of classes associated with the output”: Jeong, paragraph 0020, "In certain example embodiments, a computer system is provided that is designed to handle multi-label classification. The computer system includes multiple processing instances [hence, providing a plurality of outputs] that may share a common code base (e.g., each of the processing instances may be distinct instantiations of the same software module or software code). The multiple processing instances may each execute or process a different model that has been individually trained to classify one or more labels to an input dataset [selects a class prediction for the output from a plurality of classes associated with the output] (such as a group of documents). The processing instances of the computer system may be arranged in a hierarchical or other structured manner such that the output from one processing instance (and the corresponding model) may be used as input for another. Thus, the classification performed by one model may rely on the results of classification performed by another model. In certain example embodiments, each of the models may use a different threshold value that is used to determine, from the probability output from the classification model, whether a label should be assigned to a given member within a dataset ( e.g., a document or the like).”
“wherein the plurality of classes for an output included in the plurality of outputs is different from a plurality of classes associated with one or more other outputs included in the plurality of outputs”: Jeong, paragraph 0031, “FIG. 2A provides an illustrative example of how multiple labels for an example process may be arranged. The documents may be split into positive and negative labels e.g., those labeled with a mandate signal and those labeled with a non-mandate signal. In certain instances, these may be mutually exclusive. A document with a mandate signal may be further classified based on specific characteristics of the mandate(s) identified in the document, for example whether an identified mandate is (1) "potential" (i.e., it is not certain that the action will be taken, but it is possible), (2) "announced" (i.e., it has been announced that the action will be taken), (3) "ongoing" (i.e., the action is in process), or (4) "closed" (i.e., the action has been completed) [wherein the plurality of classes for an output included in the plurality of outputs is different from a plurality of classes associated with one or more other outputs included in the plurality of outputs].”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3–6, 10–13, and 17–19 rejected under 35 U.S.C. 103 over Jeong in view of Bonissone et al., US Pre-Grant Publication No. 2004/0220840 (hereafter Bonissone).
Regarding claim 3 and analogous claims 10 and 17:
Jeong teaches “[t]he computer-implemented method of claim 2.”
Jeong further teaches:
“wherein receiving the data about the multi-output, multi-label ML model comprises: receiving data for each of a plurality of datapoints, wherein the data for each datapoint included in the plurality of datapoints comprises: information included in the predictions and identifying a class predicted by the multi-output, multi-label ML model for each output included the plurality of outputs”: Jeong, paragraph 0008, "In certain example embodiments, each of the models [for each output included the plurality of outputs] may be assigned with a corresponding threshold value that is used to determine, from the probability output from the classification model, whether a label should be assigned to a given piece of data within a dataset ( e.g., each member of that dataset, such as a document, image, or the like)"; Jeong, paragraph 0006, "In certain example embodiments, a computer system is provided that is designed to handle multi-label classification. The computer system includes multiple processing instances that share a common code base (e.g., a single module may be reused as the basis for the distinct instances), but each processing instance may run a different model that is individually trained to classify one or more labels to an input dataset (such as a group of documents) [receiving data for each of a plurality of datapoints]. Each model may be trained to make different classification predictions [information included in the predictions and identifying a class predicted by the multi-output, multi-label ML model ]."
“information included in the truth data and identifying a ground truth class for each output included the plurality of outputs”: Jeong, paragraph 0047, “Results 306 are then passed to a predication validation process 116A. In certain example embodiments, this may be a manual process where the results are reviewed by a human that validates the confirmed instances of the label (e.g., documents 1-14 from the above table) [information included in the truth data and identifying a ground truth class for each output included the plurality of outputs]. The validation performed at 116A is used to produce a dataset 310, which are instances where the label has been confirmed to be applied against the corresponding documents."
Jeong does not explicitly teach “partitioning the received data.”
Bonissone teaches “partitioning the received data”: Bonissone, paragraph 0242, “At step 4220, five-fold partitioning and resampling occurs, while a development and validation set is generated at step 4225. According to an embodiment of the invention, a stratified sampling methodology may be used to partition the data set into five equal parts [partitioning the received data].”
Bonissone and Jeong are analogous arts as they are both related to improving models through confusion matrices. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the SOMETHING of Bonissone with the teachings of Jeong to arrive at the present invention, in order to process the data in parallel to improve performance, as stated in Bonissone, paragraph 0013, “[…] where the at least one resampling technique further comprises partitioning data from the previous insurance applications and their associated underwriting decisions into five groups of equal size, removing one of the five groups and combining the remaining four groups in a development sample, code for modifying the plurality of binary classifiers based on the performance of the at least one cross-validation technique and at least one re-sampling technique, code for utilizing the validated parallel network for outputting a classification assignment for the at least one new insurance application, and code for fusing the classification assignment for the at least one insurance application with at least one other classification assignment for the insurance application, where the at least one other classifier is generated by at least one other classifier.”
Regarding claim 4 and analogous claims 11 and 18:
Jeong as modified by Bonissone teaches “[t]he computer-implemented method of claim 3.”
Jeong further teaches (bold only) “wherein partitioning the received data comprises: partitioning the data received for the multi-output, multi-label ML model into at least a first partition and a second partition, the first partition comprising data for a first set of datapoints from the plurality of datapoints and the second partition comprising data for a second set of datapoints from the plurality of datapoints”: Jeong, paragraph 0020, "In certain example embodiments, a computer system is provided that is designed to handle multi-label classification. The computer system includes multiple processing instances that may share a common code base (e.g., each of the processing instances may be distinct instantiations of the same software module or software code). The multiple processing instances may each execute or process a different model that has been individually trained to classify one or more labels to an input dataset [multi-output, multi-label ML model] (such as a group of documents).”
Bonissone further teaches:
(bold only) “wherein partitioning the received data comprises: partitioning the data received for the multi-output, multi-label ML model into at least a first partition and a second partition, the first partition comprising data for a first set of datapoints from the plurality of datapoints and the second partition comprising data for a second set of datapoints from the plurality of datapoints”: Bonissone, paragraph 0242, “At step 4220, five-fold partitioning and resampling occurs, while a development and validation set is generated at step 4225. According to an embodiment of the invention, a stratified sampling methodology may be used to partition the data set into five equal parts [partitioning the data received for the … model into at least a first partition and a second partition, the first partition comprising data for a first set of datapoints from the plurality of datapoints and the second partition comprising data for a second set of datapoints from the plurality of datapoints].”
“computing, by a first processing system, a first confusion matrix and a first set of values based upon the first partition received by the first processing system; computing, by a second processing system, a second confusion matrix and a second set of values based upon the second partition received by the second processing system; and generating a merged confusion matrix by merging the first confusion matrix and the second confusion matrix”: Bonissone, paragraph 0310. “The combined confusion matrices of the five-fold runs are illustrated in FIG. 52. For comparison, the combined confusion matrices for the five-fold runs after post processing are illustrated in FIG. 53 [generating a merged confusion matrix by merging the first confusion matrix and the second confusion matrix from a first confusion matrix and a first set of values and a second confusion matrix and a second set of values]. The performance for this example before post-processing is provided in FIG. 54, while the performance for this example after post-processing is provided in FIG. 55”; Bonissone, paragraph 0013, “An additional exemplary embodiment of the present invention involves a computer readable medium having code for causing a processor to underwrite an insurance application based on a plurality of previous insurance application underwriting decisions, where the medium comprises code for digitizing the insurance application and the plurality of previous insurance application underwriting decisions, code for generating a casebase of the plurality of previous insurance application underwriting decisions, code for creating a plurality of binary classifiers [computing, by a first processing system][computing, by a second processing system] based on a structured methodology of multivariate adaptive regression splines ("MARS"), where the plurality of binary classifiers are arranged in a parallel network, code for identifying a relevant set of MARS variables and parameters based on the plurality of previous insurance applications and their associated underwriting decisions, code for performing at least one cross-validation technique and at least one re-sampling technique on the plurality of previous insurance applications and their associated underwriting decisions, where the at least one resampling technique further comprises partitioning data from the previous insurance applications and their associated underwriting decisions into five groups of equal size, removing one of the five groups and combining the remaining four groups in a development sample, code for modifying the plurality of binary classifiers based on the performance of the at least one cross-validation technique and at least one re-sampling technique, code for utilizing the validated parallel network for outputting a classification assignment for the at least one new insurance application, and code for fusing the classification assignment for the at least one insurance application with at least one other classification assignment for the insurance application, where the at least one other classifier is generated by at least one other classifier.”
Bonissone and Jeong are combinable for the rationale given under claim 3.
Regarding claim 5 and analogous claim 12:
Jeong as modified by Bonissone teaches “[t]he computer-implemented method of claim 4.”
Jeong further teaches “further comprising using the merged confusion matrix to determine the FN, the FP, the TN, and the TP for each output and for each class”: Jeong, Table 2,
PNG
media_image4.png
176
480
media_image4.png
Greyscale
; Jeong, paragraph 0036, “The following metrics may be calculated form [sic] the above confusion matrix, with TP being true positives, TN being true negatives, FN being false negatives, and FP being false positives […] [using the merged confusion matrix to determine the FN, the FP, the TN, and the TP for each output and for each class].”
Regarding claim 6 and analogous claim 13:
Jeong as modified by Bonissone teaches “[t]he computer-implemented method of claim 4.”
Bonissone further teaches “wherein at least a portion of the computing performed by the second processing system is performed substantially contemporaneously with respect to the computing performed by the first processing system”: Bonissone, paragraph 0248, “According to another approach, a ‘parallel network’ arrangement of models may be used. A parallel network arrangement is a collection of MARS models, each of which solves a binary, or two-class problem [wherein at least a portion of the computing performed by the second processing system is performed substantially contemporaneously with respect to the computing performed by the first processing system]. This may take advantage of the fact that the response variable is ordinal e.g., the decision classes being risk categories are increasing in risk.”
Bonissone and Jeong are combinable for the rationale given under claim 3.
Regarding claim 19:
Jeong as modified by Bonissone teaches “[t]he non-transitory computer-readable memory of claim 18.”
Jeong further teaches “wherein the operations further comprise using the merged confusion matrix to determine the FN, the FP, the TN, and the TP for each output and for each class”: Jeong, Table 2,
PNG
media_image4.png
176
480
media_image4.png
Greyscale
; Jeong, paragraph 0036, “The following metrics may be calculated form [sic] the above confusion matrix, with TP being true positives, TN being true negatives, FN being false negatives, and FP being false positives […] [using the merged confusion matrix to determine the FN, the FP, the TN, and the TP for each output and for each class].”
Bonissone further teaches “wherein at least a portion of the computing performed by the second processing system is performable substantially contemporaneously with respect to the computing performed by the first processing system”: Bonissone, paragraph 0248, “According to another approach, a ‘parallel network’ arrangement of models may be used. A parallel network arrangement is a collection of MARS models, each of which solves a binary, or two-class problem [wherein at least a portion of the computing performed by the second processing system is performable substantially contemporaneously with respect to the computing performed by the first processing system]. This may take advantage of the fact that the response variable is ordinal e.g., the decision classes being risk categories are increasing in risk.”
Bonissone and Jeong are combinable for the rationale given under claim 10.
Claim 7 and analogous claims 14 and 20 rejected under 35 U.S.C. 103 over Jeong as modified by Bonissone in view of Grandini et al., “Metrics for Multi-Class Classification: An Overview,” 2020, arXiv:2008.05756v1 (hereafter Grandini).
Jeong as modified by Bonissone teaches “[t]he computer-implemented method of claim 4.”
Jeong as modified by Bonissone does not explicitly teach “generating the micro value comprises calculating the micro value for the metric by aggregating, by class, the FN, the FP, the TN, and the TP; and generating the macro value comprises (i) calculating metric values for each class of each output using the FN, the FP, the TN, and the TP and (ii) averaging the metric values.”
Grandini teaches:
“generating the micro value comprises calculating the micro value for the metric by aggregating, by class, the FN, the FP, the TN, and the TP”: Grandini, section 4.3, paragraphs 1–2, “In order to obtain Micro F1-Score, we need to compute Micro-Precision and Micro-Recall before. The idea of Micro-averaging is to consider all the units together, without taking into consideration possible differences between classes. Therefore, the Micro-Average Precision is computed as follows:
PNG
media_image5.png
52
588
media_image5.png
Greyscale
[showing that the micro average precision depends on the total values of all columns in the confusion matrix, hence, the FN, the FP, the TN, and the TP]”; Grandini, section 4.3, paragraph 5, “All in all, we may regard the Macro F1-Score as an average measure of the average precision and average recall of the classes. This measure is calculated at class level, so that each class has the same weight [aggregating, by class]. Small classes are equivalent to big ones and the algorithm performance on them is equally important, regardless of the class size.”
“generating the macro value comprises (i) calculating metric values for each class of each output using the FN, the FP, the TN, and the TP and (ii) averaging the metric values”: Grandini, section 4.2.1, paragraphs 1–3, “In order to obtain Macro F1-Score, we need to compute Macro-Precision and Macro-Recall before. They are respectively calculated by taking the average precision for each predicted class and the average recall for each actual class [averaging the metric values]. Hence, the Macro approach considers all the classes as basic elements of the calculation: each class has the same weight in the average, so that there is no distinction between highly and poorly populated classes. For the required computations, we will use the Confusion Matrix focusing on one class at a time and labelling the tiles accordingly. In particular, we consider True Positive (TP) as the only correctly classified units for our class, whereas False Positive (FP) and False Negative (FN) are the wrongly classified elements on the column and the row of the class respectively. True Negative (TN) are all the other tiles, as shown in Figure 4 where we are considering the class ‘b’ as reference focus [calculating metric values for each class of each output using the FN, the FP, the TN, and the TP]. When we switch from one class to another one, we compute the quantities again and the labels for the Confusion Matrix tiles are changed accordingly.”
Grandini and Jeong are analogous arts as they are both related to multi-class metrics. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the metric calculations of Grandini with the teachings of Jeong to arrive at the present invention, in order to measure model performance, as stated in Grandini, Abstract, “Those metrics turn out to be useful at different stage of the development process, e.g. comparing the performance of two different models or analysing the behaviour of the same model by tuning different parameters.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Khosla et al., US Pre-Grant Publication No. 2008/0235318, discloses a classification method that includes a hierarchy of classifiers producing multiple confusion matrices that are unified into a final confusion matrix.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT SPRAUL whose telephone number is (703) 756-1511. The examiner can normally be reached M-F 9:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MICHAEL HUNTLEY can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VAS/Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights