DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Preliminary Amendment
The preliminary amendment filed on 2023/12/03 has been entered. Claims 1-20 are pending in the application.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 2023/12/03. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (an abstract idea) without reciting significantly more.
Regarding independent claims 1, 10, and 16
Step 1 -- whether the claim falls within any statutory category. See MPEP 2106.03
Claim 1 is drawn to a device claim, claim 5 is drawn to a method claim, and claim 6 is drawn to a computer-readable non-transitory recording medium claim. Therefore, each of independent claims 1, 5, and 6 falls under one of the four categories of statutory subject matter: process/method, machine/product/apparatus, manufacture, or composition of matter. Accordingly, claims 1, 5, and 6 satisfy Step 1 of the subject matter eligibility analysis.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 1, claim 1 is drawn to an error determination device comprising a processor configured to execute operations including acquiring an estimated classification of data to be classified, generating an estimation process feature vector on a basis of the data, generating an estimated probability vector, determining whether a classification result of the estimated classification of data is correct or incorrect on a basis of the estimated probability vector, and outputting the classification result, a determination result, and the estimated probability vector.
Under MPEP 2106.04, subsection II, Step 2A Prong One asks whether the claim recites a judicial exception, i.e., whether an abstract idea, law of nature, or natural phenomenon is set forth or described in the claim. If the identified limitation falls within at least one of the abstract idea groupings, it is reasonable to conclude that the claim recites an abstract idea. The relevant abstract idea groupings here are mathematical concepts under MPEP 2106.04(a)(2)(I) and mental processes under MPEP 2106.04(a)(2)(III). Mathematical concepts include mathematical relationships, mathematical formulas or equations, and mathematical calculations. Mental processes include concepts performed in the human mind, including observations, evaluations, judgments, and opinions.
The limitation of “generating an estimation process feature vector on a basis of the data” recites forming a vector representation from data. Under the broadest reasonable interpretation in light of the specification, an estimation process feature vector includes numerical vector information generated from data or from an estimation process. The specification describes feature vectors as including, for example, a numerical vector, a vector of estimated probabilities, or a logit vector used in the estimation process. Thus, this limitation recites organizing or converting data into a vector representation, which is a mathematical representation and falls within the mathematical concepts grouping under MPEP 2106.04(a)(2)(I). To the extent the limitation broadly encompasses selecting or identifying features from data for inclusion in the vector, it also encompasses observation or evaluation that can practically be performed in the human mind, and therefore also falls within the mental processes grouping under MPEP 2106.04(a)(2)(III).
The limitation of “generating an estimated probability vector, wherein the estimated probability vector indicates probabilities each of which is a probability that the data to be classified belongs to one of classes on a basis of the estimation process feature vector” recites generating numerical probability values arranged as a vector. A probability value is a numerical value representing a likelihood, and the recited vector indicates probabilities corresponding to classes. Under MPEP 2106.04(a)(2)(I), a limitation that recites determining numerical values or mathematical relationships is a mathematical concept. This limitation therefore recites a mathematical concept because it requires generating numerical probabilities from the estimation process feature vector. This is analogous to the AI examples in which converting feature representations into embedding vectors or calculating values from vectors is treated as mathematical calculation when the claim recites the mathematical generation of vectorized numerical information.
The limitation of “determining whether a classification result of the estimated classification of data is correct or incorrect on a basis of the estimated probability vector” recites evaluating a classification result using the estimated probability vector. This limitation recites a mental process because determining whether a classification result is correct or incorrect based on probability values is an evaluation or judgment that can practically be performed in the human mind, with or without pen and paper. See MPEP 2106.04(a)(2)(III). The recitation of a processor does not remove the mental-process character of the limitation because both product claims and process claims may recite mental-process-type abstract ideas, including computer system/product claims. The limitation also recites or at least relies on mathematical concepts because the determination is made “on a basis of” numerical probability values in the estimated probability vector.
The limitations of “acquiring an estimated classification of data to be classified” and “outputting the classification result, a determination result indicating whether the classification result is correct or incorrect, and the estimated probability vector” are not relied upon here as separately reciting the abstract idea under Step 2A Prong One. Rather, these limitations appear to be data acquisition and output limitations that may be evaluated later, if necessary, as additional elements under Step 2A Prong Two. No Step 2A Prong Two conclusion is made here.
Accordingly, independent claim 1 recites an abstract idea because the identified limitations recite mathematical concepts under MPEP 2106.04(a)(2)(I), including generating numerical vector/probability information, and mental processes under MPEP 2106.04(a)(2)(III), including evaluating whether a classification result is correct or incorrect based on probability information. Because claim 1 recites at least one judicial exception at Step 2A Prong One, the analysis should proceed to Step 2A Prong Two when requested.
Independent claim 5 is an error determination method claim reciting limitations materially corresponding to independent claim 1, including generating an estimation process feature vector, generating an estimated probability vector including probabilities corresponding to classes, and determining whether a classification result is correct or incorrect on a basis of the estimated probability vector. Therefore, independent claim 5 recites the same abstract idea identified with respect to claim 1 for similar reasons, namely mathematical concepts under MPEP 2106.04(a)(2)(I) and mental processes under MPEP 2106.04(a)(2)(III).
Independent claim 6 is a computer-readable non-transitory recording medium claim reciting computer-executable program instructions that cause a computer system to execute limitations materially corresponding to independent claims 1 and 5, including generating an estimation process feature vector, generating an estimated probability vector including probabilities corresponding to classes, and determining whether a classification result is correct or incorrect on a basis of the estimated probability vector. Therefore, independent claim 6 recites the same abstract idea identified with respect to claim 1 for similar reasons, namely mathematical concepts under MPEP 2106.04(a)(2)(I) and mental processes under MPEP 2106.04(a)(2)(III).
Step 2A Prong 2 -- whether the claim as a whole integrates the recited judicial exception into a practical application of the exception or whether the claim is “directed to” the judicial exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d).
Regarding independent claim 1, as discussed under Step 2A Prong One, the claim recites an abstract idea including mathematical concepts and mental processes. In particular, the limitations directed to generating an estimation process feature vector, generating an estimated probability vector indicating probabilities corresponding to classes, and determining whether a classification result is correct or incorrect on a basis of the estimated probability vector recite mathematical concepts and mental evaluations under MPEP §§ 2106.04(a)(2)(I) and 2106.04(a)(2)(III).
Claim 1 further recites additional elements beyond the identified judicial exception, including “an error determination device comprising a processor configured to execute operations,” “acquiring an estimated classification of data to be classified,” and “outputting the classification result, a determination result indicating whether the classification result is correct or incorrect, and the estimated probability vector.”
The limitation “an error determination device comprising a processor configured to execute operations” does not integrate the abstract idea into a practical application. The processor is recited at a high level of generality and is used to execute the claimed data-processing operations. The specification describes implementation by a computer executing a program, including hardware resources such as a CPU and memory, and also describes the computer as potentially being a physical computer or a virtual machine on a cloud. Thus, the processor is not recited as a particular machine integral to a technological improvement, but instead merely implements the abstract idea on generic computer hardware. This is not indicative of integration into a practical application under MPEP § 2106.05(f), which addresses mere instructions to implement an abstract idea on a computer or use a computer as a tool.
The limitation “acquiring an estimated classification of data to be classified” also does not integrate the abstract idea into a practical application. This limitation merely obtains input information used by the later determining operation. The acquisition of an estimated classification is a data-gathering step that supplies information for the judicial exception. Such data gathering is insignificant extra-solution activity under MPEP § 2106.05(g), particularly where the limitation amounts to necessary data gathering for use in the recited judicial exception.
The limitation “outputting the classification result, a determination result indicating whether the classification result is correct or incorrect, and the estimated probability vector” likewise does not integrate the abstract idea into a practical application. This limitation merely outputs the result of the abstract data analysis, including the classification result, the correctness determination, and the probability vector. The outputting is post-solution activity and does not impose a meaningful limit on the judicial exception. Under MPEP § 2106.05(g), necessary data outputting may be insignificant extra-solution activity.
Claim 1 also does not recite additional elements that amount to an improvement to the functioning of a computer or to another technology or technical field under MPEP §§ 2106.04(d)(1) and 2106.05(a). Although the specification describes the problem of outputting probabilities in addition to determining whether classification data is correct or incorrect, claim 1 recites this result at a functional level by requiring generation and output of an estimated probability vector, without reciting a particular technological improvement to computer operation, classifier architecture, memory organization, network security, sensor operation, or another technical field. An asserted improvement must be reflected in the claim itself, and the claim must include the components or steps that provide the described technological improvement.
Nor does claim 1 recite a particular treatment or prophylaxis, a transformation or reduction of a particular article to a different state or thing, or another meaningful application beyond generally linking the abstract idea to a computer-implemented error determination environment. MPEP § 2106.05(h) provides that generally linking use of a judicial exception to a particular technological environment or field of use does not integrate the exception into a practical application.
Even when the additional elements are considered in combination, claim 1 does not integrate the judicial exception into a practical application. The claim as a whole recites acquiring classification-related data, mathematically generating feature/probability vector information, evaluating whether a classification result is correct or incorrect, and outputting the resulting information using a generic processor. The ordered combination amounts to data gathering, mathematical analysis/evaluation, and data output implemented on generic computer hardware. These additional elements do not impose a meaningful limit on the abstract idea and do not transform the exception into a patent-eligible practical application. MPEP 2106.04(d) requires the claim as a whole to apply, rely on, or use the exception in a manner that imposes a meaningful limit on the exception; otherwise, the claim is directed to the exception.
Accordingly, independent claim 1 does not integrate the recited judicial exception into a practical application. Therefore, under Step 2A Prong Two, claim 1 is directed to the abstract idea.
Regarding independent claim 5, claim 5 is drawn to an error determination method executed by a computer and recites limitations similar to claim 1, including generating an estimation process feature vector, generating an estimated probability vector including probabilities corresponding to classes, and determining whether a classification result is correct or incorrect on a basis of the estimated probability vector. These limitations are not reanalyzed here and are rejected under the same rationale discussed above with respect to claim 1.
Claim 5 additionally recites that the method is “executed by a computer,” and further recites “acquiring an estimated classification of data to be classified” and “outputting the classification result, a determination result indicating whether the classification result is correct or incorrect, and the estimated probability vector.” The recitation that the method is executed by a computer does not integrate the abstract ide
a into a practical application because the computer is recited generically and merely performs the claimed data-processing operations. This amounts to no more than instructions to apply the judicial exception using a generic computer, which does not meaningfully limit the claim under MPEP 2106.05(f).
The limitation of acquiring an estimated classification of data to be classified is mere data gathering used as input for the subsequent classification-error determination. The limitation is pre-solution activity incidental to the primary abstract data analysis and does not impose a meaningful limit on the judicial exception. Therefore, this limitation is insignificant extra-solution activity under MPEP 2106.05(g).
The limitation of outputting the classification result, determination result, and estimated probability vector is post-solution activity that merely presents the result of the abstract analysis. Outputting the results does not alter how the abstract idea is performed and does not impose a meaningful limit on the exception. Therefore, this limitation is also insignificant extra-solution activity under MPEP 2106.05(g).
Even when considered in combination, the additional elements of claim 5 amount to a generic computer executing the abstract data-processing limitations, with data gathering and result output. These limitations do not improve the functioning of a computer or another technology under MPEP 2106.05(a), do not apply the exception with a particular machine under MPEP 2106.05(b), do not effect a particular transformation under MPEP 2106.05(c), and do not meaningfully apply the exception beyond generally linking it to a computer-implemented error determination environment under MPEP 2106.05(h). MPEP 2106.05(h) states that merely indicating a field of use or technological environment in which to apply a judicial exception cannot integrate the exception into a practical application.
Accordingly, independent claim 5, as a whole, does not integrate the recited judicial exception into a practical application and is directed to the abstract idea.
Regarding independent claim 6, claim 6 is drawn to a computer-readable non-transitory recording medium storing computer-executable program instructions that, when executed by a processor, cause a computer system to execute operations similar to those recited in claims 1 and 5, including generating an estimation process feature vector, generating an estimated probability vector including probabilities corresponding to classes, and determining whether a classification result is correct or incorrect on a basis of the estimated probability vector. These overlapping limitations are not reanalyzed here and are rejected under the same rationale discussed above with respect to claim 1.
Claim 6 additionally recites “a computer-readable non-transitory recording medium storing a computer-executable program instructions,” “a processor,” and “a computer system” configured to execute the claimed operations. These limitations do not integrate the judicial exception into a practical application. Although the non-transitory recording medium may satisfy Step 1 as a statutory manufacture, merely storing instructions that cause generic computer components to perform the abstract data-processing operations does not meaningfully limit the abstract idea. Rather, the recording medium, processor, and computer system merely provide a generic computer environment for applying the judicial exception, which is insufficient under MPEP 2106.05(f) and MPEP 2106.05(h).
Claim 6 further recites the same additional acquisition and output limitations discussed for claim 5. The acquiring limitation is data gathering used to supply information for the recited abstract analysis, and the outputting limitation is post-solution activity that presents the results of the analysis. These limitations are insignificant extra-solution activity under MPEP 2106.05(g) and do not meaningfully limit the judicial exception.
Even when considered in combination, the additional elements of claim 6 amount to storing and executing instructions on generic computer components to perform the same abstract data-processing operations, with data gathering and result output. The claim does not recite a particular asserted improvement to computer functionality or another technical field, does not require a particular machine integral to the performance of the exception, does not transform a particular article, and does not apply the judicial exception in another meaningful way. Instead, the claim generally links the abstract idea to a computer-readable medium and computer system environment.
Accordingly, independent claim 6, as a whole, does not integrate the recited judicial exception into a practical application and is directed to the abstract idea.
Conclusion for Step 2A Prong Two: Independent claims 5 and 6 do not integrate the recited judicial exception into a practical application under MPEP 2106.04(d). Therefore, claims 5 and 6 are directed to the abstract idea. No Step 2B analysis is provided here.
Step 2B -- whether the claim amounts to significantly more than the judicial exception. See MPEP § 2106.05.
Regarding independent claim 1, as explained in Step 2A Prong Two, the claim recites additional elements including “an error determination device comprising a processor configured to execute operations,” “acquiring an estimated classification of data to be classified,” and “outputting the classification result, a determination result indicating whether the classification result is correct or incorrect, and the estimated probability vector.”
The processor/error determination device does not amount to significantly more than the abstract idea. As previously explained under Step 2A Prong Two, the processor is recited at a high level of generality and merely executes the claimed data-processing operations. The specification describes that the classification device or error determination device may be implemented by causing a computer to execute a program, that the computer may be a physical computer or a virtual machine on a cloud, and that the program is executed using hardware resources such as a CPU and memory. Thus, the processor/device is a generic computer component performing generic computer functions. Under MPEP 2106.05(f), merely applying the abstract idea using a generic computer does not add significantly more. MPEP 2106.05(d)(II) also recognizes that a generic computer performing generic computer functions at a high level of generality may be well-understood, routine, and conventional.
The limitation of acquiring an estimated classification of data to be classified was treated at Step 2A Prong Two as data gathering. Upon re-evaluation at Step 2B, this limitation remains insignificant extra-solution activity and does not add significantly more. The claim does not recite any particular sensor, acquisition architecture, network protocol, memory structure, or unconventional manner of acquiring the estimated classification. Rather, the limitation merely obtains input information for use in the claimed abstract analysis. MPEP 2106.05(g) identifies pre-solution data gathering as extra-solution activity, and further states that necessary data gathering for use in the judicial exception is considered under Step 2B.
The limitation of outputting the classification result, a determination result indicating whether the classification result is correct or incorrect, and the estimated probability vector was treated at Step 2A Prong Two as output/post-solution activity. Upon re-evaluation at Step 2B, this limitation remains insignificant extra-solution activity and does not add significantly more. The outputting step merely presents the results of the abstract analysis and does not recite any particular display mechanism, transmission protocol, storage improvement, or technical action taken based on the output. MPEP 2106.05(g) identifies post-solution activity and necessary data outputting as insignificant extra-solution activity.
The additional elements also do not, individually or in combination, improve the functioning of a computer or another technology under MPEP 2106.05(a), apply the exception with a particular machine under MPEP 2106.05(b), effect a transformation under MPEP 2106.05(c), or apply the exception in another meaningful way under MPEP 2106.05(e). Instead, the claim recites generic computer implementation, data acquisition, abstract mathematical/mental analysis, and output of the resulting information.
Even when considered as an ordered combination, the additional elements amount to no more than using a generic processor to acquire input data, perform the abstract vector/probability/error-determination analysis, and output the results. The ordered combination does not recite a non-conventional arrangement of computer components or any technical mechanism that performs a function other than generic computer implementation of the abstract idea. MPEP 2106.05 requires the additional elements to be considered both individually and in combination, but a claim remains ineligible where the ordered combination adds nothing significantly more than implementation of the abstract idea using generic computer components.
Accordingly, independent claim 1 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Step 2B = NO. Claim 1 is directed to an abstract idea without significantly more.
Regarding independent claim 5, claim 5 is drawn to an error determination method executed by a computer and recites limitations corresponding to claim 1, including acquiring an estimated classification, generating an estimation process feature vector, generating an estimated probability vector, determining whether a classification result is correct or incorrect, and outputting the classification result, determination result, and estimated probability vector.
For the same reasons discussed above for claim 1, the limitations directed to acquiring the estimated classification and outputting the classification result, determination result, and estimated probability vector remain insignificant extra-solution activity upon re-evaluation at Step 2B. The acquiring limitation merely gathers input information for the judicial exception, and the outputting limitation merely presents the result of the abstract analysis. See MPEP 2106.05(g).
Claim 5 additionally recites that the method is “executed by a computer.” This additional computer-recitation does not add significantly more because the computer is recited generically and merely performs the claimed abstract data-processing method. The specification describes implementation using ordinary computer hardware resources such as a CPU and memory and does not describe the recited computer as a specially configured machine that performs a non-generic computer function. Under MPEP 2106.05(f), merely instructing implementation of an abstract idea on a generic computer does not provide an inventive concept.
Considering claim 5 as an ordered combination, the claim merely recites a computer-executed method for acquiring classification-related data, performing the abstract vector/probability/error-determination analysis, and outputting the resulting information. The method does not recite an improvement to computer functionality or another technical field, a particular machine, a transformation of an article, or any other meaningful limitation beyond generic computer execution. The ordered combination therefore does not provide significantly more than the judicial exception.
Accordingly, independent claim 5 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Step 2B = NO. Claim 5 is directed to an abstract idea without significantly more.
Regarding independent claim 6, claim 6 is drawn to a computer-readable non-transitory recording medium storing computer-executable program instructions that, when executed by a processor, cause a computer system to execute operations corresponding to those recited in claims 1 and 5.
For the same reasons discussed above for claim 1, the acquiring and outputting limitations remain insignificant extra-solution activity upon re-evaluation at Step 2B. The acquiring limitation merely obtains input information used in the judicial exception, and the outputting limitation merely presents the result of the abstract analysis. These limitations do not add significantly more under MPEP 2106.05(g).
Claim 6 additionally recites “a computer-readable non-transitory recording medium,” “computer-executable program instructions,” “a processor,” and “a computer system.” Although these limitations place the claim in a statutory manufacture category at Step 1, they do not add an inventive concept at Step 2B. The recited medium and instructions merely store software for causing a generic processor/computer system to perform the abstract data-processing operations. The specification describes a conventional computer implementation using hardware resources such as CPU and memory and a recording medium storing the program. MPEP 2106.05(d)(II) recognizes storing and retrieving information in memory and generic computer functions, when recited at a high level of generality, as WURC activity in appropriate circumstances.
Under MPEP 2106.05(f), reciting computer-readable instructions executed by a processor to apply the abstract idea is equivalent to mere instructions to apply the exception using generic computer components. The claim does not recite any particular memory architecture, data structure, processor improvement, storage improvement, or non-conventional arrangement of computer components comparable to the types of combinations that may amount to significantly more. See MPEP 2106.05(d).
Considering claim 6 as an ordered combination, the claim merely stores and executes program instructions that cause a computer system to acquire classification-related data, perform the abstract vector/probability/error-determination analysis, and output the resulting information. The ordered combination does not transform the abstract idea into an inventive application and does not provide significantly more than the judicial exception.
Accordingly, independent claim 6 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Step 2B = NO. Claim 6 is directed to an abstract idea without significantly more.
Regarding dependent claims 2-4, 7-9, 10-14, and 15-20
Step 1 -- whether the claim falls within any statutory category. See MPEP § 2106.03.
Regarding dependent claims 2-4 and 7-9, these claims are drawn to error determination device claims. Claims 2-4 depend from claim 1, and claims 7-9 depend from claim 2. Because claim 1 is drawn to an error determination device comprising a processor, dependent claims 2-4 and 7-9 also fall under at least one statutory category of subject matter, namely a machine/product/apparatus and/or manufacture.
Regarding dependent claims 10-14, these claims are drawn to error determination method claims. Claims 10-12 depend from claim 5, and claims 13-14 depend from claim 10. Because claim 5 is drawn to an error determination method executed by a computer, dependent claims 10-14 also fall under at least one statutory category of subject matter, namely a process/method.
Regarding dependent claims 15-20, these claims are drawn to computer-readable non-transitory recording medium claims. Claims 15-17 depend from claim 6, and claims 18-20 depend from claim 15. Because claim 6 is drawn to a computer-readable non-transitory recording medium storing computer-executable program instructions, dependent claims 15-20 also fall under at least one statutory category of subject matter, namely a manufacture.
Therefore, dependent claims 2-4 and 7-20 each fall under one of the four categories of statutory subject matter: process/method, machine/product/apparatus, manufacture, or composition of matter. Accordingly, dependent claims 2-4 and 7-20 satisfy Step 1 of the subject matter eligibility analysis.
Step 2A Prong 1 – whether the claim recites a judicial exception. See MPEP § 2106.04, subsection II; MPEP § 2106.04(a)(2), subsections I and III.
Regarding claims 2, 10, and 15, these claims recite the limitation:
“the generating an estimated probability vector uses a machine learning model learned by using a ratio in classification of each piece of learning data into the classes as correct answer data, the ratio being acquired during learning of estimating classification of data.”
This limitation recites a mathematical concept under MPEP 2106.04(a)(2)(I) because it expressly requires use of a ratio in classification of each piece of learning data into classes. A ratio is a mathematical relationship, and using classification ratios as correct-answer data for learning a machine-learning model further recites mathematical processing of numerical/classification information. MPEP 2106.04(a)(2)(I) states that mathematical concepts include mathematical relationships and calculations, including calculations expressed in words rather than symbols.
This limitation also overlaps with the mental-process grouping to the extent it encompasses classifying learning data into classes and using the resulting ratios as correct-answer data. Classification of data and evaluation of class ratios can be practically performed as observation, evaluation, judgment, or opinion, with or without pen and paper. See MPEP 2106.04(a)(2)(III).
Accordingly, the added limitation of claims 2, 10, and 15 recites an abstract idea in the form of at least a mathematical concept, and also a mental process to the extent the classification/evaluation aspects are broadly read.
Regarding claims 3, 7, 11, 13, 16, and 18, these claims recite the limitation:
“the determining further comprises determining whether the classification result is correct or incorrect by comparing a maximum value of the estimated probabilities each corresponding to one of the classes in the estimated probability vector with a threshold value.”
This limitation recites a mathematical concept under MPEP 2106.04(a)(2)(I) because it requires identifying or using a maximum value of estimated probabilities and comparing that numerical value with a threshold value. The limitation therefore involves numerical comparison and mathematical evaluation of probability values.
This limitation also recites a mental process under MPEP 2106.04(a)(2)(III) because determining whether a classification result is correct or incorrect by comparing a maximum probability value with a threshold is an evaluation or judgment that can practically be performed in the human mind, with or without pen and paper. The recitation of a device, method, or CRM form does not negate the mental-process nature of the limitation because product and process claims may recite mental-process-type abstract ideas.
Accordingly, the added limitation of claims 3, 7, 11, 13, 16, and 18 recites an abstract idea in the form of mathematical concepts and mental processes.
Regarding claims 4, 8, 12, 14, 17, and 19, these claims recite the limitation:
“the determining further comprises determining whether the classification result is correct or incorrect by comparing an average information amount of the estimated probabilities each corresponding to one of the classes in the estimated probability vector with a threshold value.”
This limitation recites a mathematical concept under MPEP 2106.04(a)(2)(I) because it requires use of an average information amount of estimated probabilities and comparison of that value with a threshold value. The specification describes the average information amount using an entropy-type formula, further confirming that the limitation involves mathematical calculation of a numerical value from probability values.
This limitation also recites a mental process under MPEP 2106.04(a)(2)(III) because comparing the calculated average information amount with a threshold and determining correctness/incorrectness is an evaluation or judgment that can practically be performed in the human mind, with or without pen and paper.
Accordingly, the added limitation of claims 4, 8, 12, 14, 17, and 19 recites an abstract idea in the form of mathematical concepts and mental processes.
Regarding claims 9 and 20, these claims recite the limitation:
“the machine learning model is learned based on supervised learning.”
This limitation further characterizes the machine-learning model recited in parent claims 2 and 15. Under the broadest reasonable interpretation in light of the specification, supervised learning uses learning data and correct-answer information to adjust parameters of a machine-learning model. The specification describes supervised learning and parameter adjustment of the classifier/probability-correction model using learning data and correct-answer ratios.
This limitation recites a mathematical concept under MPEP 2106.04(a)(2)(I) because supervised learning of a machine-learning model involves mathematical training of model parameters using labeled/correct-answer data. This is consistent with the 2024 AI training examples, where machine-learning training steps that explicitly involve model training calculations are treated as mathematical concepts.
To the extent the limitation broadly encompasses selecting labeled learning data, assigning correct-answer data, and evaluating the model against correct answers, it also implicates mental processes under MPEP 2106.04(a)(2)(III) because such selecting, assigning, and evaluating are observations, evaluations, judgments, or opinions that can practically be performed in the human mind at a high level of generality.
Accordingly, the added limitation of claims 9 and 20 recites an abstract idea in the form of at least a mathematical concept, and also a mental process to the extent the supervised-learning setup/evaluation is broadly read.
Step 2A Prong 2 -- whether the claim as a whole integrates the recited judicial exception into a practical application of the exception or whether the claim is “directed to” the judicial exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d).
Regarding dependent claims 2, 10, and 15, these claims recite that “the generating an estimated probability vector uses a machine learning model learned by using a ratio in classification of each piece of learning data into the classes as correct answer data, the ratio being acquired during learning of estimating classification of data.”
This limitation does not integrate the recited judicial exception into a practical application. As discussed under Step 2A Prong One, the limitation itself is part of the abstract idea because it recites use of classification ratios and model learning based on numerical/correct-answer data. To the extent the recited machine learning model is treated as an additional element, it merely applies the mathematical classification-ratio information in a generic machine-learning environment. The claim does not recite a particular model architecture, training architecture, loss function improvement, optimization improvement, hardware improvement, memory improvement, or other technical mechanism that improves operation of a computer or another technology under MPEP 2106.05(a).
The specification describes the classification probability correction vector calculation unit as being constructed with a machine-learning model capable of estimating a plurality of real values, such as a neural network, logistic regression, or support vector regression. This supports that the claim is not limited to a particular machine-learning implementation that improves computer functionality. Rather, the limitation uses a machine-learning model as a tool to perform the abstract calculation/evaluation. This amounts to mere instructions to apply the judicial exception using a computer model under MPEP 2106.05(f) and generally links the judicial exception to a machine-learning technological environment under MPEP 2106.05(h).
Accordingly, dependent claims 2, 10, and 15 do not integrate the judicial exception into a practical application.
Regarding dependent claims 3, 7, 11, 13, 16, and 18, these claims recite that “the determining further comprises determining whether the classification result is correct or incorrect by comparing a maximum value of the estimated probabilities each corresponding to one of the classes in the estimated probability vector with a threshold value.”
This limitation does not integrate the recited judicial exception into a practical application. The limitation further specifies the abstract determining operation by reciting a numerical comparison between a maximum estimated probability value and a threshold value. This is a mathematical decision rule and mental evaluation, not an additional element that applies the exception in a technologically meaningful way.
The limitation does not recite an improvement to the functioning of a computer or another technology under MPEP 2106.05(a). It also does not apply the exception with a particular machine under MPEP 2106.05(b), effect a transformation under MPEP 2106.05(c), or apply the exception in another meaningful way under MPEP 2106.05(e). Instead, the limitation merely narrows the abstract correctness determination by specifying the mathematical comparison used to make the determination.
Accordingly, dependent claims 3, 7, 11, 13, 16, and 18 do not integrate the judicial exception into a practical application.
Regarding dependent claims 4, 8, 12, 14, 17, and 19, these claims recite that “the determining further comprises determining whether the classification result is correct or incorrect by comparing an average information amount of the estimated probabilities each corresponding to one of the classes in the estimated probability vector with a threshold value.”
This limitation does not integrate the recited judicial exception into a practical application. The limitation further specifies the abstract determining operation by reciting use of an average information amount of estimated probabilities and comparison of that value with a threshold value. The specification describes the average information amount using an entropy-type calculation. Thus, the limitation further narrows the abstract idea through a mathematical calculation and numerical comparison.
The limitation does not recite a technical improvement to the classifier, model architecture, processor, memory, network, sensor, or any other technology. It also does not recite a particular machine, transformation, treatment, or other meaningful application of the exception. Accordingly, under MPEP 2106.05(a)-(c), 2106.05(e), 2106.05(f), and 2106.05(h), the limitation does not integrate the exception into a practical application.
Accordingly, dependent claims 4, 8, 12, 14, 17, and 19 do not integrate the judicial exception into a practical application.
Regarding dependent claims 9 and 20, these claims recite that “the machine learning model is learned based on supervised learning.”
This limitation does not integrate the recited judicial exception into a practical application. The limitation further characterizes the model-learning process used in the abstract probability-vector generation. Under BRI, supervised learning refers to learning a model from training data and correct-answer information. The specification similarly describes supervised learning using learning data, classification ratios, and parameter adjustment.
The limitation does not recite how supervised learning improves computer functionality or another technical field. It does not require a particular supervised-learning architecture, particular loss function, particular optimization process, or hardware configuration that improves operation of the computer or classifier. Rather, it merely identifies the learning paradigm used by the machine-learning model. Therefore, this limitation amounts to using a machine-learning model as a tool to apply the judicial exception under MPEP 2106.05(f) and generally linking the exception to a machine-learning environment under MPEP 2106.05(h).
Accordingly, dependent claims 9 and 20 do not integrate the judicial exception into a practical application.
Step 2B -- whether the claim amounts to significantly more than the judicial exception. See MPEP § 2106.05.
Claims 2, 10, and 15 add the limitation that the generating an estimated probability vector uses a machine learning model learned by using a ratio in classification of each piece of learning data into the classes as correct answer data, the ratio being acquired during learning of estimating classification of data.
This added limitation does not provide significantly more than the judicial exception. The limitation further defines the abstract mathematical/mental process by specifying that a machine-learning model is learned using classification ratios as correct-answer data. The recited ratio is a mathematical relationship, and the recited learning of the model using such ratios is part of the abstract idea identified at Step 2A Prong One.
To the extent the recited machine learning model is treated as an additional element, it is recited at a high level of generality and does not impose a nonconventional technical implementation. The specification describes the machine-learning model broadly, including examples such as a neural network, logistic regression, and support vector regression, and states that any machine-learning model may be used so long as it can estimate plural real values. This supports that the limitation merely uses a generic machine-learning model as a tool to perform the abstract probability-vector calculation.
The limitation also does not recite an improvement to computer functionality or another technical field under MPEP 2106.05(a). It does not recite a particular model architecture, loss function, training architecture, parameter-update rule, hardware arrangement, memory structure, or other technical improvement. Rather, it applies the abstract calculation using a generic ML model, which is insufficient under MPEP 2106.05(f) and 2106.05(h).
Accordingly, claims 2, 10, and 15 do not include additional elements that amount to significantly more than the judicial exception.
Claims 3, 7, 11, 13, 16, and 18 add the limitation that the determining further comprises determining whether the classification result is correct or incorrect by comparing a maximum value of the estimated probabilities each corresponding to one of the classes in the estimated probability vector with a threshold value.
This limitation does not provide significantly more than the judicial exception. The added limitation further narrows the abstract determining step by specifying a mathematical decision rule: comparing a maximum estimated probability value with a threshold value. The limitation remains part of the abstract idea because it recites a numerical comparison and evaluation of probability values.
The specification describes this method as determining a certainty factor using the maximum value of the estimated probabilities and comparing that value with a threshold. This confirms that the limitation is a mathematical comparison used to determine whether the classification result is correct or erroneous.
Because the limitation itself is part of the abstract idea, it cannot supply the inventive concept. Further, even if considered as an additional element, the limitation does not recite any unconventional computer component, particular machine, transformation, or technical improvement. It merely refines the abstract correctness determination using a threshold comparison.
Accordingly, claims 3, 7, 11, 13, 16, and 18 do not include additional elements that amount to significantly more than the judicial exception.
Claims 4, 8, 12, 14, 17, and 19 add the limitation that the determining further comprises determining whether the classification result is correct or incorrect by comparing an average information amount of the estimated probabilities each corresponding to one of the classes in the estimated probability vector with a threshold value.
This limitation does not provide significantly more than the judicial exception. The added limitation further defines the abstract determining operation by specifying an entropy-type mathematical calculation and threshold comparison. The specification describes the average information amount as calculated from the estimated probabilities using an equation of the form u=−∑pilogpiu = -\sum p_i \log p_iu=−∑pilogpi.
This limitation is therefore itself a mathematical concept and mental evaluation. It does not add a separate technological implementation beyond the abstract calculation. It does not recite an improvement to the operation of a classifier, processor, memory, network, or other technology. It also does not recite a particular machine or transformation.
Accordingly, claims 4, 8, 12, 14, 17, and 19 do not include additional elements that amount to significantly more than the judicial exception.
Claims 9 and 20 add the limitation that the machine learning model is learned based on supervised learning.
This limitation does not provide significantly more than the judicial exception. The limitation further characterizes the generic model-learning process recited in parent claims 2 and 15. The specification describes supervised learning broadly, including preparing learning data and correct-answer data, adjusting parameters of the classification estimation unit, and acquiring classification ratios during learning.
The limitation does not recite a particular supervised-learning algorithm, loss function, gradient-update rule, model architecture, training-data structure, or hardware implementation that improves computer functionality. Instead, it merely states the general learning paradigm used by the machine-learning model. A generic supervised-learning characterization is not enough to provide an inventive concept when the claimed model is used only to perform the abstract probability-vector generation.
Accordingly, claims 9 and 20 do not include additional elements that amount to significantly more than the judicial exception.
Accordingly, dependent claims 2-4 and 7-20 are directed to abstract ideas without significantly more and are patent-ineligible under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3, 5, 6, 11, and 16 are rejected under 35 U.S.C. § 103 as being unpatentable over Kawaguchi (Kawaguchi), JP 2020-024513 A, cited in the IDS filed on 12/03/2023 (English-language equivalent US 2021/0201087 A1, per MPEP § 2120, and cited in the IDS filed on 12/03/2023), in view of Ide et al. (Ide), English translation of WO 2021/014746 A1, published June 18, 2021 and further in view of Hendrycks et al. (Hendrycks), Non-Patent Literature, "A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks," ICLR 2017 (arXiv:1610.02136), published October 7, 2016, and relied upon at pages 1-12.
As to independent Claim 1, Kawaguchi teaches an error determination device comprising a processor configured to execute operations comprising (Kawaguchi US '087, ¶ 0025, p. 1: "the classifier 100 has a classification estimation unit 110 and a self-rejecting unit 120. The self-rejecting unit 120 includes a classification estimation process observation unit 121 and an error determination unit 122.”):
acquiring an estimated classification of data to be classified, where Kawaguchi's "classification object data" supplied to the classification estimation unit 110 reads on the recited "data to be classified," and the classification output by the classification estimation unit 110 reads on the recited "estimated classification" that is thereafter acquired by the error determination unit 122 (Kawaguchi US '087, ¶ 0027, p. 2: "classification object data is input to the classification estimation unit 110. The classification object data is data to be classified using the present system"; ¶ 0028, p. 2: "The classification estimation unit 110 estimates the classification of the input classification object data"; ¶ 0045, p. 3: "The error determination unit 122 receives the estimated classification from the classification estimation unit 110");
generating an estimation process feature vector on a basis of the data (Kawaguchi US '087, ¶ 0030, p. 2: "The classification estimation process observation unit 121 observes a calculation process in estimating the classification of the classification object data by the classification estimation unit 110, acquires data in an estimation process, converts the data into a feature vector, and outputs the feature vector to the error determination unit 122";
determining whether a classification result of the estimated classification of data is correct or incorrect (Kawaguchi US '087, ¶ 0031, p. 2: "The error determination unit 122 ... determines whether the classification estimated by the classification estimation unit 110 is 'correct' or 'incorrect' based on the observation data"; and
outputting the classification result and a determination result indicating whether the classification result is correct or incorrect (Kawaguchi US '087, ¶ 0058, p. 4: "the error determination unit outputs the classification result of the classification estimation unit when determining that the classification result is correct, and outputs information indicating that the classification is unknown when determining that the classification result is incorrect”).
Kawaguchi teaches that the values of the output layer of the neural network -- which are themselves the per-class probabilities -- are embedded as elements within the estimation process feature vector, and that the error determination unit may threshold a particular value of the feature vector to determine correctness (Kawaguchi US '087, ¶¶ 0041, 0047, p. 3). However, Kawaguchi does not expressly teach generating an estimated probability vector, wherein the estimated probability vector indicates probabilities each of which is a probability that the data to be classified belongs to one of classes on a basis of the estimation process feature vector.
In the same field of endeavor of supervised neural-network classification, Ide teaches this missing architectural element. Ide expressly discloses that "a neural network that handles a classification problem is composed of an arithmetic unit that calculates a feature vector from data and an arithmetic unit that calculates the probability that it belongs to each category of data from the feature vector" (Ide, p. 2, Background-Art). Ide reiterates this architecture in the Abstract (p. 1): "The neural network (3) computes a feature vector from the input data and, on the basis of the feature vector, computes a probability or score for the category to which the input data belongs, using a decoding computation corresponding to prescribed error-correction coding." Ide further teaches at p. 3, § 2-2 ("Model weight reduction") that "the neural network that estimates the category to which the data belongs from the data is a calculation unit that calculates the feature vector from the data and the calculation that calculates the degree of belonging / non-affiliation (hereinafter referred to as class score, probability) for each category from the feature vector." Ide's Figure 1 (cover page) depicts the architecture: data input AA enters feature extraction layer 31, which outputs the feature vector f; the feature vector then feeds into LDPC/Turbo decoding unit 32 and loss layer 33, which together produce the per-category probability or score as output. The per-category probability output of Ide reads on the recited "estimated probability vector," and Ide's express recitation that it is computed "on the basis of the feature vector" reads on the recited "on a basis of the estimation process feature vector."
Kawaguchi and Ide are analogous to the claimed invention as both are from the same field of endeavor of neural-network classification architectures that produce intermediate feature representations and downstream per-class outputs. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the error determination apparatus of Kawaguchi by adding the explicit feature-vector-to-probability-vector architecture taught by Ide. The motivation to combine Kawaguchi and Ide is as recited by Ide (Abstract, p. 1: the architecture "make[s] it possible to improve inference precision for a classification problem and reduce model parameters and the amount of computation"; § 2-1, p. 3: "an object of the present disclosure is to provide an information processing method for reducing errors in a classification problem of machine learning"), and is reinforced by Kawaguchi's own express recognition of the desirability of distinguishing classifications likely to be correct from those unlikely to be correct (Kawaguchi US '087, ¶ 0056, p. 4: "it is possible to distinguish the classification that is likely to be correct from the classification that is less likely to be correct"). One of ordinary skill in the art would have recognized that adopting Ide's architecture would yield, with predictable results, the per-class probability output that Kawaguchi presently embeds only implicitly within its feature vector -- providing an interpretable confidence signal for the error-determination step.
The combination of Kawaguchi and Ide, however, does not expressly teach “determining whether a classification result of the estimated classification of data is correct or incorrect on a basis of the estimated probability vector”, and “outputting the classification result, a determination result indicating whether the classification result is correct or incorrect, and the estimated probability vector.”
In the same field of endeavor, Hendrycks teaches both of these features. Hendrycks expressly establishes that the per-class probability distribution output by a classifier supplies a reliable basis for determining whether a classification is correct, observing that "Correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified and out-of-distribution examples, allowing for their detection" (Hendrycks, p. 1, Abstract). Hendrycks further teaches that "capturing prediction probability statistics about correct or in-sample examples is often sufficient for detecting whether an example is in error or abnormal" (p. 2, § 1), and frames the operative method as one of binary error/success prediction in which "we retrieve the maximum/predicted class probability from a softmax distribution and thereby detect whether an example is erroneously classified or out-of-distribution" (p. 3, § 3). Hendrycks's softmax distribution -- the per-class probability vector itself -- is both the basis for the correct/incorrect determination and the operative detection signal output for downstream use, thereby reading on both missing portions of limitations “determining whether a classification result of the estimated classification of data is correct or incorrect on a basis of the estimated probability vector” and “outputting the classification result, a determination result indicating whether the classification result is correct or incorrect, and the estimated probability vector.”
Kawaguchi, Ide, and Hendrycks are analogous to the claimed invention as all are from the same field of endeavor of determining the reliability of machine-learning classifications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the estimated probability vector produced by the Kawaguchi/Ide combination as the basis for the correct/incorrect determination, and to output that probability vector alongside the classification result and the determination result, in the manner taught by Hendrycks. The motivation to combine Kawaguchi, Ide, and Hendrycks is as recited by Hendrycks (p. 1, § 1: "estimating when a model is in error is of great concern to AI Safety"; Abstract: the softmax-probability baseline is demonstrated effective "across all" of computer vision, natural language processing, and automatic speech recognition tasks), such that one would enhance the error-determination apparatus of the Kawaguchi/Ide combination by adopting the established baseline technique for misclassification detection and by outputting the probability vector itself as a quantitative confidence measure usable for downstream triage -- precisely the function Kawaguchi identifies as desirable (Kawaguchi US '087, ¶ 0056, p. 4). The pertinence of Hendrycks to this art is further confirmed by its citation on the front page of JP 7655382 B2, the Japanese counterpart of the present application.
As to dependent Claim 3, the Kawaguchi/Ide/Hendrycks combination teaches the limitations of Claim 1 as set forth in Rejection 1. Claim 3 further requires comparing a maximum value of the estimated probabilities each corresponding to one of the classes in the estimated probability vector with a threshold value, as the basis for the correct/incorrect determination.
Hendrycks expressly teaches this limitation. Hendrycks's, p. 3, § 3 ("Softmax Prediction Probability as a Baseline") states: "we retrieve the maximum/predicted class probability from a softmax distribution and thereby detect whether an example is erroneously classified or out-of-distribution. Specifically, we separate correctly and incorrectly classified test set examples and, for each example, compute the softmax probability of the predicted class, i.e., the maximum softmax probability" (Hendrycks, p. 3, § 3). Hendrycks further teaches that detection performance is assessed by "summariz[ing] the performance of a binary classifier discriminating with values/scores (in this case, maximum probabilities from the softmaxes) across different thresholds" (p. 3, § 3). The Abstract reiterates the operative principle: "Correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified and out-of-distribution examples, allowing for their detection" (p. 1, Abstract). Hendrycks's maximum-softmax-probability-versus-threshold construct reads directly on the recited "comparing a maximum value of the estimated probabilities ... with a threshold value."
As to independent Claim 5, this claim recites the same operative limitations as Claim 1 in method form. Claim 5 is rejected on the same rationale as set forth in the rejection of Claim 1 above.
As to independent Claim 6, this claim recites the same operative limitations as Claim 1 in the form of a computer-readable non-transitory recording medium storing program instructions. Claim 6 is rejected on the same rationale as set forth in the rejection of Claim 1 above.
As to dependent Claim 11, this claim recites the same limitation as Claim 3 in the method form depending from Claim 5. Claim 11 is therefore rejected on the same grounds set forth for Claim 3.
As to dependent Claim 16, this claim recites the same limitation as Claim 3 in the recording-medium form depending from Claim 6. Claim 16 is therefore rejected on the same grounds set forth for Claim 3.
Claims 2, 7, 9, 10, 13, 15, 18, and 20 are rejected under 35 U.S.C. § 103 as being unpatentable over Kawaguchi in view of Ide, Hendrycks, and further in view of Takahashi et al. (Takahashi), Non-Patent Literature, "Estimation of Class Membership Probabilities in the Document Classification," in Advances in Knowledge Discovery and Data Mining, PAKDD 2007, Lecture Notes in Computer Science, Vol. 4426, pp. 260–271 (Springer-Verlag Berlin Heidelberg, 2007), published April 27, 2007.
As to dependent Claim 2, the Kawaguchi/Ide/Hendrycks combination teaches the limitations of Claim 1 as set forth in Rejection 1 above. Claim 2 further requires that the generating an estimated probability vector use a machine learning model learned by using a ratio in classification of each piece of learning data into the classes as correct answer data, the ratio being acquired during learning of estimating classification of data.
The Kawaguchi/Ide/Hendrycks combination does not expressly teach this training paradigm -- namely, training the probability-estimation model with per-class classification-ratio labels derived from the underlying classifier's training data.
In the same field of endeavor of confidence-probability estimation for machine-learning classifiers, Takahashi teaches precisely this technique under the rubric of the "accuracy table" method. Takahashi's Abstract (p. 1) states: "In the proposed method, we first make an accuracy table by counting the number of correctly classified training samples in each range or cell of classification scores. We then apply smoothing methods such as a moving average method with coverage to the accuracy table. In order to determine the class membership probability of an unknown sample, we first calculate the classification scores of the sample, then find the range or cell that corresponds to the scores and output the values associated in the range or cell in the accuracy table." Takahashi develops this method at § 1, p. 2: "we first make an accuracy table by counting the number of correctly classified training samples in each range or cell (hereafter referred to as cell) of scores. We then apply smoothing methods such as a moving average method to the accuracy table to yield reliable probabilities (accuracies). In order to determine the class membership probability of an unknown sample, we first calculate the scores of the sample, then find the cell that corresponds to the scores, and output the values associated in the cell in the accuracy table." The count of "correctly classified training samples in each range or cell of classification scores," normalized by the total number of training samples in that cell, IS the "ratio in classification of each piece of learning data into the classes as correct answer data, the ratio being acquired during learning" recited in Claim 2. Takahashi's Abstract (p. 1) further teaches that "the estimated class membership probabilities by the proposed method are useful in the detection of the misclassified samples," confirming the linkage of this probability-estimation paradigm to the error-determination context of the Kawaguchi/Ide/Hendrycks combination.
Kawaguchi, Ide, Hendrycks, and Takahashi are analogous to the claimed invention as all are from the same field of endeavor of determining and calibrating the reliability of machine-learning classifications. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to train the probability-estimation model of the Kawaguchi/Ide/Hendrycks combination using empirical classification-ratio data acquired during training of the underlying classifier, as taught by Takahashi. The motivation to combine Kawaguchi, Ide, Hendrycks, and Takahashi is as recited by Takahashi (Abstract, p. 1: the proposed method's "estimated class membership probabilities ... are useful in the detection of the misclassified samples"; § 1, p. 1: "Class membership probabilities are important in many applications in document classification, in which multiclass classification is often applied"), such that one would enhance the probability-vector generation in the Kawaguchi/Ide/Hendrycks combination by training it via the proven accuracy-table approach -- which directly addresses the misclassification-detection objective that Kawaguchi identifies as desirable (Kawaguchi US '087, ¶ 0056, p. 4).
As to dependent Claim 7, the Kawaguchi/Ide/Hendrycks/Takahashi combination teaches the limitations of Claim 2 as set forth in Rejection 2. Claim 7 further requires comparing a maximum value of the estimated probabilities each corresponding to one of the classes in the estimated probability vector with a threshold value, as the basis for the correct/incorrect determination. This is the same maximum-value-versus-threshold limitation analyzed for Claim 3 in Rejection 3, and Hendrycks expressly teaches it for the reasons set forth therein (Hendrycks, p. 1, Abstract; p. 3, § 3). The rationale to combine Kawaguchi, Ide, Hendrycks, and Takahashi articulated in Rejection 2 applies equally to Claim 7, as the addition of Hendrycks's specific maximum-probability-threshold criterion to the Claim 2 chain produces the predictable result of misclassification detection using the calibrated probability vector that Takahashi trains.
As to dependent Claim 9, the Kawaguchi/Ide/Hendrycks/Takahashi combination teaches the limitations of Claim 2 as set forth in Rejection 2. Claim 9 further requires that the machine learning model is learned based on supervised learning.
Kawaguchi expressly teaches this limitation. At US '087 ¶ 0048, p. 3, Kawaguchi states: "the error determination unit 122 may be implemented by parameter-tuning the models by supervised learning." Kawaguchi then describes the supervised-learning procedure in detail across ¶¶ 0049–0055 (pp. 3–4), comprising four steps: preparing a learning classification object data list (A) together with a manually generated correct classification list (B) (¶ 0051, p. 3: "Both of the learning classification object data list (A) and the correct classification list (B) thereof must be manually prepared"); generating the estimation process feature vector list (C) and classification result list (D) by passing the training data through the classification estimation unit (¶ 0053, p. 4); comparing (B) and (D) to acquire a learning correct/incorrect list (E) (¶ 0054, p. 4); and "performing machine learning using the estimation process feature vector list (C) as an input to the neural network (or SVM), and the learning correct/incorrect list (E) as a correct output from the neural network (or SVM)" (¶ 0055, p. 4). The manual preparation of correct classification labels and their use as training targets is the defining feature of supervised learning, and Kawaguchi labels its procedure as such expressly (¶ 0048). Ide independently confirms this paradigm: "Among machine learning, the technique of learning by pairing data and the correct answer to be derived from the data is called supervised learning. This 'correct answer' information is called a 'label' in machine learning. Labels are usually given to each piece of data by hand" (Ide, p. 2, § 1-2 "Supervised learning"). Takahashi additionally teaches supervised learning of the logistic-regression variant of its probability-estimation model (Takahashi, p. 4: "parameters A and B were estimated with the maximum likelihood method beforehand").
The rationale to combine Kawaguchi, Ide, Hendrycks, and Takahashi set forth in Rejection 2 applies equally to Claim 9, as Kawaguchi already discloses the supervised-learning training paradigm within the operative combination and the limitation of Claim 9 is the express training paradigm that Kawaguchi itself proposes.
As to dependent Claim 10, this claim recites the same limitation as Claim 2 in the method form depending from Claim 5. Claim 10 is therefore rejected on the same grounds set forth for Claim 2.
As to dependent Claim 13, this claim recites the same limitation as Claim 7 in the method form depending from Claim 10. Claim 13 is therefore rejected on the same grounds set forth for Claim 7.
As to dependent Claim 15, this claim recites the same limitation as Claim 2 in the recording-medium form depending from Claim 6. Claim 15 is therefore rejected on the same grounds set forth for Claim 2.
As to dependent Claim 18, this claim recites the same limitation as Claim 7 in the recording-medium form depending from Claim 15. Claim 18 is therefore rejected on the same grounds set forth for Claim 7.
As to dependent Claim 20, this claim recites the same limitation as Claim 9 in the recording-medium form depending from Claim 15. Claim 20 is therefore rejected on the same grounds set forth for Claim 9.
Claims 4, 12, and 17 are rejected under 35 U.S.C. § 103 as being unpatentable over Kawaguchi in view of Ide, Hendrycks, and further in view of Settles (Settles), Non-Patent Literature, "Active Learning Literature Survey," University of Wisconsin–Madison Computer Sciences Technical Report TR1648 (Jan. 26, 2010 revision), and relied upon at pages 1-67.
As to dependent Claim 4, the Kawaguchi/Ide/Hendrycks combination teaches the limitations of Claim 1 as set forth in Rejection 1. Claim 4 further requires comparing an average information amount of the estimated probabilities each corresponding to one of the classes in the estimated probability vector with a threshold value, as the basis for the correct/incorrect determination. As is well established in the art and consistent with the present specification, the recited "average information amount" of a probability distribution is Shannon entropy.
The Kawaguchi/Ide/Hendrycks combination does not expressly teach entropy thresholding as the criterion for correct/incorrect determination.
In the same field of endeavor of probabilistic-classifier confidence measurement, Settles teaches the entropy uncertainty criterion. At § 3.1 ("Uncertainty Sampling"), p. 13 of the Jan. 26, 2010 revision, Settles states: "A more general uncertainty sampling strategy (and possibly the most popular) uses entropy (Shannon, 1948) as an uncertainty measure: x*_H = argmax_x − Σ_i P_θ(y_i|x) log P_θ(y_i|x), where y_i ranges over all possible labelings. Entropy is an information-theoretic measure that represents the amount of information needed to 'encode' a distribution. As such, it is often thought of as a measure of uncertainty or impurity in machine learning." Settles further teaches that "the entropy-based approach generalizes easily to probabilistic multi-label classifiers" (§ 3.1). The formula H = −Σ P_θ(y_i|x) log P_θ(y_i|x), evaluated over the per-class probability vector, is the "average information amount" recited in Claim 4. The active-learning argmax formulation of Settles is translated to a thresholded misclassification-detection criterion through the same engineering substitution that Hendrycks footnote 2 (p. 3) validates: "We also tried using the KL divergence of the softmax distribution from the uniform distribution for detection. With divergence values, detector AUROCs and AUPRs were highly correlated with AUROCs and AUPRs from a detector using the maximum softmax probability." KL divergence from the uniform distribution is mathematically equivalent -- up to an additive constant -- to negative entropy of the probability distribution.
Kawaguchi, Ide, Hendrycks, and Settles are analogous to the claimed invention as all are from the same field of endeavor of assessing the reliability of probabilistic classifier outputs. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute or supplement Hendrycks's maximum-probability criterion with the entropy criterion of Settles in the Kawaguchi/Ide/Hendrycks combination. The motivation to combine Kawaguchi, Ide, Hendrycks, and Settles is as recited by Settles (§ 3.1: entropy is "possibly the most popular" general uncertainty-sampling strategy and is "often thought of as a measure of uncertainty"), reinforced by Hendrycks's own demonstration (p. 3, footnote 2) that information-theoretic measures of the softmax distribution yield detection performance highly correlated with the maximum-probability baseline. The substitution yields the predictable result of a confidence measure that accounts for the full per-class probability distribution rather than the maximum value alone.
As to dependent Claim 12, this claim recites the same limitation as Claim 4 in the method form depending from Claim 5. Claim 12 is therefore rejected on the same grounds set forth for Claim 4.
As to dependent Claim 17, this claim recites the same limitation as Claim 4 in the recording-medium form depending from Claim 6. Claim 17 is therefore rejected on the same grounds set forth for Claim 4.
Claims 8, 14, and 19 are rejected under 35 U.S.C. § 103 as being unpatentable over Kawaguchi in view of Ide, Hendrycks, Takahashi, and Settles.
As to dependent Claim 8, the Kawaguchi/Ide/Hendrycks/Takahashi combination teaches the limitations of Claim 2 as set forth in Rejection 2. Claim 8 further requires comparing an average information amount of the estimated probabilities each corresponding to one of the classes in the estimated probability vector with a threshold value. This is the same entropy-versus-threshold limitation analyzed for Claim 4 in Rejection 5, and Settles expressly teaches it for the reasons set forth therein (Settles, § 3.1). The rationale to combine Kawaguchi, Ide, Hendrycks, Takahashi, and Settles is articulated in Rejection 5 as to the entropy criterion and in Rejection 2 as to the Takahashi-trained probability-estimation model; together they yield the predictable result of an entropy-thresholded misclassification detector operating on the calibrated probability vector that Takahashi trains.
As to dependent Claim 14, this claim recites the same limitation as Claim 8 in the method form depending from Claim 10. Claim 14 is therefore rejected on the same grounds set forth for Claim 8.
As to dependent Claim 19, this claim recites the same limitation as Claim 8 in the recording-medium form depending from Claim 15. Claim 19 is therefore rejected on the same grounds set forth for Claim 8.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUNG VAN LE whose telephone number is (571)270-0164. The examiner can normally be reached 8 a.m. - 5 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUNG VAN LE/Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights