DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
Method claim 1 recites the following contingent limitation(s): identifying, using the at least one processor, the ground truth label as a clean label in response to determining that the degree of mismatch between the expert consensus label and the ground truth label does not exceed a threshold;
placing, using the at least one processor, the ground truth label in a training dataset in response to identifying the ground truth label as a clean label;
identifying, using the at least one processor, a problem with the ground truth label in response to determining that the degree of mismatch between the expert consensus label and the ground truth label exceeds the threshold; and
triggering reassessment of at least one guideline among the one or more guidelines for classifying the sample by the human grader in response to the identification of the problem with the ground truth label. The limitation(s) is/are contingent because it provides a mutually exclusive alternative of either a degree of mismatch 1) not exceeding a threshold, 2) exceeding a threshold, or 3) implicitly equaling a threshold. The broadest reasonable interpretation of the claim requires only one particular degree of mismatch occurring i.e., 1) not exceeding a threshold, 2) exceeding a threshold, or 3) implicitly equaling a threshold. For example, the mismatch could equal the threshold, rendering the above limitations to not have patentable weight. Since it can be interpreted to not activate one (or more) condition(s), the method claim thereby represents a broader scope than other identical claims from different statutory categories. See MPEP 2111.04(II) for more information.
Method claim 5 recites the following contingent limitation(s): marking the sample for reassessment in response to determining that the degree of mismatch between the expert consensus label and the ground truth label exceeds the threshold. The limitation(s) is/are contingent because it is possible for the degree of mismatch to not exceed, equal, or exceed the threshold. The broadest reasonable interpretation of the claim requires the degree of mistake to either not exceed or equal the threshold thereby rending this claim to not have patentable weight. Since it can be interpreted to not activate one (or more) condition(s), the method claim thereby represents a broader scope than other identical claims from different statutory categories. See MPEP 2111.04(II) for more information.
Claim Rejections - 35 USC § 112(a)
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Exemplary claim 1 recites, an expert consensus label according to the plurality of machine learned classifiers based on a first count of the plurality of machine learned classifiers and the degree of mismatch based on a second count of the plurality of machine learned classifiers. However, the originally filed disclosure does not disclose two counts. The specification states, “the largest group of experts 240 that agree with each other is tallied as a consensus count 255, and the expert label 235 on which the largest group of experts 240 agree is deemed to be an expert consensus label 250” in paragraph 49 and “the consensus count 255 can represent a degree of mismatch” in paragraph 53. Therefore, there is only a single count used. Applicant’s Remarks on page 11 also cites this disclosure from the specification. For this reason, the above listed claims are rejected for containing this language or being dependent on a claim that contains this language.
For examination purposes, the first count is interpreted as the same as the second count.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 8 and 17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Specifically, exemplary claim 8 recites the consensus which lacks antecedent basis. Claim 17 contains the same issue.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 1 is a method claim. Claim 10 is a device claim. Claim 19 is a CRM claim. Therefore, claims 1, 10, and 19 are directed to either a process, machine, manufacture or composition of matter.
With respect to Claim 1:
Step 2A Prong 1:
obtaining, using at least one processor of an electronic device, a ground truth label associated with a sample among a plurality of samples, the ground truth label determined by a human grader according to one or more guidelines for classifying the sample by the human grader, wherein the ground truth label is one of correct or incorrect (mental process – user can manually for the ground truth label determine by a human grader according to one or more guidelines for classifying the sample by the human grader, wherein the ground truth label is one of correct or incorrect)
generating, using a plurality of machine learned classifiers of different types executing on the at least one processor and including at least one of a random forest classifier, a gradient boosted classifier, and a support vector machine classifier, a plurality of expert labels for the sample, the plurality of expert labels including an expert label generated by each of the plurality of machine learned classifiers (mental process – user can manually generate a plurality of expert labels for the sample)
determining, using the at least one processor, that an expert label among the plurality of expert labels is an expert consensus label according to the plurality of machine learned classifiers based on a first count of the plurality of machine learned classifiers generating the expert label for the sample, the expert consensus label representing consensus among the plurality of machine learned classifiers(mental process – user can manually determine that an expert label among the plurality of expert labels is an expert consensus label )
in response to determining that the expert label is the expert consensus label, comparing, using the at least one processor, the expert consensus label to the ground truth label (mental process – user can manually in response to determining that the expert label is the expert consensus label, comparing the expert consensus label to the ground truth label)
determining, using the at least one processor, a degree of mismatch between the expert consensus label and the ground truth label, the degree of mismatch based on a second count of the plurality of machine learned classifiers generating the expert label for the sample (mental process – user can manually determine a degree of mismatch between the expert consensus label and the ground truth label)
identifying, using the at least one processor, the ground truth label as a clean label in response to determining that the degree of mismatch between the expert consensus label and the ground truth label does not exceed a threshold (mental process – user can manually identify the ground truth label as a clean label in response to determining that the degree of mismatch between the expert consensus label and the ground truth label does not exceed a threshold)
identifying, using the at least one processor, a problem with the ground truth label in response to determining that the degree of mismatch between the expert consensus label and the ground truth label exceeds the threshold (mental process – user can manually identify a problem with the ground truth label in response to determining that the degree of mismatch between the expert consensus label and the ground truth label exceeds the threshold)
triggering reassessment of at least one guideline among the one or more guidelines for classifying the sample by the human grader in response to the identification of the problem with the ground truth label (mental process – user can manually trigger reassessment of at least one guideline among the one or more guidelines for classifying the sample by the human grader in response to the identification of the problem with the ground truth label)
Step 2A Prong 2: This judicial exception is not integrated into a practical application. Additional elements:
obtaining, using at least one processor of an electronic device, a ground truth label associated with a sample among a plurality of samples, the ground truth label determined by a human grader according to one or more guidelines for classifying the sample by the human grader, wherein the ground truth label is one of correct or incorrect (Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g))
obtaining, using at least one processor of an electronic device, a ground truth label associated with a sample among a plurality of samples, the ground truth label determined by a human grader according to one or more guidelines for classifying the sample by the human grader, wherein the ground truth label is one of correct or incorrect (mere instructions to apply the exception using a generic computer component)
generating, using a plurality of machine learned classifiers of different types executing on the at least one processor and including at least one of a random forest classifier, a gradient boosted classifier, and a support vector machine classifier, a plurality of expert labels for the sample, the plurality of expert labels including an expert label generated by each of the plurality of machine learned classifiers (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
determining, using the at least one processor, that an expert label among the plurality of expert labels is an expert consensus label according to the plurality of machine learned classifiers based on a first count of the plurality of machine learned classifiers generating the expert label for the sample, the expert consensus label representing consensus among the plurality of machine learned classifiers (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
determining, using the at least one processor, a degree of mismatch between the expert consensus label and the ground truth label, the degree of mismatch based on a second count of the plurality of machine learned classifiers generating the expert label for the sample (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
placing, using the at least one processor, the ground truth label in a training dataset in response to identifying the ground truth label as a clean label (Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g))
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception. Additional elements:
obtaining, using at least one processor of an electronic device, a ground truth label associated with a sample among a plurality of samples, the ground truth label determined by a human grader according to one or more guidelines for classifying the sample by the human grader, wherein the ground truth label is one of correct or incorrect (MPEP 2106.05(d)(II) indicate that merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed step is well-understood, routine, conventional activity is supported under Berkheimer)
obtaining, using at least one processor of an electronic device, a ground truth label associated with a sample among a plurality of samples, the ground truth label determined by a human grader according to one or more guidelines for classifying the sample by the human grader, wherein the ground truth label is one of correct or incorrect (mere instructions to apply the exception using a generic computer component)
generating, using a plurality of machine learned classifiers of different types executing on the at least one processor and including at least one of a random forest classifier, a gradient boosted classifier, and a support vector machine classifier, a plurality of expert labels for the sample, the plurality of expert labels including an expert label generated by each of the plurality of machine learned classifiers (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
determining, using the at least one processor, that an expert label among the plurality of expert labels is an expert consensus label according to the plurality of machine learned classifiers based on a first count of the plurality of machine learned classifiers generating the expert label for the sample, the expert consensus label representing consensus among the plurality of machine learned classifiers (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
determining, using the at least one processor, a degree of mismatch between the expert consensus label and the ground truth label, the degree of mismatch based on a second count of the plurality of machine learned classifiers generating the expert label for the sample (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
placing, using the at least one processor, the ground truth label in a training dataset in response to identifying the ground truth label as a clean label (MPEP 2106.05(d)(II) indicate that merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed step is well-understood, routine, conventional activity is supported under Berkheimer)
Conclusion: The claim is not patent eligible.
Claims 10 and 19 are rejected on the same grounds as claim 1. Additionally for claims 10 and 19: Claim 10 has the additional elements of a memory and a processing device. These elements are mere instructions to apply the exception using a generic computer component under Step 2A prong 2 and Step 2B. Claim 19 has the additional element of a non-transitory machine-readable medium. This element is mere instructions to apply the exception using a generic computer component under Step 2A prong 2 and Step 2B.
Regarding Claims 2, 11, 20: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually identifying that the at least one guideline needs to be revised based on the degree of mismatch between the expert consensus label and the ground truth label.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claims 3 and 12: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually determining whether to reassess the sample by the human grader using a revised guideline for the at least one guideline after the at least one guideline is revised.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claims 4 and 13: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually wherein the ground truth label determined by the human grader using the one or more guidelines classifies the sample based on content.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claims 5 and 14: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually marking the sample for reassessment in response to determining that the degree of mismatch between the expert consensus label and the ground truth label exceeds the threshold.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claims 6 and 15: The limitation(s), as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, other than the additional elements, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) includes the additional elements of wherein the machine learned classifiers are trained using multi-fold cross validation.
These judicial exceptions are not integrated into a practical application. The additional element(s) of wherein the machine learned classifiers are trained using multi-fold cross validation recite merely adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f). Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of wherein the machine learned classifiers are trained using multi-fold cross validation recite adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f). Accordingly, the claims are not patent eligible.
Regarding Claims 7 and 16: The limitation(s), as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, other than the additional elements, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) includes the additional elements of wherein the machine learned classifiers include classifiers selected to reduce bias in label generation.
These judicial exceptions are not integrated into a practical application. The additional element(s) of wherein the machine learned classifiers include classifiers selected to reduce bias in label generation recite merely adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f). Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of wherein the machine learned classifiers include classifiers selected to reduce bias in label generation recite adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f). Accordingly, the claims are not patent eligible.
Regarding Claims 8 and 17: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually wherein the consensus among the plurality of machine learned classifiers is determined based on a largest number of matches among the plurality of expert labels.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claims 9 and 18: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually wherein: the sample is one of a plurality of samples; and each of the plurality of samples is associated with a verbal utterance.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 8-14, 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Al-Rawi et al. (hereinafter Al-Rawi) On the Labeling Correctness in Computer Vision Datasets in view of Komedani et al. (hereinafter Komedani), U.S. Patent Application Publication 2017/0255628, further in view of Chavez et al. (hereinafter Chavez), U.S. Patent 9,985,984.
Regarding Claim 1, Al-Rawi discloses a method comprising:
generating, using a plurality of machine learned classifiers of different types executing on the at least one processor and including at least one of a random forest classifier, a gradient boosted classifier, and a support vector machine classifier, a plurality of expert labels for the sample [“measure the per sample confidence-level is by using ensemble classification methods. In ensemble learning, multiple classifiers can be combined to solve a specific classification task” §1 ¶2] the plurality of expert labels including an expert label generated by each of the plurality of machine learned classifiers [“implemented voting schemes based on the predicted labels of the used classifiers” §2 ¶1];
determining, using the at least one processor, that an expert label among the plurality of expert labels is an expert consensus label according to the plurality of machine learned classifiers based on a first count of the plurality of machine learned classifiers generating the expert label for the sample, the expert consensus label representing consensus among the plurality of machine learned classifiers [“used majority voting ensemble based on the classifiers’ output labels, and the ensemble chooses the category/class that receives the largest total vote” §2.1 ¶1];
determining, using the at least one processor, a degree of mismatch between the expert consensus label and the ground truth label, the degree of mismatch based on a second count of the plurality of machine learned classifiers generating the expert label for the sample [“used majority voting ensemble based on the classifiers’ output labels, and the ensemble chooses the category/class that receives the largest total vote” §2.1 ¶1];
identifying, using the at least one processor, the ground truth label as a clean label [“Correct labels” Fig. 1] in response to determining that the degree of mismatch between the expert consensus label and the ground truth label does not exceed a threshold;
placing, using the at least one processor, the ground truth label in a training dataset in response to identifying the ground truth label as a clean label [“Data ready for usage” Fig. 1]; and
identifying, using the at least one processor, a problem with the ground truth label [“Incorrect labels” Fig. 1] in response to determining that the degree of mismatch between the expert consensus label and the ground truth label exceeds the threshold; and
triggering reassessment of at least one guideline among the one or more guidelines for classifying the sample by the human grader in response to the identification of the problem with the ground truth label [“Annotation/Labeling” after “Incorrect labels” Fig. 1; “verify the labeling is by having a system that returns a confidence-level for each sample in the dataset” §1 ¶1; Fig. 1; Table 2].
However, Al-Rawi fails to explicitly disclose obtaining, using at least one processor of an electronic device, a ground truth label associated with a sample among a plurality of samples, the ground truth label determined by a human grader according to one or more guidelines for classifying the sample by the human grader, wherein the ground truth label is one of correct or incorrect;
triggering reassessment of at least one guideline among the one or more guidelines in response to the identification of the problem with the ground truth label.
Komedani discloses obtaining, using at least one processor of an electronic device [Figs. 1 and 2] a ground truth label associated with a sample among a plurality of samples, the ground truth label determined by a human grader according to one or more guidelines for classifying the sample by the human grader [“annotators may add annotations to document elements constituting the document. The annotations may be categories of the document elements. The document elements may include words, phrases, and sentences. Thus, an annotation "Person" is assumed to be added to a word "Lincoln". Meanwhile, the annotators may be persons who are responsible for adding the annotations to the document elements. In this exemplary embodiment, the annotators are assumed to add the annotations to the document elements according to an annotation guideline as one example of a guideline. The annotation guideline may establish standards regarding what kinds of annotations are to be added to what kinds of document elements.” ¶13; “annotations may be manually added in order give a semantic structure to a text document.” ¶2], wherein the ground truth label is one of correct or incorrect [“If an annotation has low quality, this exemplary embodiment may detect whether it is because the annotation guideline has low quality, or because any of the annotators has low proficiency.” §14; Examiner Note: a low-quality label is incorrect];
triggering reassessment of at least one guideline among the one or more guidelines for classifying the sample by the human grader in response to the identification of the problem with the ground truth label [“If the evaluation result shows that the low quality of the annotation guideline has caused the low quality of the annotation, the evaluation module 240 may output information to that effect and the support information for supporting revision of the annotation guide” ¶20; “annotations may be manually added in order give a semantic structure to a text document. A plurality of annotators may add annotations according to an annotation guideline.” ¶2].
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi and Komedani before him before the effective filing date of the claimed invention, to modify the method of Al-Rawi to incorporate the human grader and guidelines of Komedani.
Given the advantage of using machine learning to verify human labeled data to ensure proper labels for better training data, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Al-Rawi fails to explicitly disclose generating, using a plurality of machine learned classifiers of different types executing on the at least one processor and including at least one of a random forest classifier, a gradient boosted classifier, and a support vector machine classifier, a plurality of expert labels for the sample, the plurality of expert labels including an expert label generated by each of the plurality of machine learned classifiers.
Chavez discloses generating, using a plurality of machine learned classifiers of different types executing on the at least one processor and including at least one of a random forest classifier, a gradient boosted classifier, and a support vector machine classifier [“An ensemble of classifiers are trained on each of the feature subsets 327 and 328. Each ensemble contains one each of the Naïve Bayes, logistic regression, support vector machine (SVM), and random forest classifiers, and are termed level 1 classifiers 330 (for application with subset A 327) and 331 (for application with subset B 328). It is to be appreciated that as well as the previously mentioned classifiers, any suitable classifier can be utilized” col. 16, lines 3-12; “Combining heterogeneous classifiers into an ensemble in this way has been shown to increase prediction accuracy” col. 16, lines 61-63], a plurality of expert labels for the sample, the plurality of expert labels including an expert label generated by each of the plurality of machine learned classifiers.
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi, Komedani, and Chavez before him before the effective filing date of the claimed invention, to modify the combination to incorporate heterogeneous classifiers of Chavez.
Given the advantage of increase prediction accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Al-rawi fails to explicitly disclose in response to determining that the expert label is the expert consensus label, comparing, using the at least one processor, the expert consensus label to the ground truth label;
determining, using the at least one processor, a degree of mismatch between the expert consensus label and the ground truth label, the degree of mismatch based on a second count of the plurality of machine learned classifiers generating the expert label for the sample;
identifying, using the at least one processor, the ground truth label as a clean label in response to determining that the degree of mismatch between the expert consensus label and the ground truth label does not exceed a threshold;
placing, using the at least one processor, the ground truth label in a training dataset in response to identifying the ground truth label as a clean label;
identifying, using the at least one processor, a problem with the ground truth label in response to determining that the degree of mismatch between the expert consensus label and the ground truth label exceeds the threshold.
Shapiro discloses in response to determining that the expert label is the expert consensus label, comparing, using the at least one processor, the expert consensus label to the ground truth label [“The accuracy of the annotations can be determined based on the annotations describing the features (e.g. severity, cost to repair/replace, etc.) as compared to the ground truth labels for those features indicated in the ground truth data.” ¶58];
determining, using the at least one processor, a degree of mismatch between the expert consensus label and the ground truth label [“if the performance data indicates that the obtained annotation data is below a particular quality threshold” ¶58], the degree of mismatch based on a second count of the plurality of machine learned classifiers generating the expert label for the sample;
identifying, using the at least one processor, the ground truth label as a clean label in response to determining that the degree of mismatch between the expert consensus label and the ground truth label does not exceed a threshold [“if the performance data indicates that the obtained annotation data is below a particular quality threshold” ¶58];
placing, using the at least one processor, the ground truth label in a training dataset in response to identifying the ground truth label as a clean label;
identifying, using the at least one processor, a problem with the ground truth label in response to determining that the degree of mismatch between the expert consensus label and the ground truth label exceeds the threshold [“if the performance data indicates that the obtained annotation data is below a particular quality threshold” ¶58].
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi, Komedani, Chavez, and Shapiro before him before the effective filing date of the claimed invention, to modify the combination to incorporate the annotation comparison, quality threshold, and further action of Shapiro
Given the advantage of comparing annotations and ensuring a minimal threshold of accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 2, Al-Rawi, Komedani, Chavez, and Shapiro disclose the method of Claim 1.
However, Al-Rawi fails to explicitly disclose identifying that the at least one guideline needs to be revised based on the degree of mismatch between the expert consensus label and the ground truth label.
Komedani discloses identifying that the at least one guideline needs to be revised based on the degree of mismatch between the expert consensus label and the ground truth label [If the evaluation result shows that the low quality of the annotation guideline has caused the low quality of the annotation, the evaluation module 240 may output information to that effect and the support information for supporting revision of the annotation guide” ¶20].
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi, Komedani, Chavez, and Shapiro before him before the effective filing date of the claimed invention, to modify the combination to incorporate the adjustment of guidelines of Komedani.
Given the advantage of improving guidelines to improve further annotations, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 3, Al-Rawi, Komedani, Chavez, and Shapiro disclose the method of Claim 2.
However, Al-Rawi fails to explicitly disclose determining whether to reassess the sample by the human grader using a revised guideline for the at least one guideline after the at least one guideline is revised.
Komedani discloses determining whether to reassess the sample by the human grader using a revised guideline for the at least one guideline after the at least one guideline is revised [“The operation above is repeated until all annotation types are processed” ¶58; “information supporting revision to the annotation guideline, which may be used to improve the annotation guideline by correcting aspects of the annotation guideline associated with the poor quality rating.” ¶60; Fig. 5].
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi, Komedani, Chavez, and Shapiro before him before the effective filing date of the claimed invention, to modify the combination to incorporate the use of the revised guidelines of Komedani.
Given the advantage of improving guidelines to improve further annotation accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Al-Rawi fails to explicitly disclose determining whether to reassess the sample by the human grader using a revised guideline for the at least one guideline after the at least one guideline is revised.
Shapiro discloses determining whether to reassess the sample by the human grader [“The secondary review can include a manual review of the item data and/or initiating a second request for annotations” ¶58] using a revised guideline for the at least one guideline after the at least one guideline is revised.
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi, Komedani, Chavez, and Shapiro before him before the effective filing date of the claimed invention, to modify the combination to incorporate reassessing an annotation of Shapiro.
Given the advantage of a secondary review to increase accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 4, Al-Rawi, Komedani, Chavez, and Shapiro disclose the method of Claim 2.
However, Al-Rawi fails to explicitly disclose wherein the ground truth determined by the human grader using the one or more guidelines classifies the sample cased on content.
Komedani discloses wherein the ground truth determined by the human grader using the one or more guidelines classifies the sample cased on content [“The annotations may be categories of the document elements.” ¶13].
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi, Komedani, Chavez, and Shapiro before him before the effective filing date of the claimed invention, to modify the combination to incorporate the guidelines for annotation of Komedani.
Given the advantage of using guidelines for increased consistency and accuracy of labeling, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 5, Al-Rawi, Komedani, Chavez, and Shapiro disclose the method of Claim 1.
However, Al-Rawi fails to explicitly disclose marking the sample for reassessment in response to determining that the degree of mismatch between the expert consensus label and the ground truth label exceeds the threshold.
Shapiro discloses marking the sample for reassessment in response to determining that the degree of mismatch between the expert consensus label and the ground truth label exceeds the threshold [“if the performance data indicates that the obtained annotation data is below a particular quality threshold, a secondary review can be initiated. The secondary review can include a manual review of the item data and/or initiating a second request for annotations” ¶58].
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi, Komedani, Chavez, and Shapiro before him before the effective filing date of the claimed invention, to modify the combination to incorporate the reassessment of Shapiro.
Given the advantage of reassessment in order to increase accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 8, Al-Rawi, Komedani, Chavez, and Shapiro disclose the method of Claim 1. Al-Rawi further discloses wherein the consensus among the plurality of machine learned classifiers is determined based on a largest number of matches among the plurality of expert labels [“a majority voting ensemble” Abstract].
Regarding Claim 9, Al-Rawi, Komedani, Chavez, and Shapiro disclose the method of Claim 1.
However, Al-Rawi fails to explicitly disclose wherein: the sample is one of a plurality of samples; and each of the plurality of samples is associated with a verbal utterance.
Komedani discloses wherein: the sample is one of a plurality of samples; and each of the plurality of samples is associated with a verbal utterance [“annotations may be manually added in order give a semantic structure to a text document.” ¶2; “The annotations may be categories of the document elements. The document elements may include words, phrases, and sentences.” ¶13].
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi, Komedani, Chavez, and Shapiro before him before the effective filing date of the claimed invention, to modify the combination to incorporate the plurality of samples that are associated with a verbal utterance.
Given the advantage of using the method efficiently on many samples that pertain of words used in numerous fields, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claims 10-14 are rejected on the same grounds as claims 1-5 respectively.
Claims 17-18 are rejected on the same grounds as claims 8-9 respectively.
Claims 19-20 are rejected on the same grounds as claims 1-2 respectively.
Claim(s) 6 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Al-Rawi, Komedani, Chavez, and Shapiro , in view of Doan et al. (hereinafter Doan), Recognition of medication information from discharge summaries using ensembles of classifiers.
Regarding Claim 6, Al-Rawi, Komedani, Chavez, and Shapiro discloses the method of Claim 1.
However, Al-Rawi fails to explicitly disclose wherein the machine learned classifiers are trained using multi-fold cross validation.
Doan discloses wherein the machine learned classifiers are trained using multi-fold cross validation [“10-fold cross-validation” pg. 5 §Experimental settings ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi, Komedani, Chavez, Shapiro, and Doan before him before the effective filing date of the claimed invention, to modify the combination to incorporate the cross-validation of Doan.
Given the advantage of training and testing the models in the ensemble to ensure accurate results, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim 15 is rejected on the same grounds as claims 6.
Claim(s) 7 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Al-Rawi, Komedani, Chavez, and Shapiro , in view of Sesmero et al. (hereinafter Sesmero), Generating ensembles of heterogeneous classifiers using Stacked Generalization.
Regarding Claim 7, Al-Rawi, Komedani, Chavez, and Shapiro discloses the method of Claim 1.
However, Al-Rawi fails to explicitly disclose wherein the machine learned classifiers include classifiers selected to reduce bias in label generation.
Sesmero discloses wherein the machine learned classifiers include classifiers selected to reduce bias in label generation [“ensembles with base classifiers trained from different learning algorithms (heterogeneous ensembles) exploit the different biases of each learning algorithm. Therefore, most studies in the field of ensembles have focused on the combination of different inducers, such as artificial neural networks, decision trees, Bayesian models, nearest neighbor, and support vector machines” pg. 4, col. 1, lines 12-19; Examiner Note: Using heterogeneous models with different biases in an ensemble reduces the overall bias of the final result.].
It would have been obvious to one having ordinary skill in the art, having the teachings of Al-Rawi, Komedani, Chavez, Shapiro, and Sesmero before him before the effective filing date of the claimed invention, to modify the combination to incorporate bias reduction of Sesmero.
Given the advantage of reducing bias in order to get more accurate results, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim 16 is rejected on the same grounds as claims 7.
Examiner’s Note
The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well.
Additionally, any claim amendments for any reason should include remarks indicating clear support in the originally filed specification.
Response to Arguments
Regarding the 101 rejections, Applicant's arguments have been fully considered but have been found unpersuasive. Applicant argues that 1) the claims provide a practical application and are directed to an improvement of machine learning technology, 2) the claims recite significantly more than the judicial exception, and 3) the improvement is not directed to the abstract idea. Examiner disagrees for at least the following reasons.
First, the additional elements, alone or in combination and when viewing the claim as a whole, do not integrate the abstract idea into a practical application. The additional elements include generic computer components (e.g., processor), extra solution activity (e.g., transmission of data), and merely applying the abstract idea using machine learned classifiers. Any alleged improvement is not to a technological field; for example, there is no improvement to the functioning of the computers or models. Instead, the alleged improvement is to revising annotation guidelines by comparing annotations done under those guidelines to expert annotations. This is an abstract idea. The fact the expert annotations are performed by an ensemble is merely applying the models with the judicial exception.
Second, the claims do not recite significantly more than the judicial exception. The inventive concept in the claims is directed to updating the annotation guidelines, which is part of the abstract idea. Note, the annotation guidelines are claimed to be used by “a human grader.” Again, the classifiers are additional elements merely applying the judicial exception to a computing environment.
Third, any alleged improvement is directed to the abstract idea. Applicant admits in Remarks page 12 in line 15 that “[c]laim 1 is directed to improving guidelines for human grading.” Examiner agrees. The claim does include additional elements which merely permits performance in a general computing environment.
Therefore, the rejections are maintained.
Regarding the prior art rejections, Applicant's arguments with respect to most of the claims have been considered but are moot because the arguments do not apply to the references being used in the current rejection of the limitations.
Four arguments advanced by Applicant that pertain to previously cited prior art are 1) Al-Rawi does not disclose employing guidelines for classifying a sample used by a human grader, 2) Al-Rawi’s ensembles are not “machine learned classifiers of different types,” 3) nothing suggests using the annotation guidelines of Komedani in connection to the ensemble of Al-Rawi, and 4) Chavez does not disclose or suggest that SVMs and random forest classifiers are each used to determine "that an expert label among the plurality of expert labels is an expert consensus label" for purposes of comparing a degree of mismatch between a ground truth and the expert consensus label. Examiner disagrees for at least the following reasons.
First, as shown in the rejection above, Komedani, not Al-Rawi, discloses employing guidelines for classifying a sample used by a human grader.
Second, as shown in the rejection above, Chavez, not Al-Rawi, discloses ensembles of classifiers of different types.
Third, Al-Rawi, Komedani, and the instant application are all directed to issues with annotating/labeling data. Al-Rawi utilizes ensembles to label data, while Komedani utilizes human annotators using an annotation guideline to label data. A person would be motivated by increasing labeling accuracy and speed to have both ensemble and human labeling of data. Human labeling is often more accurate, while machine labeling is faster. Combining the two can give the best of both.
Fourth, in response to applicant's arguments against Chavez individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). Chavez discloses the use of heterogenous classifiers in an ensemble as outlined in the rejection above, while the other references, specifically Al-Rawi, disclose determining a consensus label from a plurality of labels. It is a combination of references which reject the argued limitation.
Therefore, the rejections are maintained.
Conclusion
Any prior art made of record and not relied upon is considered pertinent to Applicant's disclosure. Applicant is reminded that in amending in response to a rejection of claims, the patentable novelty must be clearly shown in view of the state of the art disclosed by the references cited and the objections made. Applicant must also show how the amendments avoid such references and objections. See 37 CFR §1.111(c). Additionally when amending, in their remarks Applicant should particularly cite to the supporting paragraphs in the original disclosure for the amendments.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT H BEJCEK II whose telephone number is (571)270-3610. The examiner can normally be reached Monday - Friday: 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.B./ Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights