DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 rejected under 35 U.S.C. 101 for reciting abstract ideas without additional elements that amount to significantly more than the abstract ideas or integrate the abstract ideas into a practical application.
Step 1
According to the first part of the analysis, in the instant case, claims 1-9 are directed to a method, and claims 10-20 are directed to an apparatus. Each of these claims fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
For claim 1,
Step 2A Prong One
determining a set of aspects corresponding to the one or more annotations;
(This step for determining aspects corresponding to annotations is a mental process)
determining, using a machine learning model and based at least on the set of aspects, a probability of the one or more annotations containing an error;
(This step for determining a probability using a machine learning model is a mathematical concept)
or approving, in response to the one or more annotations having less than the threshold probability of containing an error, the one or more annotations without providing the one or more annotations to the human reviewer to review.
(This step for approving annotations determined to have less than a threshold probability of error is a mental process)
one or more annotations being determined to have higher than a threshold probability of containing an error
(This step for determining is a mental process)
Step 2A Prong Two
A computer-implemented method, comprising:
(This step for performing the methods on a generic computer is mere-instructions to apply an exception. See MPEP § 2106.05(f))
receiving, with respect to an instance of data including one or more objects, one or more annotations generated using one or more inputs from a human labeler;
(This step for receiving data from a human is insignificant extra-solution activity. See MPEP § 2106.05(g))
and one of: providing, in response to the [determining], the one or more annotations and the instance of data to a human reviewer;
(This step for providing / transmitting data to a reviewer after determining the data has a high probability of containing errors is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts while the additional elements are mere-instructions to apply an exception and insignificant extra-solution activity.
Additionally, the courts have determined that functions such as data gathering (receiving, … one or more annotations generated using one or more inputs from a human labeler) and transmitting (providing, in response to the [determining], the one or more annotations and the instance of data to a human reviewer;) are well understood, routine, and conventional activity or insignificant extra-solution activity when recited at a high degree of generality. See MPEP § 2106.05(d, g);
For claim 2
Step 2A Prong One
(Claim 2 depends on claim 1, which has been determined to recite abstract ideas including mental processes and mathematical concepts. Therefore, claim 2 also recites an abstract idea.)
Step 2A Prong Two
wherein the providing the one or more annotations and the instance of data to the human reviewer includes causing display of the instance of data along with at least one annotation of the one or more annotations within a user interface (UI).
(This step for providing / transmitting annotation data by displaying the annotation data is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts while the additional elements are insignificant extra-solution activity.
Additionally, the courts have determined that functions such as data transmitting and displaying are well understood, routine, and conventional activity or insignificant extra-solution activity when recited at a high degree of generality. See MPEP § 2106.05(d, g);
For claim 3,
Step 2A Prong One
the one or more annotations are classified as: high-risk based at least on the probability of the one or more annotations containing an error being higher than the threshold probability; or low-risk based at least on the probability of the one or more annotations containing an error being less than the threshold probability.
(These steps for classifying annotations based on a comparison between probabilities and a threshold are mental processes)
Step 2A Prong Two
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application.
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts without any technological improvement or inventive step.
For claim 4,
Step 2A Prong One
the threshold probability is adjustable based in part upon a risk tolerance.
(Adjusting a threshold used for comparison based on a ‘risk tolerance’ can be performed mentally / with pen and paper, i.e., it can be considered a mental process)
Step 2A Prong Two
(The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application.
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts without any technological improvement or inventive step.
For claim 5,
Step 2A Prong One
the set of aspects includes at least one of a nature of a task for which the one or more annotations are generated, a proportion of objects labeled with a same object classification, a proportion of objects in each of a plurality of distance bins, or a historic performance of the human labeler.
(The aforementioned determined set of aspects are a mental process)
Step 2A Prong Two
(The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application.
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts without any technological improvement or inventive step.
For claim 6,
Step 2A Prong One
wherein the machine learning model includes a linear regression model.
(The aforementioned machine learning model for determining a probability [which includes a linear regression model] is a mathematical concept)
Step 2A Prong Two
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application.
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts without any technological improvement or inventive step.
For claim 7,
Step 2A Prong One
(Claim 7 depends on claim 1, which has been determined to recite abstract ideas including mental processes and mathematical concepts. Therefore, claim 7 also recites an abstract idea.)
Step 2A Prong Two
wherein the one or more annotations are to be used as ground truth data to update one or more parameters of a second machine learning model.
(This step for using the annotations as ground truth data to update parameters of a ML model is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts while the additional element is mere-instructions to apply an exception.
Additionally, the courts have determined that functions such as selecting a particular data source to be manipulated [one or more annotations are to be used as ground truth data] and data updating [updating one or more parameters of a second machine learning model] are well understood, routine, and conventional activity or insignificant extra-solution activity when recited at a high degree of generality. See MPEP § 2106.05(d, g);
For claim 8,
Step 2A Prong One
(Claim 8 depends on claim 1, which has been determined to recite abstract ideas including mental processes and mathematical concepts. Therefore, claim 8 also recites an abstract idea.)
Step 2A Prong Two
modifying at least one annotation of the one or more annotations before providing the one or more annotations to the human reviewer, the modifying performed to influence an engagement of the human reviewer.
(This step for using the modifying the annotation data is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts while the additional element is mere-instructions to apply an exception.
Additionally, the courts have determined that functions such as selecting a particular data source to be manipulated / modified [modifying at least one annotation of the one or more annotations…] are well understood, routine, and conventional activity or insignificant extra-solution activity when recited at a high degree of generality. See MPEP § 2106.05(d, g);
For claim 9,
Step 2A Prong One
(Claim 9 depends on claim 1, which has been determined to recite abstract ideas including mental processes and mathematical concepts. Therefore, claim 9 also recites an abstract idea.)
Step 2A Prong Two
wherein the at least one annotation is modified by, at least one of: adding an annotation; deleting an annotation; or shifting a location of the annotation.
(This step for using the modifying the annotation data is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts while the additional element is mere-instructions to apply an exception.
Additionally, the courts have determined that functions such as selecting a particular data source to be manipulated / modified [wherein the at least one annotation is modified by, at least one of: adding an annotation; deleting an annotation; or shifting a location of the annotation.] are well understood, routine, and conventional activity or insignificant extra-solution activity when recited at a high degree of generality. See MPEP § 2106.05(d, g);
For claim 10,
Step 2A Prong One
determine one or more aspects corresponding to a set of annotations generated using a human labeler as part of a labeling task;
(This step for determining aspects corresponding to annotations is a mental process)
generate, using a machine learning model and based at least on the one or more aspects, a risk score indicating a probability that the set of annotations includes at least one error;
(This step for generating a probability using a machine learning model is a mathematical concept)
or automatically approve the set of annotations in response to the risk score being less than the risk threshold.
(This step for approving annotations determined to have less than a threshold probability of error is a mental process)
determining that the risk score exceeds a risk threshold
(This step for determining is a mental process)
Step 2A Prong Two
At least one processor, comprising: one or more circuits to:
(This step for performing the methods on a generic computer is mere-instructions to apply an exception. See MPEP § 2106.05(f))
and one of: provide the set of annotations to a human reviewer to review
(This step for providing / transmitting data to a reviewer after determining the data has a high probability of containing errors is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim recites mental processes and mathematical concepts while the additional elements are mere-instructions to apply an exception and insignificant extra-solution activity.
Additionally, the courts have determined that functions such as data transmitting (provide the set of annotations to a human reviewer to review) are well understood, routine, and conventional activity or insignificant extra-solution activity when recited at a high degree of generality. See MPEP § 2106.05(d, g);
For claim 11,
Step 2A Prong One
(Claim 11 depends on claim 10, which has been determined to recite abstract ideas including mental processes and mathematical concepts. Therefore, claim 11 also recites an abstract idea.)
Step 2A Prong Two
the set of annotations are provided to the human reviewer via presentation in a user interface (UI).
(This step for providing / transmitting annotation data by displaying the annotation data is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts while the additional elements are insignificant extra-solution activity.
Additionally, the courts have determined that functions such as data transmitting and displaying are well understood, routine, and conventional activity or insignificant extra-solution activity when recited at a high degree of generality. See MPEP § 2106.05(d, g);
For claim 12,
Step 2A Prong One
provide, from a plurality of sets of annotations for a region, a limited number of sets of annotations, having a risk score less than the risk threshold, for review by the human reviewer.
(This step for selecting a limited number of sets of annotations meeting a condition for review is a mental process)
Step 2A Prong Two
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application.
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts without any technological improvement or inventive step.
For claim 13,
Step 2A Prong One
the set of aspects includes at least one of a nature of a task for which the one or more annotations are generated, a proportion of objects labeled with a same object classification, a proportion of objects in each of a plurality of distance bins, or a historic performance of the human labeler.
(The aforementioned determined set of aspects are a mental process)
Step 2A Prong Two
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application.
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts without any technological improvement or inventive step.
For claim 14,
Step 2A Prong One
(Claim 14 depends on claim 10, which has been determined to recite abstract ideas including mental processes. Therefore, claim 14 also recites an abstract idea.)
Step 2A Prong Two
modify at least one annotation of the one or more annotations before providing the one or more annotations to the human reviewer in order to influence an engagement of the human reviewer.
(This step for using the modifying the annotation data is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts while the additional element is insignificant extra-solution activity.
Additionally, the courts have determined that functions such as selecting a particular data source to be manipulated / modified [modifying at least one annotation of the one or more annotations…] are well understood, routine, and conventional activity or insignificant extra-solution activity when recited at a high degree of generality. See MPEP § 2106.05(d, g);
For claim 15,
Step 2A Prong One
(Claim 15 depends on claim 10, which has been determined to recite abstract ideas including mental processes and mathematical concepts. Therefore, claim 15 also recites an abstract idea.)
Step 2A Prong Two
the processor is comprised in at least one of: [various systems]
(This step for applying the abstract ideas on a generic computer is considered mere-instructions to apply an exception. See MPEP § 2106.05(f))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes and mathematical concepts while the additional element is considered mere-instructions to apply an exception.
For claim 16,
Step 2A Prong One
determine whether to provide one or more annotations, … based at least on whether a risk score, … meets or exceeds a risk threshold.
(determining based on a threshold is a mental process)
[annotations] to be reviewed by a human reviewer
(reviewing annotations is a mental process)
[risk score] inferred for the one or more annotations
(inferring a risk score is a mental process)
Step 2A Prong Two
[annotations] generated using one or more inputs of a human labeler,
(This step for gathering inputs to generate annotation data is insignificant extra-solution activity. See MPEP § 2106.05(g))
[risk score inferred] using a machine learning model
(This step for inferring risk score for annotation using a machine learning model is merely limiting the abstract idea to a technological environment. See MPEP § 2106.05(h))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim recites mental processes while the additional elements are merely limiting the abstract idea to a technological environment and insignificant extra-solution activity.
Additionally, the courts have determined that functions such as data gathering and outputting ([annotations] generated / output using one or more inputs of a human labeler) are well understood, routine, and conventional activity or insignificant extra-solution activity when recited at a high degree of generality. See MPEP § 2106.05(d, g);
For claim 17,
Step 2A Prong One
infer the risk score based at least on one or more aspects corresponding to the one or more annotations, including at least one of a nature of a task for which the one or more annotations are generated, a proportion of objects labeled with a same object classification, a proportion of objects in each of a plurality of distance bins, or a historic performance of the human labeler.
(The inferred risk score, based on one or more ‘aspects’ is a mental process)
Step 2A Prong Two
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application.
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim recites mental processes without any technological improvement or inventive step.
For claim 18,
Step 2A Prong One
automatically approve the one or more annotations in response to the risk score being less than the risk threshold.
(Approving annotations based on a comparison between a risk score and a threshold is a mental process)
Step 2A Prong Two
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application.
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim recites mental processes without any technological improvement or inventive step.
For claim 19,
Step 2A Prong One
(Claim 19 depends on claim 16, which has been determined to recite abstract ideas including mental processes. Therefore, claim 19 also recites an abstract idea.)
Step 2A Prong Two
modify at least one annotation of the one or more annotations before providing the one or more annotations to the human reviewer in order to influence an engagement of the human reviewer.
(This step for using the modifying the annotation data is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes while the additional element is mere-instructions to apply an exception.
Additionally, the courts have determined that functions such as selecting a particular data source to be manipulated / modified [modifying at least one annotation of the one or more annotations…] are well understood, routine, and conventional activity or insignificant extra-solution activity when recited at a high degree of generality. See MPEP § 2106.05(d, g);
For claim 20,
Step 2A Prong One
(Claim 20 depends on claim 16, which has been determined to recite abstract ideas including mental processes. Therefore, claim 20 also recites an abstract idea.)
Step 2A Prong Two
wherein the system comprises at least one of: [various components]
(This step for applying the abstract ideas on a generic computer is considered mere-instructions to apply an exception. See MPEP § 2106.05(f))
Step 2B
The claim does not include additional elements, when considered separately and in combination, that integrate the judicial exception into a practical application. The claim includes mental processes while the additional element is considered mere-instructions to apply an exception.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 5, 6, 10-11, 13, 15-18, 20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Marc T. Edgar et al. (hereinafter Edgar) (US 20210035015 A1, 2021-02-04).
Regarding claim 1, Edgar teaches;
A computer-implemented method, comprising: receiving, with respect to an instance of data including one or more objects, one or more annotations generated using one or more inputs from a human labeler;
([0027] In a supervised paradigm, the learning system is first given examples of data by which human teachers or annotators apply classification labels to a corpus of data. [0029] human annotators must evaluate image data sets to detect and interpret a large variety of pathophysiology and artifacts in medical imagery and further accurately and consistently label the artifacts.)
NOTE: Edgar teaches receiving, with respect to an instance of data including one or more objects (the data includes artifacts and pathophysiology, which are considered objects), one or more annotations generated using one or more inputs from a human labeler (the learning system is given / receives human annotated data)
determining a set of aspects corresponding to the one or more annotations;
([0034] for manually (i.e., human) applied annotations, the confidence can be determined based on historical accuracy of the entity in association with annotating a particular type of image, disease, etc., self-reported by the annotator, determined based on collaborative review of the annotation by other experts, or the like.)
NOTE: Edgar teaches determining a set of aspects corresponding to the one or more annotations, such as historic performances of each human annotator.
determining, using a machine learning model and based at least on the set of aspects, a probability of the one or more annotations containing an error;
([0082] the annotation accuracy evaluation component 604 can apply the machine learning model M1 to a manually annotated data sample and/or a metadata annotated data sample to generate an inference output and a confidence level in the accuracy of the inference output… ([0034] annotation accuracy evaluation component that determines … a confidence level in the accuracy of an annotation … for manually (i.e., human) applied annotations, the confidence can be determined based on historical accuracy of the entity)
NOTE: Edgar teaches determining, using a machine learning model and based at least on the set of aspects (annotation accuracy component uses a machine learning model and the aforementioned set of aspects [historical annotator accuracy] to determine confidence levels for annotations), a probability of the one or more annotations containing an error (confidence level / probability that the one or more annotations do or do not contain an error).
and one of: providing, in response to the one or more annotations being determined to have higher than a threshold probability of containing an error, the one or more annotations and the instance of data to a human reviewer;
([0043] identify those annotated data samples having annotations with low confidence levels based on their confidence levels being less than a defined threshold level of confidence... send the low confidence annotated data samples 118 back to the annotation queue 114 for additional review and processing (e.g., annotation using a different annotation technique or different entity in implementations in which the incorrect annotation was applied by a manual annotator))
NOTE: Edgar teaches a confidence level which indicates the probability that an annotation is accurate / does not include errors (which is the inverse of indicating the probability that an annotation includes an error), and in response to the confidence level being lower than a confidence threshold (where the confidence threshold is the inverse of an error probability threshold), the one or more annotations and the instance of data (annotated data samples) are sent back to a human / manual reviewer.
This equivalently teaches providing, in response to the one or more annotations being determined to have higher than a threshold probability of containing an error (annotation accuracy confidence level is lower than a confidence threshold, which equivalently indicates that the probability of error is higher than an error threshold), the one or more annotations and the instance of data to a human reviewer (when the aforementioned condition is met, the annotated data samples are sent back to a human / manual annotator).
or approving, in response to the one or more annotations having less than the threshold probability of containing an error, the one or more annotations without providing the one or more annotations to the human reviewer to review.
([0043] identify those annotated data samples having annotations with high confidence levels based on their confidence levels exceeding a defined threshold level of confidence... add the high confidence annotated data samples 116 to the annotated training data set 106.)
NOTE: Edgar teaches in response to the one or more annotations having less than the threshold probability of containing an error (‘in response to annotations having confidence levels exceeding confidence threshold’ equivalently teaches ‘in response to annotations having error probability less than an error threshold’, using the same reasoning provided above), approving the one or more annotations (add the high confidence annotated data samples to the annotated training data set) without providing the one or more annotations to the human reviewer to review (the high confidence annotated data samples are not sent back to human reviewers).
Regarding claim 2, Edgar teaches;
wherein the providing the one or more annotations and the instance of data to the human reviewer includes causing display of the instance of data along with at least one annotation of the one or more annotations within a user interface (UI).
([0043] send the low confidence annotated data samples 118 back to the annotation queue 114 for additional review and processing (e.g., annotation using a … different entity in implementations in which the incorrect annotation was applied by a manual annotator))
NOTE: Providing the annotations and the instance of data (annotated data samples) to the human reviewer (e.g. manual / human annotator).
([0067] interface with a manual annotation application that presents one or more manual annotators (humans) with unannotated (or in some implementations previously annotated) data samples…)
NOTE: Edgar teaches wherein the providing the one or more annotations and the instance of data to the human reviewer (providing previously annotated data samples / instances to the human reviewer / annotator) includes causing display of the instance of data along with at least one annotation of the one or more annotations within a user interface (UI) (an interface which presents / displays the annotated data samples to human users).
Regarding claim 3, Edgar teaches;
wherein the one or more annotations are classified as: high-risk based at least on the probability of the one or more annotations containing an error being higher than the threshold probability;
([0043] The annotation pipeline module 112 can also identify those annotated data samples having annotations with low confidence levels based on their confidence levels being less than a defined threshold level of confidence.)
NOTE: Edgar teaches identifying / classifying annotations as low confidence / high-risk based at least on the annotations containing a confidence level being lower than the confidence threshold (which equivalently teaches annotations containing an error being higher than a threshold indicating probability of error, using the same reasoning provided in claim 1).
or low-risk based at least on the probability of the one or more annotations containing an error being less than the threshold probability.
([0043] The annotation pipeline module 112 can further identify those annotated data samples having annotations with high confidence levels based on their confidence levels exceeding a defined threshold level of confidence.)
NOTE: Edgar teaches identifying / classifying annotations as high confidence / low-risk based at least on the annotations having confidence levels exceeding the confidence threshold (which is equivalent to annotations containing an error less than a threshold indicating probability of error, using the same reasoning from claim 1).
Regarding claim 5, Edgar teaches;
The computer-implemented method of claim 1, wherein the set of aspects includes at least one of a nature of a task for which the one or more annotations are generated, a proportion of objects labeled with a same object classification, a proportion of objects in each of a plurality of distance bins, or a historic performance of the human labeler.
([0034] for manually (i.e., human) applied annotations, the confidence can be determined based on historical accuracy of the entity in association with annotating a particular type of image)
NOTE: Edgar teaches the set of aspects including at least a historic performance of the human labeler.
Regarding claim 6, Edgar teaches;
wherein the machine learning model includes a linear regression model.
([0082] the annotation accuracy evaluation component 604 can apply the machine learning model M1 to a manually annotated data sample and/or a metadata annotated data sample to generate an inference output and a confidence level in the accuracy of the inference output... [0045] For example, the machine learning model M1 can be or include ... a linear regression model)
Regarding claim 10, Edgar teaches;
At least one processor, comprising:
([0004] According to an embodiment, a system can comprise a memory that stores computer executable components and a processor)
determine one or more aspects corresponding to a set of annotations generated using a human labeler as part of a labeling task;
([0034] for manually (i.e., human) applied annotations, the confidence can be determined based on historical accuracy of the entity in association with annotating a particular type of image, disease, etc., self-reported by the annotator, determined based on collaborative review of the annotation by other experts, or the like.)
NOTE: Edgar teaches determining one or more aspects (historical accuracy of the entity in association with annotating a particular type of image) to a set of annotations generated using a human labeler as part of a labeling task (manually / human applied annotations / labels)
generate, using a machine learning model and based at least on the one or more aspects, a risk score indicating a probability that the set of annotations includes at least one error;
([0082] the annotation accuracy evaluation component 604 can apply the machine learning model M1 to a manually annotated data sample and/or a metadata annotated data sample to generate an inference output and a confidence level in the accuracy of the inference output… ([0034] annotation accuracy evaluation component that determines … a confidence level in the accuracy of an annotation … for manually (i.e., human) applied annotations, the confidence can be determined based on historical accuracy of the entity)
NOTE: Edgar teaches a confidence score, which indicates a confidence in the level of accuracy for an annotation. The confidence score can equivalently be represented as the inverse of the confidence score, which would be the probability that an annotation includes errors, i.e., a ‘risk score’.
Thus, Edgar teaches generating, using a machine learning model (apply machine learning model M1) and based at least on the one or more aspects (determined based at least on the aforementioned ‘historical accuracy’ aspects), a risk score indicating a probability that the set of annotations includes at least one error (the confidence score, which can be equivalently represented as the inverse ‘risk score’, indicating probability of error).
and one of: provide the set of annotations to a human reviewer to review in response to determining that the risk score exceeds a risk threshold;
([0043] identify those annotated data samples having annotations with low confidence levels based on their confidence levels being less than a defined threshold level of confidence... send the low confidence annotated data samples 118 back to the annotation queue 114 for additional review and processing (e.g., annotation using a different annotation technique or different entity in implementations in which the incorrect annotation was applied by a manual annotator))
NOTE: Edgar teaches a confidence level which indicates the probability that an annotation is accurate / does not include errors (which is the inverse of indicating the probability that an annotation includes an error, i.e., a risk score), and in response to the confidence level being lower than a confidence threshold (where the confidence threshold is the inverse of an error probability / risk threshold), the one or more annotations and the instance of data (annotated data samples) are sent back to a human / manual reviewer.
This equivalently teaches providing, in response to determining that the risk score exceeds a risk threshold (annotation accuracy confidence level is lower than a confidence threshold, which equivalently indicates that the inverse ‘risk score’ indicating probability of error is higher than the ‘risk threshold’), provide the set of annotations to a human reviewer (when the aforementioned condition is met, the annotated data samples are sent back to a human / manual annotator).
or automatically approve the set of annotations in response to the risk score being less than the risk threshold.
([0043] identify those annotated data samples having annotations with high confidence levels based on their confidence levels exceeding a defined threshold level of confidence... add the high confidence annotated data samples 116 to the annotated training data set 106.)
NOTE: Edgar teaches in response to the risk score being less than the risk threshold (‘in response to annotations having confidence levels exceeding confidence threshold’ equivalently teaches ‘in response to annotations having risk score less than a risk threshold,’ using the reasoning provided above), automatically approve the set of annotations (add the high confidence annotated data samples to the annotated training data set without being sent back to human reviewers).
Regarding claim 11, Edgar teaches;
the set of annotations are provided to the human reviewer via presentation in a user interface (UI).
([0043] send the low confidence annotated data samples 118 back to the annotation queue 114 for additional review and processing (e.g., annotation using a … different entity in implementations in which the incorrect annotation was applied by a manual annotator))
NOTE: Providing the annotations and the instance of data (annotated data samples) to the human reviewer (e.g. manual / human annotator).
([0067] interface with a manual annotation application that presents one or more manual annotators (humans) with unannotated (or in some implementations previously annotated) data samples…)
NOTE: The annotations are provided to the human reviewer via a user interface.
Regarding claim 13, Edgar teaches;
wherein the set of aspects includes at least one of a nature of a task for which the one or more annotations are generated, a proportion of objects labeled with a same object classification, a proportion of objects in each of a plurality of distance bins, or a historic performance of the human labeler.
([0034] for manually (i.e., human) applied annotations, the confidence can be determined based on historical accuracy of the entity in association with annotating a particular type of image)
NOTE: Edgar teaches the set of aspects including at least a historic performance of the human labeler.
Regarding claim 15, Edgar teaches;
wherein the processor is comprised in at least one of:
PNG
media_image1.png
444
676
media_image1.png
Greyscale
PNG
media_image2.png
83
631
media_image2.png
Greyscale
([0126] a component can be an apparatus that provides specific functionality through electronic components without mechanical parts, wherein the electronic components can include a processor or other means to execute software or firmware that confers at least in part the functionality of the electronic components. In an aspect, a component can emulate an electronic component via a virtual machine, e.g., within a cloud computing system.)
Regarding claim 16, Edgar teaches;
A system, comprising: one or more processors
to determine whether to provide one or more annotations, … to be reviewed by a human reviewer based at least on whether a risk score, … meets or exceeds a risk threshold.
[annotations] generated using one or more inputs of a human labeler,
[risk score] inferred for the one or more annotations using a machine learning model,
(These limitations are substantially similar to the limitations of claim 10, and are rejected using the same reasoning provided in claim 10)
Regarding claim 17, Edgar teaches;
infer the risk score based at least on one or more aspects corresponding to the one or more annotations, including at least one of a nature of a task for which the one or more annotations are generated, a proportion of objects labeled with a same object classification, a proportion of objects in each of a plurality of distance bins, or a historic performance of the human labeler.
([0082] the annotation accuracy evaluation component 604 can apply the machine learning model M1 to a manually annotated data sample and/or a metadata annotated data sample to generate an inference output and a confidence level in the accuracy of the inference output… ([0034] annotation accuracy evaluation component that determines … a confidence level in the accuracy of an annotation … for manually (i.e., human) applied annotations, the confidence can be determined based on historical accuracy of the entity)
NOTE: The inferred risk score (the inverse of the confidence score, as previously taught) is based on at least the historic performance of a human labeler / annotator.
Regarding claim 18,
Claim 18 is substantially similar to the last limitation of claim 10, and is taught using the same reasoning.
Regarding claim 20,
Claim 20 is substantially similar to claim 15, and is taught using the same reasoning.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Edgar (US 20210035015 A1, 2021-02-04) as applied to claim 1 above, further in view of Pietro Perona et al. (hereinafter Perona) (US 20190034831 A1, 2019-01-31).
Regarding claim 4, Edgar fails to teach but Perona teaches;
wherein the threshold probability is adjustable based in part upon a risk tolerance.
([0055] In a variety of embodiments, the risk associated with a predicted annotation can be utilized to calculate the confidence in the predicted annotation using any of a variety of techniques, including by calculating probability estimations and by calculating an inverse relationship between risk and confidence. A logical criterion is to accept y.sub.i once the risk drops below a certain threshold (y.sub.i)≤τ.sub.∈, with τ.sub.∈ being the minimum tolerable error per piece of source data… and the risk criterion can be used to determine whether or not a piece of source data is finished (e.g. when a label correctly identifying the piece of source data has been determined) or may require more annotations.)
NOTE: Perona teaches a risk probability threshold for annotation data which is adjustable based on the minimum tolerable error / risk per piece of source data, and using this data to determine if data needs to be reviewed / receive additional annotation.
OBVIOUSNESS TO COMBINE PERONA WITH EDGAR:
Perona is analogous art to the present disclosure as it pertains to annotation data, human annotators, and a risk threshold for annotation data.
Edgar already teaches determining whether annotation data needs to be reviewed / reannotated using a threshold value, while Perona teaches a very similar process with a threshold value that is adjustable based on a risk tolerance.
Having the threshold adjustable based on a risk tolerance would predictably allow the annotation Pipeline of Edgar to adjust the confidence / probability threshold according to the amount of annotation error that can be tolerated for a given data item, thereby improving balance between annotation accuracy and efficiency.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to configure the threshold in the annotation pipeline of Edgar to be adjustable based on a risk tolerance (as taught by Perona) to improve the balance between annotation accuracy and efficiency.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Edgar (US 20210035015 A1, 2021-02-04) as applied to claim 1 above, further in view of Tilman Wekel et al. (hereinafter Wekel) (US 20220277193 A1, 2022-09-01).
Regarding claim 7, Edgar teaches;
wherein the one or more annotations are to be used as ground truth data to update one or more parameters of a
([0044] the annotated training data set 106 can include an initial set of annotated training data samples that can be used to initiate training and development of the machine learning model M1. For example, the initial annotated training data samples can include manually labeled/annotated data samples that are known to be accurate (e.g., providing ground truth examples))
NOTE: Edgar teaches the one or more annotations are to be used as ground truth data to update one or more parameters of a machine learning model, because training a machine learning model includes updating one or more parameters of the machine learning model.
Edgar fails to teach but Wekel teaches;
a second machine learning model
([0042] After some or all annotations tasks have been completed, the resulting ground truth data may be exported from the labeling tool in any suitable format, whether automatically or manually triggered (ground truth data export 180), and the ground truth data may be consumed (ground truth data consumption 190), for example, by using the ground truth data to train a corresponding DNN.)
NOTE: Wekel teaches using annotations as ground truth data to update parameters (train) of a second (external) machine learning model.
OBVIOUSNESS TO COMBINE WEKEL WITH EDGAR:
Wekel is analogous art to the present disclosure as it pertains to using annotations as ground truth data for training a machine learning model.
Edgar already provides the base annotation pipeline and using annotation data as ground truth for training a machine learning model, while Wekel teaches using annotation data as ground truth data for training a second, external, machine learning model.
Wekel further indicates that the ground truth data produced by their annotation pipeline can be customized according to the needs of the particular downstream (second / external) machine learning model;
([0038] the ground truth data produced by an annotation pipeline may be customized based on the type of DNN to be trained.)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the annotation pipeline of Edgar to use model specific annotation data as ground truth to train a second, external, machine learning model (as taught by Wekel), predictably improving the accuracy of the second, external machine learning model.
Claim(s) 8, 14, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Edgar (US 20210035015 A1, 2021-02-04) as applied to claims 1, 10, and 16 above, further in view of David Oleson et al. (hereinafter Oleson) (“Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing,” 2011).
Regarding claim 8, Edgar teaches;
One or more annotations
Using the same reasoning presented in claim 1
modifying
Edgar fails to teach but Oleson teaches;
([pg. 4] Programmatic gold is a process of generating gold units with known answers. New units are generated from previously collected correct data. By injecting known types of errors into the data, we gain control over the training experience of the workers and force all workers to consider most common error cases. When the workers make mistakes, they see automatically-generated feedback)
NOTE: Oleson teaches modifying data by injecting errors into the data before providing the data to human reviewers / workers, the modifying performed to influence engagement of the human reviewer by forcing them to consider common error cases.
OBVIOUSNESS TO COMBINE OLESON WITH EDGAR:
Oleson is analogous art to the present disclosure as it pertains to modifying data to be presented to human reviewers to influence engagement.
Edgar already teaches providing annotations to human reviewers, while Oleson teaches modifying data before providing it to human reviewers / workers.
Oleson additionally indicates that their process of modifying data before review improves accuracy of responses from the human reviewers / workers;
([pg. 3] Gold questions are used to improve the accuracy of results collected in crowdsourced tasks… These questions aim to: a) remove unethical workers from a task and b) educate untrained or incompetent workers to improve the accuracy of their responses.)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify at least one of the annotations of the one or more annotations of Edgar before providing them to the human reviewer using the process described by Oleson to improve accuracy of the human reviewers.
Claim 14 is substantially similar to claim 8, and is rejected using the same reasoning.
Claim 19 is substantially similar to claim 8, and is rejected using the same reasoning.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Edgar (US 20210035015 A1, 2021-02-04) as applied to claim 1 above, further in view of Aybora Koksal et al. (hereinafter Koksal) (“Effect of Annotation Errors on Drone Detection with YOLOv3,” 2021).
Regarding claim 9, Edgar and Oleson fail to teach but Koksal teaches;
wherein the at least one annotation is modified by, at least one of: adding an annotation; deleting an annotation; or shifting a location of the annotation.
([pg. 5] additional annotation errors are applied to the dataset in order to create separate new datasets each consisting different type of errors and some combined ones.)
NOTE: Koksal adds annotation errors to data.
([pg. 5-6] Simulations of Various Annotation Errors… Additional boxes: This type of error includes an extra box which does not contain any target… Missing boxes: A missing box error is simply due to the unavailability of the annotation of a true object… Shifted boxes: A shifted box error is a slightly translated version of the true object box.)
NOTE: The added annotation errors disclosed by Koksal include modifying annotations by adding annotations (additional boxes), deleting annotations (missing boxes), and shifting annotations (shifted boxes).
OBVIOUSNESS TO COMBINE KOKSAL WITH EDGAR AND OLESON: Koksal is analogous art to the present disclosure as it pertains to modifying annotations by adding, removing, or shifting annotations.
Oleson supplies the reason for inserting known errors into data to be provided to human reviewers (improving accuracy);
([Oleson, pg. 4] By injecting known types of errors into the data, we gain control over the training experience of the workers and force all workers to consider most common error cases… [Oleson, pg. 3] These questions aim to: a) remove unethical workers from a task and b) educate untrained or incompetent workers to improve the accuracy of their responses.)
While Koksal teaches the modifying annotations by including common annotation errors in data (added, removed, shifted annotations).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the annotations of Edgar before providing them to human reviewers using the process of Oleson, the specific modifications including removing, adding, and shifting annotations (additional, missing, and shifted annotations, as taught by Koksal) to have workers consider common annotation errors, thereby improving the accuracy of the reviewers when responding to the provided annotations.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Edgar (US 20210035015 A1, 2021-02-04) as applied to claim 1 above, further in view of Jun Zhan et al. (hereinafter Zhan) (US 20180108146 A1, 2018-04-19).
Regarding claim 12, Edgar teaches;
a plurality of sets of annotations for a region
[fig. 4]
PNG
media_image3.png
210
254
media_image3.png
Greyscale
NOTE: The data samples of Edgar each pertain to a region of data
([0105] a data sample of the data samples for annotation using different annotation processes, resulting in generation of a plurality of annotations for the data sample)
NOTE: Edgar teaches a plurality data samples that each contain a set of annotations for a region. Thus, Edgar teaches a plurality of sets of annotations (data samples each containing a set of annotations) for a region.
Edgar fails to teach but Zhan teaches;
provide, from a plurality of
([Abstract] The present application discloses a method and apparatus for annotating point cloud data… [0064] In some alternative implementations of this embodiment, the check operation may be a manual spot check performed by the user on the point cloud segmentation and tracking result. In practice, the point cloud segmentation and tracking result of which the confidence level is greater than the confidence level threshold may be presented to the user for spot check. If the spot check is passed, the point cloud segmentation and tracking result can be used in subsequent steps.)
NOTE: Zhan teaches selecting results with a confidence score higher than a confidence threshold (which is equivalent to a risk-score being lower than a risk threshold, using the same reasoning from claim 1) for review by a human reviewer (presented to the user to be checked).
Thus, Zhan teaches providing, from a plurality of results, a limited number of annotated results (the selected portion teaches a limited number of results), having a risk score less than the risk threshold (the selected results having confidence above the confidence threshold / risk less than the risk threshold), for review by the human reviewer (presented to the user to be checked).
OBVIOUSNESS TO COMBINE ZHAN WITH EDGAR:
Zhan is analogous art to the present disclosure as it pertains to data annotation.
Edgar already teaches a plurality of sets of annotations for a region, and providing the sets of annotations for human review based on a threshold value.
Zhan teaches selecting a limited number of results having a risk score less than a risk threshold (confidence score greater than confidence threshold) to be provided for review by a human reviewer.
Zhan additionally states that this method of checking the selected low-risk results allows for more accurate annotation results to be obtained;
([0067] the flow 300 of the method for annotating point cloud data in this embodiment highlights the check operation on the point cloud segmentation and tracking result of which the confidence level is greater than the confidence level threshold. Whereby, a more accurate annotation result can be obtained, and a better point cloud recognition algorithm can be obtained through training by using the annotation result.)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to apply the checking method taught by Zhan to the plurality of sets of annotations for a region of Edgar allow for more accurate annotation results to be obtained in the annotation pipeline.
From this, in combination, Edgar and Zhan reasonably teach;
provide, from a plurality of sets of annotations for a region [Edgar], a limited number of sets of annotations, having a risk score less than the risk threshold, for review by the human reviewer [Zhan].
CONCLUSION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights