3Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 4 and 14 objected to because of the following informalities: “logisitic” is misspelled. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
101 Subject Matter Eligibility Analysis
Step 1: Claims 1-20 are within the four statutory (a process, machine, manufacture or composition of matter.) Claims 1-10 describe a process and 11-20 describes a machine.
With respect to claim 1:
Step 2A Prong 1: The claim recites an abstract idea enumerated in the 2019 PEG
calculating at least one confidence metric for each crowdworker based on the plurality of annotations received from each crowdworker; (This is an abstract idea of a "Mental Process." The "calculating" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The calculation could be made manually by an individual.)
identifying a plurality of super recognizers from the plurality of crowdworkers based on the at least one confidence metric associated with the at least one crowdworker; (This is an abstract idea of a "Mental Process." The "identifying" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The identification could be made manually by an individual.)
aggregating the second plurality of annotations; (This is an abstract idea of a "Mental Process." The "aggregating" step under its broadest reasonable interpretation, covers concepts that can be practically performed by a human using a pen and paper.)
generating a training data set by merging the aggregated second plurality of annotations and the unlabeled data set; and (This is an abstract idea of a "Mental Process." The "generating" step under its broadest reasonable interpretation, covers concepts that can be practically performed by a human using a pen and paper.)
Step 2A Prong 2: The judicial exception is not integrated into a practical application
Additional elements:
obtaining an evaluation data set comprising a plurality of inputs and a plurality of outputs: (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
where each output in the plurality of outputs uniquely corresponds to an input in the plurality of inputs; and (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
where each output is assumed to accurately label its corresponding input; (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
providing the plurality of inputs to a plurality of crowdworkers; (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
receiving a plurality of annotations from each crowdworker in the plurality of crowdworkers, where each plurality of annotations comprises an annotation for at least one input in the plurality of inputs; (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
obtaining an unlabeled data set; (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
providing the unlabeled data set to each crowdworker in the plurality of super recognizers; (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
receiving a second plurality of annotations from each crowdworker in the plurality of super recognizers;(this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
training a machine learning model using the generated training data set.(This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional elements add insignificant extra-solution activity to the judicial exception and cannot provide an inventive concept. Storing and retrieving information in memory is directed to a well understood routine conventional activity of data transmission (MPEP 2106.05(d)(II)(iv)).
The additional element “training a machine learning model using the generated training data set” is recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept (MPEP 2106.05(f)).
When considered in combination, these additional elements represent insignificant extra-solution activity and mere instructions to apply an expectation, which do not provide an inventive concept.
Therefore, claim 1 is ineligible.
With respect to claim 2:
Step 2A Prong 1: claim 2, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
the at least one confidence metric is the probability of correct classification (PCC) of a pretrained machine learning model. (this limitation merely limits the judicial exception to a particular field of use.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element merely limits the judicial exception to a particular field of use and cannot provide an inventive concept (MPEP 2106.05(h)).
Therefore, claim 2 is ineligible.
With respect to claim 3:
Step 2A Prong 1: claim 3, which incorporates the rejection of claim 2, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
the PCC is calculated for a given crowdworker in the plurality of crowdworkers by providing a recognizer machine learning model with a given plurality of annotations for the plurality of inputs generated by the given crowdworker. (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element is recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept (MPEP 2106.05(f)).
Therefore, claim 3 is ineligible.
With respect to claim 4:
Step 2A Prong 1: claim 4, which incorporates the rejection of claim 3, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
the recognizer machine learning model is a binary logisitic regression classifier. (this limitation merely limits the judicial exception to a particular field of use.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element merely limits the judicial exception to a particular field of use and cannot provide an inventive concept (MPEP 2106.05(h)).
Therefore, claim 4 is ineligible.
With respect to claim 5:
Step 2A Prong 1: claim 5, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
the at least one confidence metric is selected from the group consisting of: a test-retest metric, a reliability metric, a penalized time metric, and a time spent metric. (this limitation merely limits the judicial exception to a particular field of use.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element merely limits the judicial exception to a particular field of use and cannot provide an inventive concept (MPEP 2106.05(h)).
Therefore, claim 5 is ineligible.
With respect to claim 6:
Step 2A Prong 1: claim 6, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
the plurality of inputs are provided to a plurality of crowdworkers in response to crowdworkers in the plurality of crowdworkers responding to a request on a crowdsourcing platform. (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element adds insignificant extra-solution activity to the judicial exception and cannot provide an inventive concept. Storing and retrieving information in memory is directed to a well understood routine conventional activity of data transmission (MPEP 2106.05(d)(II)(iv)).
Therefore, claim 6 is ineligible.
With respect to claim 7:
Step 2A Prong 1: claim 7, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
the plurality of crowdworkers comprise crowdworkers who have completed one or more requests. (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element adds insignificant extra-solution activity to the judicial exception and cannot provide an inventive concept. Storing and retrieving information in memory is directed to a well understood routine conventional activity of data transmission (MPEP 2106.05(d)(II)(iv)).
Therefore, claim 7 is ineligible.
With respect to claim 8:
Step 2A Prong 1: claim 8, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
the plurality of inputs and the unlabeled data set both comprise anonymized videos of children with Autism Spectrum Disorder (ASD). (this limitation merely limits the judicial exception to a particular field of use.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element merely limits the judicial exception to a particular field of use and cannot provide an inventive concept (MPEP 2106.05(h)).
Therefore, claim 8 is ineligible.
With respect to claim 9:
Step 2A Prong 1: claim 9, which incorporates the rejection of claim 8, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
the plurality of outputs, the plurality of annotations, and the second plurality of annotations comprise responses to a questionnaire. (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element adds insignificant extra-solution activity to the judicial exception and cannot provide an inventive concept. Storing and retrieving information in memory is directed to a well understood routine conventional activity of data transmission (MPEP 2106.05(d)(II)(iv)).
Therefore, claim 9 is ineligible.
With respect to claim 10:
Step 2A Prong 1: claim 10, which incorporates the rejection of claim 8, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
the machine learning model is trained to identify ASD in videos of children. (this limitation merely limits the judicial exception to a particular field of use.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element merely limits the judicial exception to a particular field of use and cannot provide an inventive concept (MPEP 2106.05(h)).
Therefore, claim 10 is ineligible.
With respect to claim 11:
The claim recites similar limitations as corresponding to claim 1. Therefore, the same subject matter analysis that was utilized for claim 1, as described above, is equally applicable to claim 11. Therefore, claim 11 is ineligible.
With respect to claim 12:
The claim recites similar limitations as corresponding to claim 2. Therefore, the same subject matter analysis that was utilized for claim 2, as described above, is equally applicable to claim 12. Therefore, claim 12 is ineligible.
With respect to claim 13:
The claim recites similar limitations as corresponding to claim 3. Therefore, the same subject matter analysis that was utilized for claim 3, as described above, is equally applicable to claim 13. Therefore, claim 13 is ineligible.
With respect to claim 14:
The claim recites similar limitations as corresponding to claim 4. Therefore, the same subject matter analysis that was utilized for claim 4, as described above, is equally applicable to claim 14. Therefore, claim 14 is ineligible.
With respect to claim 15:
The claim recites similar limitations as corresponding to claim 5. Therefore, the same subject matter analysis that was utilized for claim 5, as described above, is equally applicable to claim 15. Therefore, claim 15 is ineligible.
With respect to claim 16:
The claim recites similar limitations as corresponding to claim 6. Therefore, the same subject matter analysis that was utilized for claim 6, as described above, is equally applicable to claim 16. Therefore, claim 16 is ineligible.
With respect to claim 17:
The claim recites similar limitations as corresponding to claim 7. Therefore, the same subject matter analysis that was utilized for claim 7, as described above, is equally applicable to claim 17. Therefore, claim 17 is ineligible.
With respect to claim 18:
The claim recites similar limitations as corresponding to claim 8. Therefore, the same subject matter analysis that was utilized for claim 8, as described above, is equally applicable to claim 18. Therefore, claim 18 is ineligible.
With respect to claim 19:
The claim recites similar limitations as corresponding to claim 9. Therefore, the same subject matter analysis that was utilized for claim 9, as described above, is equally applicable to claim 19. Therefore, claim 19 is ineligible.
With respect to claim 20:
The claim recites similar limitations as corresponding to claim 10. Therefore, the same subject matter analysis that was utilized for claim 10, as described above, is equally applicable to claim 20. Therefore, claim 20 is ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7 and 11-17 are rejected under 35 U.S.C. 103 as being unpatentable over Larlus-Larrondo (US 2015/0235160 A1) in view of Rodrigues (NPL: ‘Learning from multiple Annotators: Distinguishing good from random labelers’).
Regarding claim 1, Larlus-Larrondo teaches:
A method for crowdsourced machine learning, comprising ([0017] “In accordance with another aspect of the exemplary embodiment, a method for generating a human intelligence task includes computing a measure of popularity for each of a set of classes to be used in labeling documents.”)
obtaining an evaluation data set comprising a plurality of inputs and a plurality of outputs: ([0006] “One mechanism to identify unreliable workers is to hide what is called a "gold question" in the HIT. This is a question for which the answer is known a priori. The assumption is that, if a worker provides the correct answer for the gold question, then the worker is likely to provide reliable answers for the rest of the task.”)
where each output in the plurality of outputs uniquely corresponds to an input in the plurality of inputs; and ([0051] “At S112 at least one gold question 44 is generated. In particular, a document, e.g., an image 40, that has, as its label, the label of the positive class, is selected from the labeled samples 62 by the gold question generator 50.”)
where each output is assumed to accurately label its corresponding input; ([0051] “At S112 at least one gold question 44 is generated. In particular, a document, e.g., an image 40, that has, as its label, the label of the positive class, is selected from the labeled samples 62 by the gold question generator 50.” Positive class represents accurate label)
providing the plurality of inputs to a plurality of crowdworkers; ([0033] “One query image 40 and several candidate labels 38 (e.g., 5 choices) are provided. The crowdworker is asked to select the most appropriate label.”)
receiving a plurality of annotations from each crowdworker in the plurality of crowdworkers, where each plurality of annotations comprises an annotation for at least one input in the plurality of inputs; ([0054] “At S118, the responses 72 are received from human annotators and checked by the task outsourcer 64 for reliability, e.g., by comparing the answer to each gold question 44 with the true answer.”)
Larlus-Larrondo does not teach:
calculating at least one confidence metric for each crowdworker based on the plurality of annotations received from each crowdworker;
identifying a plurality of super recognizers from the plurality of crowdworkers based on the at least one confidence metric associated with the at least one crowdworker;
obtaining an unlabeled data set;
providing the unlabeled data set to each crowdworker in the plurality of super recognizers;
receiving a second plurality of annotations from each crowdworker in the plurality of super recognizers;
aggregating the second plurality of annotations;
generating a training data set by merging the aggregated second plurality of annotations and the unlabeled data set; and
training a machine learning model using the generated training data set.
Rodrigues does:
calculating at least one confidence metric for each crowdworker based on the plurality of annotations received from each crowdworker; (Section 4. ‘Proposed Model’ “can be interpreted as the probability of an annotator providing a correct label or, in other words, as an indicator of how reliable an annotator is.”)
identifying a plurality of super recognizers from the plurality of crowdworkers based on the at least one confidence metric associated with the at least one crowdworker; (Section 4.2 ‘Expectation-Maximization’ “in this case, the contributions of the labels provided by each annotator to the loglikelihood are being weighted by her reliability, or in other words, by how likely it is for her to be correct. This makes our proposed approach quite easy to implement in practice.”)
obtaining an unlabeled data set; (Section 5.2 ‘Amazon Mechanical Turk’ describes the datasets the gave to annotators)
providing the unlabeled data set to each crowdworker in the plurality of super recognizers; (Section 5.2 ‘Amazon Mechanical Turk’ describes the datasets the gave to annotators)
receiving a second plurality of annotations from each crowdworker in the plurality of super recognizers; (Table 2 summarizes the answers they got from the annotators)
aggregating the second plurality of annotations; (Table 2 summarizes the answers they got from the annotators)
generating a training data set by merging the aggregated second plurality of annotations and the unlabeled data set; and (Introduction describes using the datasets to train machine learning models).
training a machine learning model using the generated training data set. (Introduction describes using the datasets to train machine learning models).
Larlus-Larrondo and Rodrigues are considered analogous art to the claimed invention because they are in the same field of endeavor being crowdsourcing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the crowdsourcing method of Larlus-Larrondo with the evaluation method of Rodrigues. One would want to do this to get more reliable crowdworkers.
Regarding claim 2, Rodrigues teaches:
the at least one confidence metric is the probability of correct classification (PCC) of a pretrained machine learning model. (Section 4. ‘Proposed Model’ “can be interpreted as the probability of an annotator providing a correct label or, in other words, as an indicator of how reliable an annotator is.”)
Regarding claim 3, Rodrigues teaches:
the PCC is calculated for a given crowdworker in the plurality of crowdworkers by providing a recognizer machine learning model with a given plurality of annotations for the plurality of inputs generated by the given crowdworker (Section 4. ‘Proposed Model’ “can be interpreted as the probability of an annotator providing a correct label or, in other words, as an indicator of how reliable an annotator is.” Also in this section they describe the model being a Logistic Regression model)
Regarding claim 4, Rodrigues teaches:
the recognizer machine learning model is a binary logisitic regression classifier. (Section 4. ‘Proposed Model’ describes the logistic regression model that is used on binary variables).
Regarding claim 5, Larlus-Larrondo teaches:
the at least one confidence metric is selected from the group consisting of: a test-retest metric, a reliability metric, a penalized time metric, and a time spent metric. ([0088] “For example, if the percentage of errors on a gold question is .epsilon.=10% and the aim is to declare a sincere worker to be insincere not more than 1% of the time, then m=2 (or more) gold questions per HIT may be chosen and a worker is considered insincere if all gold questions are answered incorrectly.” This describes using many gold questions and testing the workers multiple times.)
Regarding claim 6, Larlus-Larrondo teaches:
the plurality of inputs are provided to a plurality of crowdworkers in response to crowdworkers in the plurality of crowdworkers responding to a request on a crowdsourcing platform. ([0100] “Each HIT is composed of 3 questions. Two of them (standard questions) are based on query images from the test set. A gold question is placed in each HIT (third question) to assess the motivation of workers. The order of the three questions is randomized. To assist the annotator, an image of a bird prelabeled with the class is provided with each candidate answer (except for the answer "none"). The annotator is also given the opportunity to request one more additional photographs for each bird class. The annotator (typically a crowdworker volunteering to perform the task for a small payment on a crowdsourcing marketplace) is asked to click on one of the answers (one of them being the "none" option).”).
Regarding claim 7, Rodrigues teaches:
the plurality of crowdworkers comprise crowdworkers who have completed one or more requests. (Section 5.2 ‘Amazon Mechanical Turk’ “the AMT workers were required to have an HIT approval rate – an AMT quality indicator that reflects the percentage of accepted answers of a worker – of 95%, which ensures some reliability on the quality of the answers.”).
Regarding claim 11, Larlus-Larrondo teaches:
A crowdsourced machine learning device, comprising: a processor; and a memory, the memory containing a crowdsourced machine learning application capable of direction the processor to: ([0024] “With reference to FIG. 1, a computer-implemented system 10 for formulation of gold questions for annotation tasks is shown. The system includes memory 12 which stores instructions 14 for generating gold questions 16 to be incorporated into crowdsourcing tasks 18 and a processor 20 in communication with the memory for executing the instructions.”)
obtaining an evaluation data set comprising a plurality of inputs and a plurality of outputs: ([0006] “One mechanism to identify unreliable workers is to hide what is called a "gold question" in the HIT. This is a question for which the answer is known a priori. The assumption is that, if a worker provides the correct answer for the gold question, then the worker is likely to provide reliable answers for the rest of the task.”)
where each output in the plurality of outputs uniquely corresponds to an input in the plurality of inputs; and ([0051] “At S112 at least one gold question 44 is generated. In particular, a document, e.g., an image 40, that has, as its label, the label of the positive class, is selected from the labeled samples 62 by the gold question generator 50.”)
where each output is assumed to accurately label its corresponding input; ([0051] “At S112 at least one gold question 44 is generated. In particular, a document, e.g., an image 40, that has, as its label, the label of the positive class, is selected from the labeled samples 62 by the gold question generator 50.” Positive class represents accurate label)
providing the plurality of inputs to a plurality of crowdworkers; ([0033] “One query image 40 and several candidate labels 38 (e.g., 5 choices) are provided. The crowdworker is asked to select the most appropriate label.”)
receiving a plurality of annotations from each crowdworker in the plurality of crowdworkers, where each plurality of annotations comprises an annotation for at least one input in the plurality of inputs; ([0054] “At S118, the responses 72 are received from human annotators and checked by the task outsourcer 64 for reliability, e.g., by comparing the answer to each gold question 44 with the true answer.”)
Larlus-Larrondo does not teach:
calculating at least one confidence metric for each crowdworker based on the plurality of annotations received from each crowdworker;
identifying a plurality of super recognizers from the plurality of crowdworkers based on the at least one confidence metric associated with the at least one crowdworker;
obtaining an unlabeled data set;
providing the unlabeled data set to each crowdworker in the plurality of super recognizers;
receiving a second plurality of annotations from each crowdworker in the plurality of super recognizers;
aggregating the second plurality of annotations;
generating a training data set by merging the aggregated second plurality of annotations and the unlabeled data set; and
training a machine learning model using the generated training data set.
Rodrigues does:
calculating at least one confidence metric for each crowdworker based on the plurality of annotations received from each crowdworker; (Section 4. ‘Proposed Model’ “can be interpreted as the probability of an annotator providing a correct label or, in other words, as an indicator of how reliable an annotator is.”)
identifying a plurality of super recognizers from the plurality of crowdworkers based on the at least one confidence metric associated with the at least one crowdworker; (Section 4.2 ‘Expectation-Maximization’ “in this case, the contributions of the labels provided by each annotator to the loglikelihood are being weighted by her reliability, or in other words, by how likely it is for her to be correct. This makes our proposed approach quite easy to implement in practice.”)
obtaining an unlabeled data set; (Section 5.2 ‘Amazon Mechanical Turk’ describes the datasets the gave to annotators)
providing the unlabeled data set to each crowdworker in the plurality of super recognizers; (Section 5.2 ‘Amazon Mechanical Turk’ describes the datasets the gave to annotators)
receiving a second plurality of annotations from each crowdworker in the plurality of super recognizers; (Table 2 summarizes the answers they got from the annotators)
aggregating the second plurality of annotations; (Table 2 summarizes the answers they got from the annotators)
generating a training data set by merging the aggregated second plurality of annotations and the unlabeled data set; and (Introduction describes using the datasets to train machine learning models).
training a machine learning model using the generated training data set. (Introduction describes using the datasets to train machine learning models).
Larlus-Larrondo and Rodrigues are considered analogous art to the claimed invention because they are in the same field of endeavor being crowdsourcing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the crowdsourcing method of Larlus-Larrondo with the evaluation method of Rodrigues. One would want to do this to get more reliable crowdworkers.
Regarding claim 12, Rodrigues teaches:
the at least one confidence metric is the probability of correct classification (PCC) of a pretrained machine learning model. (Section 4. ‘Proposed Model’ “can be interpreted as the probability of an annotator providing a correct label or, in other words, as an indicator of how reliable an annotator is.”)
Regarding claim 13, Rodrigues teaches:
the PCC is calculated for a given crowdworker in the plurality of crowdworkers by providing a recognizer machine learning model with a given plurality of annotations for the plurality of inputs generated by the given crowdworker (Section 4. ‘Proposed Model’ “can be interpreted as the probability of an annotator providing a correct label or, in other words, as an indicator of how reliable an annotator is.” Also in this section they describe the model being a Logistic Regression model)
Regarding claim 14, Rodrigues teaches:
the recognizer machine learning model is a binary logisitic regression classifier. (Section 4. ‘Proposed Model’ describes the logistic regression model that is used on binary variables).
Regarding claim 15, Larlus-Larrondo teaches:
the at least one confidence metric is selected from the group consisting of: a test-retest metric, a reliability metric, a penalized time metric, and a time spent metric. ([0088] “For example, if the percentage of errors on a gold question is .epsilon.=10% and the aim is to declare a sincere worker to be insincere not more than 1% of the time, then m=2 (or more) gold questions per HIT may be chosen and a worker is considered insincere if all gold questions are answered incorrectly.” This describes using many gold questions and testing the workers multiple times.)
Regarding claim 16, Larlus-Larrondo teaches:
the plurality of inputs are provided to a plurality of crowdworkers in response to crowdworkers in the plurality of crowdworkers responding to a request on a crowdsourcing platform. ([0100] “Each HIT is composed of 3 questions. Two of them (standard questions) are based on query images from the test set. A gold question is placed in each HIT (third question) to assess the motivation of workers. The order of the three questions is randomized. To assist the annotator, an image of a bird prelabeled with the class is provided with each candidate answer (except for the answer "none"). The annotator is also given the opportunity to request one more additional photographs for each bird class. The annotator (typically a crowdworker volunteering to perform the task for a small payment on a crowdsourcing marketplace) is asked to click on one of the answers (one of them being the "none" option).”).
Regarding claim 17, Rodrigues teaches:
the plurality of crowdworkers comprise crowdworkers who have completed one or more requests. (Section 5.2 ‘Amazon Mechanical Turk’ “the AMT workers were required to have an HIT approval rate – an AMT quality indicator that reflects the percentage of accepted answers of a worker – of 95%, which ensures some reliability on the quality of the answers.”).
Claims 8-10 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Larlus-Larrondo in view of Rodrigues and Tariq (NPL: ‘Mobile detection of autism through machine learning on home video: A development and prospective validation study’).
Regarding claim 8, neither Larlus-Larrondo nor Rodrigues teach:
the plurality of inputs and the unlabeled data set both comprise anonymized videos of children with Autism Spectrum Disorder (ASD).
Tariq does teach this:
the plurality of inputs and the unlabeled data set both comprise anonymized videos of children with Autism Spectrum Disorder (ASD). (Methods Section: “We assembled 8 published machine learning classifiers to test viability for use in the rapid mobile detection of autism in short home videos.”)
Larlus-Larrondo, Rodrigues and Tariq are considered analogous art to the claimed invention because they are in the same field of endeavor being crowdsourcing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the crowdsourcing method of Larlus-Larrondo with the evaluation method of Rodrigues with the data/videos of Tariq. One would want to do this to develop machine learning models to identify Autism Spectrum Disorder.
Regarding claim 9, Larlus-Larrondo teaches:
the plurality of outputs, the plurality of annotations, and the second plurality of annotations comprise responses to a questionnaire. ([0065] “In other embodiments, questionnaires or other methods could be employed to identify the most popular labels. This may be useful when surrounding information from the webpages can be used. A combination of different approaches to computing a popularity measure may be employed.”).
Regarding claim 10, Tariq teaches:
the machine learning model is trained to identify ASD in videos of children. (Methods Section: “We assembled 8 published machine learning classifiers to test viability for use in the rapid mobile detection of autism in short home videos.”)
Regarding claim 18, neither Larlus-Larrondo nor Rodrigues teach:
the plurality of inputs and the unlabeled data set both comprise anonymized videos of children with Autism Spectrum Disorder (ASD).
Tariq does teach this:
the plurality of inputs and the unlabeled data set both comprise anonymized videos of children with Autism Spectrum Disorder (ASD). (Methods Section: “We assembled 8 published machine learning classifiers to test viability for use in the rapid mobile detection of autism in short home videos.”)
Larlus-Larrondo, Rodrigues and Tariq are considered analogous art to the claimed invention because they are in the same field of endeavor being crowdsourcing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the crowdsourcing method of Larlus-Larrondo with the evaluation method of Rodrigues with the data/videos of Tariq. One would want to do this to develop machine learning models to identify Autism Spectrum Disorder.
Regarding claim 19, Larlus-Larrondo teaches:
the plurality of outputs, the plurality of annotations, and the second plurality of annotations comprise responses to a questionnaire. ([0065] “In other embodiments, questionnaires or other methods could be employed to identify the most popular labels. This may be useful when surrounding information from the webpages can be used. A combination of different approaches to computing a popularity measure may be employed.”).
Regarding claim 20, Tariq teaches:
the machine learning model is trained to identify ASD in videos of children. (Methods Section: “We assembled 8 published machine learning classifiers to test viability for use in the rapid mobile detection of autism in short home videos.”)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL PATRICK GRUSZKA whose telephone number is (571)272-5259. The examiner can normally be reached M-F 9:00 AM - 6:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL GRUSZKA/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121