DETAILED ACTION
This non-final Office action is responsive to the Request for Continued Examination filed February 27th, 2026. Claims 21, 24, and 28-34 have been amended. Claims 35-40 have been cancelled. Claims 21-34 are presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/27/26 has been entered.
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged.
Response to Arguments
Applicant's arguments regarding claim rejections under 35 USC 101 filed 01/26/26 have been fully considered but they are not persuasive.
On pages 8-13 of the provided remarks, Applicant argues that the amended claims present statutory subject matter. Beginning on pages 8-9 of the provided remarks, Applicant argues that claim 21 does not claim “utilizing crowdsourcing systems to classify items according to taxonomy hierarchy” but “executing a machine learning model to infer classification and transmitting a job via a communication network to a crowdsourcing platform”. Examiner asserts that the claimed “causing a first job to be transmitted over a communication network to a crowdsourcing platform” recites Certain Methods of Organizing Human Activity as the transmission of jobs is managing personal behavior. Applicant’s arguments are not persuasive.
Applicant continues on page 9 of the provided remarks, citing the “2024 Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence”, “the allegations of the Office action are untethered to the claim language and seek to expand the group of organizing human activity beyond anything done by the Courts in violation of the 2024 Guidance Update.” Examiner respectfully disagrees and asserts this argument is moot as the argued position by Examiner was applied to now amended claim language. As stated above, addressing the amended claim 21, the claimed “causing a first job to be transmitted over a communication network to a crowdsourcing platform” recites Certain Methods of Organizing Human Activity as the transmission of jobs is managing personal behavior. Citing the amended claim limitation, Examiner asserts that this is tethered directly to the claim language and does not seek to expand the group of organizing human activity. Applicant’s arguments are not persuasive.
Continuing on pages 9-10 of the provided remarks, Applicant cites the remarks of Ex parte Desjardins regarding the handling of 101 subject matter eligibility rejections. However, Examiner asserts that this logic regarding machine learning is moot to the above the argument addressing the claimed crowdsourcing platform. Continuing on pages 10-11 of the provided remarks, Applicant argues that the amended claims “is not simply directed to any manner of generating training data for a machine learning model but instead provides for generating of training data in a specific manner.” Examiner respectfully disagrees and begins by asserting that other than the preamble of the claim, the amended claim limitations do not recite the generation of training data within the claim. The argued “execution of the machine learning model” is recited with a high-level of generality such that this function does not include a specific application of training data at all and the claimed “infer classifications for items associated with the item records” is a mental evaluation of the human mind. Further, the bolded “applying a consensus criterion” and “identifying selected ones of the classifications based on the validation results” are mental observations, judgments, and evaluations of the human mind. The claimed “cause re-training of the machine learning model based on the one or class labels associated with the selected ones of the classifications” is recited so generically (no details whatsoever are provided other than that they are general purpose computing components) that it represents no more than mere instructions to apply the judicial exception on a computer. These limitations can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of a computer. Applicant’s arguments are not persuasive.
Citing paragraphs [0026] and [0032] of the as-filed Specification, Applicant argues that claim 21 as a whole is not directed to an abstract idea. Examiner respectfully disagrees and begins by asserting, per MPEP 2106.05(a) “An indication that the claimed invention provides an improvement can include a discussion in the specification that identifies a technical problem and explains the details of an unconventional technical solution expressed in the claim, or identifies technical improvements realized by the claim over the prior art.” Therefore, it is not just that “the claimed subject matter is grounded in the specification” as argued by Applicant, but that the specification provides “a technical problem and explains the details of an unconventional technical solution expressed in the claim, or identifies technical improvements realized by the claim over the prior art.” Examiner asserts that the identified “consistency and worker quality” of answers is not a technical problem and the argued solution of paragraph [0032] of “aggregating responses received from workers for a particular task” is not a technical solution as the claimed aggregating of responses is recited with a high-level of generality such that the aggregation could be performed as a mental observation, judgment, and evaluation of the human mind. Further, the argued solution of paragraph [0026] “asking workers to evaluate complete or partial classification paths through successive levels in the taxonomic hierarchy” is not a technical solution as the workers are merely performing a method. Therefore, the argued as-filed Specification does not identify a technical problem or explain the details of an unconventional technical solution. Applicant’s arguments are not persuasive.
On page 12 of the provided remarks, Applicant argues “Claim 1 has no mention whatsoever of making observations, judgments, and evaluations and, thus, does not set forth a mental process.” Examiner respectfully disagrees and begins by citing MPEP 2106.04(C) ‘A Claim That Requires a Computer May Still Recite a Mental Process’ for the following, “In evaluating whether a claim that requires a computer recites a mental process, examiners should carefully consider the broadest reasonable interpretation of the claim in light of the specification. For instance, examiners should review the specification to determine if the claimed invention is described as a concept that is performed in the human mind and applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept. In these situations, the claim is considered to recite a mental process.” While Applicant focuses on the claimed, “execution of a machine learning model”, the claim additionally recites, “compare evaluations of the classifications obtained from the crowdsourcing platform to a quality control metric based on the quality control test to identify quality evaluations; aggregate the quality evaluations to generate aggregated evaluations; apply a consensus criterion to the aggregated evaluations to generate validation results for the classifications; identify selected ones of the classifications based on the validation results” which Examiner asserts are concepts performed in the human mind and applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept. Therefore, Applicant’s claims recite a mental process. Applicant’s arguments are not persuasive.
On pages 12-13 of the provided remarks, Applicant argues that the amended claims are analogous to Example 39 of the Subject Matter Eligibility Examples: Abstract Ideas. Specifically, on page 13 of the provided remarks, Applicant argues “Like Example 39, claim 21 of the present application sets forth "apply a consensus criterion to the aggregated evaluations to generate validation results for the classifications; identify selected ones of the classifications based on the validation results; and cause re-training of the machine learning model based on the one or more class labels associated with the selected ones of the classifications." Thus, like Example 39, claim 21 of the present application "does not recite a mental process because the steps are not practically performed in the human mind." See Example 39, Subject Matter Eligibility Examples: Abstract Ideas, p. 9.” Examiner respectfully disagrees and asserts, as stated above, that the argued “apply a consensus criterion to the aggregated evaluations to generate validation results for the classifications; identify selected ones of the classifications based on the validation results;” are concepts performed in the human mind and applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept. Additionally, as stated above, the transmission of jobs within a crowdsourcing platform recites a certain method of organizing human activity in the form of managing personal behavior. Therefore, the amended claims are not analogous to the claims of Example 39 as the limitations recite abstract ideas. The 35 USC 101 rejection is maintained. Applicant’s arguments are not persuasive.
Applicant’s arguments, see pages 13-14, filed 01/26/26, with respect to the rejection(s) of claim(s) 21-40 under 35 USC 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Garera (U.S 2014/0314311 A1) in view of Van Pelt (U.S 2012/0265573 A1).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 21-34 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter;
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter. If the claim does fall within one of the statutory categories, it must then be determined whether the claim is directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), and if so, it must additionally be determined whether the claim is a patent-eligible application of the exception. If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim amounts to significantly more than the abstract idea itself.
Step 1: Independent claims 21 (apparatus), 28 (non-transitory computer-readable medium), and dependent claims 22-27 and 29-34 respectively, fall within at least one of the four statutory categories of 35 U.S.C. 101: (i) process; (ii) machine; (iii) manufacture; or (iv) composition of matter. Claim 21 is directed to an apparatus (i.e. machine) and claim 28 is directed to a non-transitory computer-readable medium (i.e. manufacture).
Step 2A Prong 1: The independent claims recite execute the machine learning model to infer classifications for items associated with the item records, respective ones of the classifications to include one or more class labels in a hierarchical classification taxonomy; cause a first job to be transmitted over a communication network to a crowdsourcing platform, the first job to include (a) a first portion of pre-labeled data corresponding to the classifications and a second portion of unlabeled data, and (b) a randomly selected quality control test; compare evaluations of the classifications obtained from the crowdsourcing platform to a quality control metric based on the quality control test to identify quality evaluations; aggregate the quality evaluations to generate aggregated evaluations; apply a consensus criterion to the aggregated evaluations to generate validation results for the classifications; identify selected ones of the classifications based on the validation results; and cause re-training of the machine learning model based on the one or more class labels associated with the selected ones of the classifications (Certain Methods of Organizing Human Activity & Mental Process), which are considered to be abstract ideas (See PEG 2019 and MPEP 2106.05). [Examiner notes the underlined limitations above recite the abstract idea].
The steps/functions disclosed above and in the independent claims recite the abstract idea of Certain Methods of Organizing Human Activity because the claimed limitations are transmitting jobs to a crowdsourcing platform, which is managing personal behavior. The Applicant’s claimed limitations are transmitting jobs to a crowdsourcing platform, which recite the abstract idea of Certain Methods of Organizing Human Activity.
The steps/functions disclosed above and in the independent claims recite the abstract idea of Mental Process because the claimed limitations are inferring classifications for items associated with item records; comparing evaluations of the classifications obtained from the crowdsourcing platform to a quality control metric based on the quality control test to identify quality evaluations; aggregating quality evaluation decisions to generate aggregated evaluations; applying a consensus criterion to the aggregated evaluations to generate validation results for the classifications; and identifying selected ones of the classifications based on the validation results, which are observations, judgments, and evaluations of the human mind.
In addition, dependent claims 22-27 and 29-34 further narrow the abstract idea and recite further defining generating a report corresponding to the machine learning model including the accuracy information of the model; the validation results; relabeling the classifications; the selection of classifications; and the consensus criterion algorithm. These processes are similar to the abstract idea noted in the independent claims because they further the limitations of the independent claims which recite certain methods of organizing human activity & mental processes. Accordingly, these claim elements do not serve to confer subject matter eligibility to the claims since they recite abstract ideas.
Step 2A Prong 2: In this application, the above “obtain item records (claim 21); cause a first job to be transmitted over a communication network to a crowdsourcing platform” steps/functions of the independent claims would not account for additional elements that integrate the judicial exception (e.g. abstract idea) into a practical application because receiving/storing data and displaying data merely add insignificant extra-solution activity and merely adds the words to apply it with the judicial exception. Also, the claimed “An apparatus comprising: interface circuitry; machine-readable instructions; and at least one processor circuit to be programmed by the machine-readable instructions; a communication network to a crowdsourcing platform; At least one non-transitory machine-readable medium comprising machine-readable instructions to cause at least one processor circuit to; at least one processor circuit programmed by at least one instruction” would not account for additional elements that integrate the judicial exception (e.g. abstract idea) into a practical application because the claimed structure merely adds the words to apply it with the judicial exception and mere instructions to implement an abstract idea on a computer (See PEG 2019 and MPEP 2106.05).
Independent claims 21 and 28 recite the following limitation, “generate training data for a machine learning model”, “execute a machine learning model”, and “cause re-training of the machine learning model based on the one or more class labels associated with the selected ones of the classifications”. The “training”, “re-training”, and “machine learning model” is recited so generically (no details whatsoever are provided other than that they are general purpose computing components) that it represents no more than mere instructions to apply the judicial exception on a computer. These limitations can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of a computer. These limitations would not account for additional elements that integrate the judicial exception (e.g. abstract idea) into a practical application because the claimed structure merely adds the words to apply it with the judicial exception and mere instructions to implement an abstract idea on a computer (See PEG 2019 and MPEP 2106.05).
In addition, dependent claims 22-27 and 29-34 further narrow the abstract idea and dependent claims 24 and 31 additionally recite “transmit a first one of the classifications associated with the first one of the validation results to at least one domain expert for relabeling, the first ones of the classifications different than the selected ones of the classifications” which do not account for additional elements that integrate the judicial exception (e.g. abstract idea) into a practical application because receiving/storing data and displaying data merely add insignificant extra-solution activity and the claimed “at least one processor circuit” which do not account for additional elements that integrate the judicial exception (e.g. abstract idea) into a practical application because the claimed structure merely adds the words to apply it with the judicial exception and mere instructions to implement an abstract idea on a computer (See PEG 2019 and MPEP 2106.05).
The claimed “An apparatus comprising: interface circuitry; machine-readable instructions; and at least one processor circuit to be programmed by the machine-readable instructions; a communication network to a crowdsourcing platform; At least one non-transitory machine-readable medium comprising machine-readable instructions to cause at least one processor circuit to; at least one processor circuit programmed by at least one instruction” are recited so generically (no details whatsoever are provided other than that they are general purpose computing components and regular office supplies) that they represent no more than mere instructions to apply the judicial exception on a computer. These limitations can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of a computer. Even when viewed in combination, the additional elements in the claims do no more than use the computer components as a tool. There is no change to the computers and other technology that is recited in the claim, and thus the claims do not improve computer functionality or other technology (See PEG 2019).
Step 2B: When analyzing the additional element(s) and/or combination of elements in the claim(s) other than the abstract idea per se the claim limitations amount(s) to no more than: a general link of the use of an abstract idea to a particular technological environment and merely amounts to the application or instructions to apply the abstract idea on a computer (See MPEP 2106.05 and PEG 2019). Further, system claims 21-27; and non-transitory computer-readable medium claims 28-34 recite “An apparatus comprising: interface circuitry; machine-readable instructions; and at least one processor circuit to be programmed by the machine-readable instructions; a communication network to a crowdsourcing platform; At least one non-transitory machine-readable medium comprising machine-readable instructions to cause at least one processor circuit to; at least one processor circuit programmed by at least one instruction”; however, these elements merely facilitate the claimed functions at a high level of generality and they perform conventional functions and are considered to be general purpose computer components which is supported by Applicant’s specification in Paragraphs 0055 and 0057 and Figures 2 & 6. The Applicant’s claimed additional elements are mere instructions to implement the abstract idea on a general purpose computer and generally link of the use of an abstract idea to a particular technological environment. Also, the above “obtain item records (claim 21); cause a first job to be transmitted over a communication network to a crowdsourcing platform” steps/functions of the independent claims would not account for significantly more than the abstract idea because receiving data and displaying/presenting data (See MPEP 2106.05) have been identified as well-known, routine, and conventional steps/functions to one of ordinary skill in the art. When viewed as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself.
Next, when the “machine learning” of claims 21 and 28 is evaluated as an additional element, this feature is recited at a high level of generality and encompasses well-understood, routine, and conventional prior art activity. See, e.g., Balsiger et al., US 2012/0054642, noting in paragraph [0077] that “Machine learning is well known to those skilled in the art.” See also, Djordjevic et al. US 2013/0018651, noting in paragraph [0019] that “As known in the art, a generative model can be used in machine learning to model observed data directly.” See also, Bauer et al., US 2017/0147941, noting at paragraph [0002] that “Problems of understanding the behavior or decisions made by machine learning models have been recognized in the conventional art and various techniques have been developed to provide solutions.” Accordingly, the use of machine learning to infer classifications does not add significantly more to the claim.
In addition, claims 22-27 and 29-34 further narrow the abstract idea identified in the independent claims. The Examiner notes that the dependent claims merely further define the data being analyzed and how the data is being analyzed. Similarly, claims 24 and 31 additionally recite “transmit a first one of the classifications associated with the first one of the validation results to at least one domain expert for relabeling, the first ones of the classifications different than the selected ones of the classifications” which do not account for additional elements that amount to significantly more than the abstract idea because receiving data and displaying/presenting data (See MPEP 2106.05) have been identified as well-known, routine, and conventional steps/functions to one of ordinary skill in the art and the claimed “at least one processor” which do not account for additional elements that amount to significantly more than the abstract idea because the claimed structure merely amounts to the application or instructions to apply the abstract idea on a computer and does not move beyond a general link of the use of an abstract idea to a particular technological environment (See MPEP 2106.05). The additional limitations of the independent and dependent claim(s) when considered individually and as an ordered combination do not amount to significantly more than the abstract idea. The examiner has considered the dependent claims in a full analysis including the additional limitations individually and in combination as analyzed in the independent claim(s). Therefore, the claim(s) are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 21-34 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garera (U.S 2014/0314311 A1) in view of Van Pelt (U.S 2012/0265573 A1).
Claims 21 and 28
Regarding Claim 21, Garera discloses the following:
An apparatus to generate training data for a machine learning model, the apparatus comprising [see at least Paragraph 0024 for reference to the system including one or more server systems that may each be embodied as one or more server computers each including one or more processors that are in data communication with one another; Paragraph 0028 for reference to the computing device being used to perform various procedures; Figure 1 and related text regarding the system for performing methods; Figure 2 and related text regarding the computing device suitable for implementing embodiments; Figures 4 & 5 and related text regarding a method for training a classification model]
interface circuitry to obtain item records [see at least Paragraph 0034 for reference to interface(s) including various interfaces that allow computing device to interact with other systems, devices, or computing environments; Paragraph 0056 for reference to the method receiving an initial training set; Figure 2 and related text regarding item 206 ‘Interface(s)’; Figure 3 and related text regarding item 306 ‘Product Records’; Figure 4 and related text regarding item 402 ‘Receive Initial Training Set’]
machine-readable instructions [see at least Paragraph 0017 for reference to the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium; Paragraph 0018 for reference to a computer-readable medium may comprise any non-transitory medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device; Paragraph 0021 for reference to computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner]
at least one processor circuit to be programmed by the machine-readable instructions to [see at least Paragraph 0020 for reference to computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks; Paragraph 0029 for reference to one or more processors or controllers that execute instructions; Figure 2 and related text regarding item 202 ‘Processor’]
execute the machine learning model to infer classifications for items associated with the item records, ones of the classifications to include one or more class labels in a hierarchical classification taxonomy [see at least Paragraph 0048 for reference to a category of a taxonomy having one or more classification values as descendants thereof may be selected by the analyst module for the generation of training data; Paragraph 0056 for reference to the classifier model being trained using the any machine learning algorithm to produce classification output; Paragraph 0056 for reference to some or all records in the records corpus being classified using the classifier model; Paragraph 0068 for reference to classifying a selected portion of a product records using the classifier model; Figure 4 and related text regarding the training of a classification model including item 406 ‘Classify Product Records Using Classifier’; Figure 5 and related text regarding item 504 ‘Classify Selected Product Records’]
cause a first job to be transmitted over a communication network to a crowdsourcing platform, the first job to include (a) a first portion of pre-labeled data corresponding to the classifications and a second portion of unlabeled data [see at least Paragraph 0040 for reference to the classifications sent to the crowdsourcing forum including all or less than all classifications in a given iteration method; Paragraph 0042 for reference to the classifications being submitted to a crowdsourcing forum; Paragraph 0058 for reference to classification not identified at a high confidence being submitted to a crowdsourcing for validation; Paragraph 0058 for reference to distributing the some or all of the classifications to participants in a crowdsourcing forum, such as by transmitting classifications to crowdsourcing workstations for display; Figure 3 and related text regarding item 312 ‘Crowdsourcing Forum’; Figure 4 and related text regarding item 410 ‘Submit Remaining Product Records and Classifications to Crowdsourcing’]
compare evaluations of the classifications obtained from the crowdsourcing platform to identify quality evaluations [see at least Paragraph 0039 for reference to using the confidence score classifications output by the classifier may be divided into high confidence classifications and other classifications; Paragraph 0040 for reference to the classifications 310 and classifications 308 may include less than all classifications in a given iteration of the methods disclosed herein, such that only data with a confidence score above a first threshold are deemed high confidence classifications 308 and only classifications with a confidence below a second threshold are deemed classifications 310, where the second threshold is below the first threshold and a nonzero quantity of classifications have confidence scores between the first and second thresholds]
aggregate the quality evaluations to generate aggregated evaluations [see at least Paragraph 0039 for reference to using the confidence score classifications output by the classifier may be divided into high confidence classifications and other classifications; Paragraph 0042 for reference to the crowdsourcing forum may implement logic for distributing tasks to individuals associated with the forum, receiving responses, and returning responses to a requesting entity; Paragraph 0060 for reference to transmitting some or all of the validations decisions to one or more analyst work stations; Figure 3 and related text regarding item 310 ‘Classifications’; Figure 4 and related text regarding item 412 ‘Receive Yes/No/Can’t Tell for Product Records from Crowdsourcing’]
apply a consensus criterion to the aggregated evaluations to generate validation results for the classifications [see at least Paragraph 0039 for reference to decisions made by the algorithm, e.g. a classification of text, may be assigned a confidence score indicating how much support exists for the decision; Paragraph 0043 for reference to the crowdsourcing forum returning a validation decision; Paragraph 0059 for reference to a validation decision may be received from the crowdsourcing forum; Paragraph 0060 for reference to validation decisions may then be displayed on the analyst workstation, possibly with a prompt to approve or disapprove of the validation decision and/or provide an alternative classification value; Figure 3 and related text regarding item 314 ‘Validation Decisions’; Figure 7 and related text regarding item 702 ‘Sample Classification Validation Decisions’]
identify selected ones of the classifications based on the validation results [see at least Paragraph 0059 for reference to a validation decision may be received from the crowdsourcing forum; Paragraph 0060 for reference to validation decisions may then be displayed on the analyst workstation, possibly with a prompt to approve or disapprove of the validation decision and/or provide an alternative classification value; Paragraph 0062 for reference to classifications validated by the crowdsourcing forum may be added to the training set; Figure 3 and related text regarding item 314 ‘Validation Decisions’; Figure 7 and related text regarding item 702 ‘Sample Classification Validation Decisions’]
cause re-training of the machine learning model based on the one or more class labels associated with the selected ones of the classifications [see at least Paragraph 0060 for reference to some or all of the validation decisions received from the crowdsourcing forum may be submitted to one or more analysts to verify that validation decisions are accurate; Paragraph 0075 for reference to the accuracy model may output in response to an arbitrary confidence score, a probability that a classification with that confidence score is accurate; Paragraph 0076 for reference to the accuracy model determining a percentage of these classifications that have been validated by a validation decision; Paragraph 0080 for reference to the accuracy model being periodically retrained; Figure 6 and related text regarding item 602 ‘accuracy model’; Figure 7 and related text regarding the process for identifying high accuracy data]
While Garera discloses the limitations above, it does not disclose the first job to include (b) a randomly selected quality control test; compare evaluations of the classifications obtained from the crowdsourcing platform to a quality control metric based on the quality control test to identify quality evaluations.
However, Van Pelt discloses the following:
cause a first job to be transmitted over a communication network to a crowdsourcing platform, the first job to include the first job to include (b) a randomly selected quality control test [see at least Paragraph 0031 for reference to evaluate a workers work quality, reliability in performing crowd sourced tasks via their own user devices 102A-N by intelligently generating evaluation or test tasks with known right answers or known false answers to be presented to a worker or potential worker (e.g., new worker being initially assessed for competency and usability) to see if the worker provides the correct response; Paragraph 0031 for reference to evaluation or test tasks can be generated (e.g., automatically or semi-automatically) from jobs actually submitted to and distributed through the job distribution platform of the host server; Paragraph 0073 for reference to test data/test standards manager 400 can be any combination of software agents and/or hardware modules (e.g., including processors and/or memory units) able to generate test data, test jobs, test or evaluation tasks to assess and evaluate worker competency, ability, and/or usability; Figure 4 and related text regarding components of a test data/test standards manager for use in evaluating worker performing and for use in providing in-task training]
compare evaluations of the classifications obtained from the crowdsourcing platform to a quality control metric based on the quality control test to identify quality evaluations [see at least Paragraph 0096 for reference to using test jobs/tasks with known responses/answers, the quality/accuracy of a worker's work product can directly be determined and used in computing a quantitative metric indicating the accuracy of a worker's work product; Paragraph 0096 for reference to quality assessment using test jobs/tasks can be performed by the gold standard metric module 242. In addition, the quality assessment can be performed by the peer review metric module 241, where a worker's work product can be evaluated using results generated by other workers (e.g., determine whether they match or not)]
Before the effective filing date, it would have been obvious to one of ordinary skill in the art to modify the crowdsourcing classification apparatus of Garera to include the quality testing of crowdsourcing workers of Van Pelt. Doing so, no additional resources need to be utilized for creation and management of test tasks, as stated by Van Pelt (Paragraph 0031).
Regarding claim 28, the claim recites limitations already addressed by the rejection of claim 21. Regarding claim 28, Garera teaches at least one non-transitory computer readable medium comprising machine-readable instructions to cause at least one processor circuit to [Paragraph 0018]. Therefore, claim 28 is rejected as being unpatentable in view of Garera and Van Pelt.
Claims 22 and 29
While the combination of Garera and Van Pelt disclose the limitations above, regarding Claim 22, Garera discloses the following:
wherein one or more of the at least one processor circuit is to generate a report corresponding to the machine learning model [see at least Paragraph 0060 for reference to some or all of the validation decisions received from the crowdsourcing forum may be submitted to one or more analysts to verify that validation decisions are accurate; Paragraph 0075 for reference to the accuracy model may output in response to an arbitrary confidence score, a probability that a classification with that confidence score is accurate; Paragraph 0076 for reference to the accuracy model determining a percentage of these classifications that have been validated by a validation decision; Figure 6 and related text regarding item 602 ‘accuracy model’; Figure 7 and related text regarding the process for identifying high accuracy data]
the report to include accuracy information includes current and historical accuracies associated with the machine learning model [see at least Paragraph 0075 for reference to accuracy model may receive with each individual validation decision received such information as the classification corresponding to the validation decision as well as the confidence score assigned to the classification by the classifier; Paragraph 0080 for reference to the accuracy model being trained continuously as validation decisions are generated and the confidence threshold for a predetermined accuracy; Paragraph 0081 for reference to the accuracy model being trained using all validation decisions, or a random sampling of validation decisions, received in a time interval, e.g. since the last time the accuracy model was trained]
Regarding claim 29, the claim recites limitations already addressed by the rejection of claim 22.
Claims 23 and 30
While the combination of Garera and Van Pelt disclose the limitations above, regarding Claim 23, Garera discloses the following:
wherein one or more of the at least one processor circuit is to generate a first one of the validation results as one of a valid classification, an invalid classification, or an uncertain classification [see at least Paragraph 0043 for reference to crowdsourcing forum may return a yes or no response indicating that the classifications was or was not correct; Paragraph 0043 for reference to the crowdsourcing forum may return an “unclear response indicating that the text in a text-classification value output of the classifier is insufficient to accurately judge whether the classification is correct and/or what an accurate classification should be; Figure 3 and related text regarding item 314 ‘Validation Decisions’, item 316a ‘Unclear’, item 316b ‘No’, and item 316c ‘Yes’; Figure 4 and related text regarding item 412 ‘Receive Yes/No/Can’t Tell for Product Records from Crowdsourcing’]
Regarding claim 30, the claim recites limitations already addressed by the rejection of claim 23.
Claims 24 and 31
While the combination of Garera and Van Pelt disclose the limitations above, regarding Claim 24, Garera discloses the following:
wherein, when the first one of the validation results is the invalid classification or the uncertain classification, one or more of the at least one processor circuit is to transmit a first one of the classifications associated with the first one of the validation results to at least one domain expert for relabeling, the first ones of the classifications different than the selected ones of the classifications [see at least Paragraph 0044 for reference to unclear and invalid classifications being further processed; Paragraph 0045 for reference to being submitted to analysts be evaluated for the correctness of the validation decision; Paragraph 0048 for reference to the analyst module may be programmed to select classification values, i.e. values for the classification value fields of the classifications for which additional training data is needed; Paragraph 0049 for reference to the analyst module selecting classification values or categories of classification values on the basis on a percentage of classifications 310 referencing that classification value or category of classification values that were marked as invalid, or either invalid or unclear, by the crowd sourcing forum; Figure 3 and related text regarding item 322 ‘Analyst Module’; Figure 4 and related text regarding item 414 ‘Submit Crowdsourcing Decisions to Analysts for Spotchecking’ and item 416 ‘Identifying Problem Categories (high No. of No/Can’t Tell)’]
Regarding claim 31, the claim recites limitations already addressed by the rejection of claim 24.
Claims 25 and 32
While the combination of Garera and Van Pelt disclose the limitations above, regarding Claim 25, Garera discloses the following:
wherein, one or more of the at least one processor circuit is to identify the first one of the classifications as rejected data based on a failure of the domain expert to relabel the first one of the classifications [see at least Paragraph 0046 for reference to validation decisions found to be incorrect by the analyst may be transmitted to the crowdsourcing forum; Paragraph 0049 for reference to the analyst module selecting classification values or categories of classification values on the basis on a percentage of classifications 310 referencing that classification value or category of classification values that were marked as invalid, or either invalid or unclear, by the crowd sourcing forum; Figure 3 and related text regarding item 320 ‘Feedback’; Figure 4 and related text regarding item 416 ‘Identifying Problem Categories (high No. of No/Can’t Tell)’ and item 418 ‘Request Training Data for Problem Categories from Analysts’]
Regarding claim 32, the claim recites limitations already addressed by the rejection of claim 25.
Claims 26 and 33
While the combination of Garera and Van Pelt disclose the limitations above, regarding Claim 26, Garera discloses the following:
wherein the first job is a filter job [see at least Paragraph 0058 for reference to classification not identified at a high confidence being submitted to a crowdsourcing for validation; Paragraph 0058 for reference to distributing the some or all of the classifications to participants in a crowdsourcing forum, such as by transmitting classifications to crowdsourcing workstations for display; Figure 3 and related text regarding item 312 ‘Crowdsourcing Forum’; Figure 4 and related text regarding item 410 ‘Submit Remaining Product Records and Classifications to Crowdsourcing’]
the classifications are a first set of classifications, one or more of the at least one processor circuit is to select the first set of classifications based on a sampling strategy and a sample count [see at least Paragraph 0039 for reference to using the confidence score classifications output by the classifier may be divided into high confidence classifications and other classifications; Paragraph 0039 for reference to the number of classifications selected for processing may be chosen in accordance with this capacity; Paragraph 0048 for reference to classification values may be selected individually or as a group; Paragraph 0048 for reference to a category of a taxonomy having one or more classification values as descendants thereof may be selected by the analyst module for the generation of training data; Figure 3 and related text regarding item 310 ‘Classifications’; Figure 4 and related text regarding item 412 ‘Receive Yes/No/Can’t Tell for Product Records from Crowdsourcing’]
Regarding claim 33, the claim recites limitations already addressed by the rejection of claim 26.
Claims 27 and 34
While the combination of Garera and Van Pelt disclose the limitations above, regarding Claim 27, Garera discloses the following:
wherein the consensus criterion include a custom aggregation algorithm having a sequence of consensus rules [see at least Paragraph 0038 for reference to machine learning algorithm used to implement the classifier ranging from various types of algorithms; Paragraph 0039 for reference to decisions made by the algorithm, e.g. a classification of text, may be assigned a confidence score indicating how much support exists for the decision; Paragraph 0039 for reference to using the confidence score classifications output by the classifier may be divided into high confidence classifications and other classifications; Paragraph 0039 for reference to the number of classifications selected for processing may be chosen in accordance with this capacity]
Regarding claim 34, the claim recites limitations already addressed by the rejection of claim 27.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lease, Matthew. "On quality control and machine learning in crowdsourcing." Human Computation 11.11 (2011): 1085.
DOCUMENT ID
INVENTOR(S)
TITLE
US 2015/0363741 A1
Chandra et al.
TASK ASSIGNMENT IN CROWN SOURCING
US 11,232,380 B2
Mathiesen et al.
Mapping assessment results to levels of experience
US 11,803,883 B2
Wu et al.
Quality Assurance For Labeled Training Data
US 8,554,605 B2
Oleson et al.
Evaluating A Worker In Performing Crowd Sourced Tasks And Providing In-task Training Through Programmatically Generated Test Tasks
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KRISTIN ELIZABETH GAVIN whose telephone number is (571)270-7019. The examiner can normally be reached M-F 7:30-4:30 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jerry O'Connor can be reached at 571-272-6787. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KRISTIN E GAVIN/Primary Examiner, Art Unit 3624
Prosecution Insights