DETAILED ACTION
This final Office action is responsive to amendments filed July 21st, 2026. Claims 21, 23-28, 30-32, and 34 have been amended. Claims 41-43 have been added. Claims 21-34 and 41-43 are presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged.
Response to Arguments
Applicants’ arguments regarding claim rejections under 35 USC 101 filed 07/21/26 have been fully considered but they are not persuasive.
On pages 10-14 of the provided remarks, Applicant argues that the amended claims present statutory subject matter. Beginning on page 10 of the provided remarks, Applicant argues that the claims are not directed to the abstract ideas of Certain Methods of Organizing Human Activity or Mental Process. As Applicant has not provided statements supporting the argument that the amended claims do not recite abstract ideas, Examiner reasserts the previous response that the claimed “causing a first job to be transmitted over a communication network to a crowdsourcing platform” recites Certain Methods of Organizing Human Activity as the transmission of jobs is managing personal behavior. Applicants’ arguments are not persuasive.
Continuing on pages 10-11 of the provided remarks, Applicant argues regarding Examiner’s previous response to the argued Ex Parte Dejardins remarks, “The Examiner has not authority to ignore Director Squires or import limitations on his directives”. Examiner asserts that Applicant has misinterpreted the previous response. No statement was made ignoring Director Squires directives, however, the previous argument stated that the logic applied by Director Squires “is moot to the above the argument addressing the claimed crowdsourcing platform.” Stating that the argued logic is not applicable does not equate to ignoring directives. Applicants’ arguments are not persuasive.
Continuing on page 11 of the provided remarks, Applicant argues “Director Squires did not state that his guidance in Desjardins only applies to the specific manner in which the claims at hand in Desjardins improved “training of machine learning models.” While Applicant has provided selective citations of Enfish and AI Visualize to support their argument, citing the argued memo in question, “On September 26, 2025, the United States Patent and Trademark Office (USPTO) issued an Appeals Review Panel decision in Ex Parte Desjardins, Appeal No. 2024-000567 (PTAB September 26, 2025, Appeals Review Panel Decision) vacating the Board’s new ground of rejection under 35 U.S.C. § 101. The decision was designated precedential on November 4, 2025. This advance notice revises the Ninth Edition, Revision 01.2024, November 2024 publication of the Manual of Patent Examining Procedure (MPEP) to include Ex Parte Desjardins, as indicated below.” The provided revisions to the MPEP 2106.04(d)(1) and 2106.05(a) list provide example analysis with respect to evaluating machine learning claim limitations. Applicants’ arguments are not persuasive.
Continuing on pages 11-14 of the provided remarks, Applicant argues that the amended claim limitations implement a technical improvement to the quality of machine learning training data that is based on data received from a crowdsourcing platform. Examiner respectfully disagrees and begins by asserting that cited paragraph [0032] of the as-filed Specification does not recite the argued technical problem of Applicant as machine learning training data quality. As previously stated, the noted problem within paragraph [0032] of the “answers returned by crowdsourcing workers not being consistent and worker quality varying” is directly related to the abstract idea of Certain Methods of Organizing Human Activity as the problem relates to the production and output of human workers. Further, the argued aggregation component present in the amended claims and supported by paragraph [0032] merely further recites the abstract idea of Mathematical Concepts as the now claimed majority voting algorithm, weighted majority algorithm, or Bayesian voting algorithm are mathematical calculations. Therefore, the argued claimed technical improvement to a technical problem are not technical in nature. Applicants’ arguments are not persuasive.
Finally, on page 14 of the provided remarks, Applicant returns to Director Squire’s guidance to support the argument that the amended claims are similar to that of argued Enfish. Per the argued memo in question, the following is stated, “Examiners are expected to consider existing precedent like Enfish, as discussed in MPEP § 2106, in addition to these updates when assessing eligibility under 35 U.S.C. § 101, particularly when evaluating claims related to machine learning or artificial intelligence.” The cited portion of Enfish within Ex Parte Desjardins recites “In particular, Enfish recognized that “[m]uch of the advancement made in computer technology consists of improvements to software that, by their very nature, may not be defined by particular physical features but rather by logical structures and processes.” 822 F.3d at 1339.” The argued logic was utilized by Director Squires to determine, per Desjardins, “We are persuaded that constitutes an improvement to how the machine learning model itself operates, and not, for example, the identified mathematical calculation.” Unlike the applied logic above, the argued method of training a machine learning model does not improve how the model itself operates, but per cited paragraph [0032] improves the quality of answers returned by human workers. The 35 USC 101 rejection is maintained. Applicants’ arguments are not persuasive.
Applicant’s arguments, see pages 15-16, filed 07/21/26, with respect to the rejection(s) of claim(s) 21-34 under 35 USC 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Garera (U.S 2014/0314311 A1) in view of Kirshenbaum (U.S 2003/0191726 A1).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 21-34 and 41-43 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter;
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter. If the claim does fall within one of the statutory categories, it must then be determined whether the claim is directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), and if so, it must additionally be determined whether the claim is a patent-eligible application of the exception. If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim amounts to significantly more than the abstract idea itself.
Step 1: Independent claims 21 (apparatus), 28 (non-transitory computer-readable medium), and dependent claims 22-27, 29-34, and 41-43 respectively, fall within at least one of the four statutory categories of 35 U.S.C. 101: (i) process; (ii) machine; (iii) manufacture; or (iv) composition of matter. Claim 21 is directed to an apparatus (i.e. machine) and claim 28 is directed to a non-transitory computer-readable medium (i.e. manufacture).
Step 2A Prong 1: The independent claims recite execute the machine learning model to infer classifications for items associated with the item records, respective ones of the classifications to include one or more class labels in a hierarchical classification taxonomy; cause a first job to be transmitted over a communication network to a crowdsourcing platform, the first job to include (a) a first portion of pre-labeled data corresponding to first ones of the classifications and (b) a second portion of unlabeled data, the second portion of unlabeled data associated with second ones of the classifications, the second ones of the classifications not included in the first job; aggregate evaluations from the crowdsourcing platform by executing one or more of a majority voting algorithm, a weighted majority algorithm, or a Bayesian voting algorithm to generate aggregated evaluations, the evaluations including first evaluations for the first portion of the pre-labeled data and second evaluations for the second portion of the unlabeled data; apply a consensus criterion to the aggregated evaluations to generate validation results for the classifications, the consensus criterion defining a level of consensus among two or more of the evaluations for identifying respective ones of the classifications as valid classifications; identify first ones of the classifications as training data based on the validation results, the first ones of the classifications corresponding to the valid classifications; identify second ones of the classifications as invalid classifications based on the validation results, the second ones of the classifications different than the first ones of the classifications; cause the one or more of the class labels associated with the first ones of the classifications but not the one or more class labels associated with the second ones of the classifications to be transmitted over the communication network for storage in a training data database; and cause re-training of the machine learning model based on the one or more class labels associated with the first ones of the classifications retrieved from the training data database via the communication network (Certain Methods of Organizing Human Activity & Mental Process & Mathematical Concept), which are considered to be abstract ideas (See PEG 2019 and MPEP 2106.05). [Examiner notes the underlined limitations above recite the abstract idea].
The steps/functions disclosed above and in the independent claims recite the abstract idea of Certain Methods of Organizing Human Activity because the claimed limitations are transmitting jobs to a crowdsourcing platform, which is managing personal behavior. The Applicant’s claimed limitations are transmitting jobs to a crowdsourcing platform, which recite the abstract idea of Certain Methods of Organizing Human Activity.
The steps/functions disclosed above and in the independent claims recite the abstract idea of Mental Process because the claimed limitations are inferring classifications for items associated with item records; aggregating quality evaluation decisions to generate aggregated evaluations; applying a consensus criterion to the aggregated evaluations to generate validation results for the classifications; identifying first ones of the classifications as training data based on the validation results; identifying second ones of the classifications as invalid classifications based on the validation results; and causing one or more labels associated with the first ones of the classifications to be transmitted, which are observations, judgments, and evaluations of the human mind.
The steps/functions disclosed above and in the independent claims recite the abstract idea of Mathematical Calculations because the claimed limitations are executing a majoring voting algorithm, a weighted majority algorithm, or a Bayesian voting algorithm to generate aggregate evaluations, which are mathematical calculations are determined by paragraph [0032] of the as-filed Specification. The Applicant’s claimed limitations are executing a majoring voting algorithm, a weighted majority algorithm, or a Bayesian voting algorithm to generate aggregate evaluations, which recite the abstract idea of Mathematical Concepts.
In addition, dependent claims 22-27, 29-34, and 41-43 further narrow the abstract idea and recite further defining generating a report corresponding to the machine learning model including the accuracy information of the model; identifying third classifications; relabeling the classifications; the selection of classifications; the consensus criterion algorithm; a randomly selected quality control test; comparing evaluations of the classifications obtains from the crowdsourcing platform to a quality control metric to identify quality evaluations; and the formatting of evaluations. These processes are similar to the abstract idea noted in the independent claims because they further the limitations of the independent claims which recite certain methods of organizing human activity & mental processes. Accordingly, these claim elements do not serve to confer subject matter eligibility to the claims since they recite abstract ideas.
Step 2A Prong 2: In this application, the above “obtain item records (claim 21); cause a first job to be transmitted over a communication network to a crowdsourcing platform; causing the one or more of the class labels associated with the first ones of the classifications but not the one or more class labels associated with the second ones of the classifications to be transmitted over the communication network for storage in a training data database” steps/functions of the independent claims would not account for additional elements that integrate the judicial exception (e.g. abstract idea) into a practical application because receiving/storing data and displaying data merely add insignificant extra-solution activity and merely adds the words to apply it with the judicial exception. Also, the claimed “An apparatus comprising: interface circuitry; machine-readable instructions; and at least one processor circuit to be programmed by the machine-readable instructions; a communication network to a crowdsourcing platform; a training data database; At least one non-transitory machine-readable medium comprising machine-readable instructions to cause at least one processor circuit to; at least one processor circuit programmed by at least one instruction” would not account for additional elements that integrate the judicial exception (e.g. abstract idea) into a practical application because the claimed structure merely adds the words to apply it with the judicial exception and mere instructions to implement an abstract idea on a computer (See PEG 2019 and MPEP 2106.05).
Independent claims 21 and 28 recite the following limitation, “generate training data for a machine learning model”, “execute a machine learning model”, and “cause re-training of the machine learning model based on the one or more class labels associated with the first ones of the classifications retrieved from the training data database via the communication network”. The “training”, “re-training”, and “machine learning model” is recited so generically (no details whatsoever are provided other than that they are general purpose computing components) that it represents no more than mere instructions to apply the judicial exception on a computer. These limitations can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of a computer. These limitations would not account for additional elements that integrate the judicial exception (e.g. abstract idea) into a practical application because the claimed structure merely adds the words to apply it with the judicial exception and mere instructions to implement an abstract idea on a computer (See PEG 2019 and MPEP 2106.05).
In addition, dependent claims 22-27, 29-34, and 41-43 further narrow the abstract idea and dependent claims 24 and 31 additionally recite “transmit the second ones and the third ones of the classifications via the communication network to at least one domain expert for relabeling” which do not account for additional elements that integrate the judicial exception (e.g. abstract idea) into a practical application because receiving/storing data and displaying data merely add insignificant extra-solution activity and the claimed “at least one processor circuit” and “via the communication network” which do not account for additional elements that integrate the judicial exception (e.g. abstract idea) into a practical application because the claimed structure merely adds the words to apply it with the judicial exception and mere instructions to implement an abstract idea on a computer (See PEG 2019 and MPEP 2106.05).
The claimed “An apparatus comprising: interface circuitry; machine-readable instructions; and at least one processor circuit to be programmed by the machine-readable instructions; a communication network to a crowdsourcing platform; a training data database; At least one non-transitory machine-readable medium comprising machine-readable instructions to cause at least one processor circuit to; at least one processor circuit programmed by at least one instruction” are recited so generically (no details whatsoever are provided other than that they are general purpose computing components and regular office supplies) that they represent no more than mere instructions to apply the judicial exception on a computer. These limitations can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of a computer. Even when viewed in combination, the additional elements in the claims do no more than use the computer components as a tool. There is no change to the computers and other technology that is recited in the claim, and thus the claims do not improve computer functionality or other technology (See PEG 2019).
Step 2B: When analyzing the additional element(s) and/or combination of elements in the claim(s) other than the abstract idea per se the claim limitations amount(s) to no more than: a general link of the use of an abstract idea to a particular technological environment and merely amounts to the application or instructions to apply the abstract idea on a computer (See MPEP 2106.05 and PEG 2019). Further, system claims 21-27 and 41-42; and non-transitory computer-readable medium claims 28-34 and 43 recite “An apparatus comprising: interface circuitry; machine-readable instructions; and at least one processor circuit to be programmed by the machine-readable instructions; a communication network to a crowdsourcing platform; a training data database; At least one non-transitory machine-readable medium comprising machine-readable instructions to cause at least one processor circuit to; at least one processor circuit programmed by at least one instruction”; however, these elements merely facilitate the claimed functions at a high level of generality and they perform conventional functions and are considered to be general purpose computer components which is supported by Applicant’s specification in Paragraphs 0055 and 0057 and Figures 2 & 6. The Applicant’s claimed additional elements are mere instructions to implement the abstract idea on a general purpose computer and generally link of the use of an abstract idea to a particular technological environment. Also, the above “obtain item records (claim 21); cause a first job to be transmitted over a communication network to a crowdsourcing platform; causing the one or more of the class labels associated with the first ones of the classifications but not the one or more class labels associated with the second ones of the classifications to be transmitted over the communication network for storage in a training data database” steps/functions of the independent claims would not account for significantly more than the abstract idea because receiving data and displaying/presenting data (See MPEP 2106.05) have been identified as well-known, routine, and conventional steps/functions to one of ordinary skill in the art. When viewed as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself.
Next, when the “machine learning” of claims 21 and 28 is evaluated as an additional element, this feature is recited at a high level of generality and encompasses well-understood, routine, and conventional prior art activity. See, e.g., Balsiger et al., US 2012/0054642, noting in paragraph [0077] that “Machine learning is well known to those skilled in the art.” See also, Djordjevic et al. US 2013/0018651, noting in paragraph [0019] that “As known in the art, a generative model can be used in machine learning to model observed data directly.” See also, Bauer et al., US 2017/0147941, noting at paragraph [0002] that “Problems of understanding the behavior or decisions made by machine learning models have been recognized in the conventional art and various techniques have been developed to provide solutions.” Accordingly, the use of machine learning to infer classifications does not add significantly more to the claim.
In addition, claims 22-27, 29-34, and 41-43 further narrow the abstract idea identified in the independent claims. The Examiner notes that the dependent claims merely further define the data being analyzed and how the data is being analyzed. Similarly, claims 24 and 31 additionally recite “transmit the second ones and the third ones of the classifications via the communication network to at least one domain expert for relabeling” which do not account for additional elements that amount to significantly more than the abstract idea because receiving data and displaying/presenting data (See MPEP 2106.05) have been identified as well-known, routine, and conventional steps/functions to one of ordinary skill in the art and the claimed “at least one processor” and “via the communication network” which do not account for additional elements that amount to significantly more than the abstract idea because the claimed structure merely amounts to the application or instructions to apply the abstract idea on a computer and does not move beyond a general link of the use of an abstract idea to a particular technological environment (See MPEP 2106.05). The additional limitations of the independent and dependent claim(s) when considered individually and as an ordered combination do not amount to significantly more than the abstract idea. The examiner has considered the dependent claims in a full analysis including the additional limitations individually and in combination as analyzed in the independent claim(s). Therefore, the claim(s) are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 21-34 and 42 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garera (U.S 2014/0314311 A1) in view of Kirshenbaum (U.S 2003/0191726 A1).
Claims 21 and 28
Regarding Claim 21, Garera discloses the following:
An apparatus to generate training data for a machine learning model, the apparatus comprising [see at least Paragraph 0024 for reference to the system including one or more server systems that may each be embodied as one or more server computers each including one or more processors that are in data communication with one another; Paragraph 0028 for reference to the computing device being used to perform various procedures; Figure 1 and related text regarding the system for performing methods; Figure 2 and related text regarding the computing device suitable for implementing embodiments; Figures 4 & 5 and related text regarding a method for training a classification model]
interface circuitry to obtain item records [see at least Paragraph 0034 for reference to interface(s) including various interfaces that allow computing device to interact with other systems, devices, or computing environments; Paragraph 0056 for reference to the method receiving an initial training set; Figure 2 and related text regarding item 206 ‘Interface(s)’; Figure 3 and related text regarding item 306 ‘Product Records’; Figure 4 and related text regarding item 402 ‘Receive Initial Training Set’]
machine-readable instructions [see at least Paragraph 0017 for reference to the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium; Paragraph 0018 for reference to a computer-readable medium may comprise any non-transitory medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device; Paragraph 0021 for reference to computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner]
at least one processor circuit to be programmed by the machine-readable instructions to [see at least Paragraph 0020 for reference to computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks; Paragraph 0029 for reference to one or more processors or controllers that execute instructions; Figure 2 and related text regarding item 202 ‘Processor’]
execute the machine learning model to infer classifications for items associated with the item records, ones of the classifications to include one or more class labels in a hierarchical classification taxonomy [see at least Paragraph 0048 for reference to a category of a taxonomy having one or more classification values as descendants thereof may be selected by the analyst module for the generation of training data; Paragraph 0056 for reference to the classifier model being trained using the any machine learning algorithm to produce classification output; Paragraph 0056 for reference to some or all records in the records corpus being classified using the classifier model; Paragraph 0068 for reference to classifying a selected portion of a product records using the classifier model; Figure 4 and related text regarding the training of a classification model including item 406 ‘Classify Product Records Using Classifier’; Figure 5 and related text regarding item 504 ‘Classify Selected Product Records’]
cause a first job to be transmitted over a communication network to a crowdsourcing platform, the first job to include (a) a first portion of pre-labeled data corresponding to first ones of the classifications and (b) a second portion of unlabeled data, the second portion of unlabeled data associated with second ones of the classifications, the second ones of the classifications not included in the first job [see at least Paragraph 0040 for reference to the classifications sent to the crowdsourcing forum including all or less than all classifications in a given iteration method; Paragraph 0042 for reference to the classifications being submitted to a crowdsourcing forum; Paragraph 0058 for reference to classification not identified at a high confidence being submitted to a crowdsourcing for validation; Paragraph 0058 for reference to distributing the some or all of the classifications to participants in a crowdsourcing forum, such as by transmitting classifications to crowdsourcing workstations for display; Figure 3 and related text regarding item 312 ‘Crowdsourcing Forum’; Figure 4 and related text regarding item 410 ‘Submit Remaining Product Records and Classifications to Crowdsourcing’; Examiner asserts that the above “less than all classification” as analogous to the “unlabeled data’]
generate the training data by: aggregating evaluations from the crowdsourcing platform by executing one or more algorithm to generate aggregated evaluations, the evaluations including first evaluations for the first portion of pre-labeled data and second evaluations for the second portion of the unlabeled data [see at least Paragraph 0039 for reference to using the confidence score classifications output by the classifier may be divided into high confidence classifications and other classifications; Paragraph 0042 for reference to the crowdsourcing forum may implement logic for distributing tasks to individuals associated with the forum, receiving responses, and returning responses to a requesting entity; Paragraph 0060 for reference to transmitting some or all of the validations decisions to one or more analyst work stations; Figure 3 and related text regarding item 310 ‘Classifications’; Figure 4 and related text regarding item 412 ‘Receive Yes/No/Can’t Tell for Product Records from Crowdsourcing’]
applying a consensus criterion to the aggregated evaluations to generate validation results for the classifications, the consensus criterion defining a level of consensus among two or more of the evaluations for identifying respective ones of the classifications as valid classifications [see at least Paragraph 0039 for reference to decisions made by the algorithm, e.g. a classification of text, may be assigned a confidence score indicating how much support exists for the decision; Paragraph 0043 for reference to the crowdsourcing forum returning a validation decision; Paragraph 0059 for reference to a validation decision may be received from the crowdsourcing forum; Paragraph 0060 for reference to validation decisions may then be displayed on the analyst workstation, possibly with a prompt to approve or disapprove of the validation decision and/or provide an alternative classification value; Figure 3 and related text regarding item 314 ‘Validation Decisions’; Figure 7 and related text regarding item 702 ‘Sample Classification Validation Decisions’]
identifying first ones of the classifications as the training data based on the validation results, the first ones of the classifications corresponding to the valid classifications [see at least Paragraph 0059 for reference to a validation decision may be received from the crowdsourcing forum; Paragraph 0060 for reference to validation decisions may then be displayed on the analyst workstation, possibly with a prompt to approve or disapprove of the validation decision and/or provide an alternative classification value; Paragraph 0062 for reference to classifications validated by the crowdsourcing forum may be added to the training set; Figure 3 and related text regarding item 314 ‘Validation Decisions’; Figure 7 and related text regarding item 702 ‘Sample Classification Validation Decisions’]
identifying second ones of the classifications as invalid classifications based on the validation results, the second ones of the classifications different than the first ones of the classifications [see at least Paragraph 0059 for reference to a validation decision may be received 412 from the crowdsourcing forum wherein a validation decision for a classification may be a simple yes or no indicating that the classification was correct or incorrect; Figure 3 and related text regarding item 314 ‘Validation Decisions’; Figure 7 and related text regarding item 702 ‘Sample Classification Validation Decisions’]
causing the one or more of the class labels associated with the first ones of the classifications but not the one or more class labels associated with the second ones of the classifications to be transmitted over the communication network for storage in a training data database [see at least Paragraph 0037 for reference to the system storing training data in a database; Paragraph 0037 for reference to the training data 302 may include entries of the form text->classification value, where text is text that might form all or part of a record analyzed or a field of a record analyzed; Paragraph 0062 for reference to classifications validated by the crowdsourcing forum may be added to the training set; Figure 3 and related text regarding item 302 ‘Training Data’ and item 316c ‘Yes’]
cause re-training of the machine learning model based on the one or more class labels associated with the first ones of the classifications retrieved from the training data database via the communication network [see at least Paragraph 0060 for reference to some or all of the validation decisions received from the crowdsourcing forum may be submitted to one or more analysts to verify that validation decisions are accurate; Paragraph 0075 for reference to the accuracy model may output in response to an arbitrary confidence score, a probability that a classification with that confidence score is accurate; Paragraph 0076 for reference to the accuracy model determining a percentage of these classifications that have been validated by a validation decision; Paragraph 0080 for reference to the accuracy model being periodically retrained; Figure 6 and related text regarding item 602 ‘accuracy model’; Figure 7 and related text regarding the process for identifying high accuracy data]
While Garera discloses the limitations above, it does not disclose generate the training data by: aggregating evaluations from the crowdsourcing platform by executing one or more of a majority voting algorithm, a weighted majority algorithm, or a Bayesian voting algorithm to generate aggregated evaluations.
However, Kirshenbaum discloses the following:
generate the training data by: aggregating evaluations from the crowdsourcing platform by executing one or more of a majority voting algorithm, a weighted majority algorithm, or a Bayesian voting algorithm to generate aggregated evaluations [see at least Paragraph 0055 for reference to embodiments of the invention being applied to various types of machine learning or decision problems including classification; Paragraph 0056 for reference to the goal of classification is for the evaluator to assign each object to the appropriate one of a set of alternatives; Paragraph 0079 for reference to if any alternative gets an absolute majority of votes (i.e. it is in the first place preference of one more than half of the intermediate rankings), then that alternative is chosen as the “winner of the election, that is, as the aggregate decision; Paragraph 0082 for reference to if no alternative receives a majority of the first preference votes, either with one vote being given to each evaluator or with each evaluator being given a weighted amount of voting power, then the preferences of the evaluators, other than the first preference, are taken into account in selecting alternatives to eliminate from consideration as the process proceeds in selecting the aggregate winner; Figure 1 and related text regarding computing an aggregate decision about the classification of an object including item 110 ‘Obtain at least two evaluators, for example, by training machine learning models’ and item 160 ‘Optionally, if some alternatives can be decided on, then add them to the aggregate decision’; Figure 2 and related text regarding the training of machine learning models producing evaluators ‘Labeled Training Examples including ‘Bayesian Network’]
Before the effective filing date, it would have been obvious to one of ordinary skill in the art to modify the machine learning algorithm classifier method of Garera to include the majority voting, weighted voting, and Bayesian training methods of Kirshenbaum. Doing so enables computing of an overall or aggregate decision, based on intermediate decisions as to which alternative or alternatives best apply to an object, as stated by Kirshenbaum (Paragraph 0173).
Regarding claim 28, the claim recites limitations already addressed by the rejection of claim 21. Regarding claim 28, Garera teaches at least one non-transitory computer readable medium comprising machine-readable instructions to cause at least one processor circuit to [Paragraph 0018]. Therefore, claim 28 is rejected as being unpatentable in view of Garera and Kirshenbaum.
Claims 22 and 29
While the combination of Garera and Kirshenbaum discloses the limitations above, regarding Claim 22, Garera discloses the following:
wherein one or more of the at least one processor circuit is to generate a report corresponding to the machine learning model [see at least Paragraph 0060 for reference to some or all of the validation decisions received from the crowdsourcing forum may be submitted to one or more analysts to verify that validation decisions are accurate; Paragraph 0075 for reference to the accuracy model may output in response to an arbitrary confidence score, a probability that a classification with that confidence score is accurate; Paragraph 0076 for reference to the accuracy model determining a percentage of these classifications that have been validated by a validation decision; Figure 6 and related text regarding item 602 ‘accuracy model’; Figure 7 and related text regarding the process for identifying high accuracy data]
the report to include accuracy information includes current and historical accuracies associated with the machine learning model [see at least Paragraph 0075 for reference to accuracy model may receive with each individual validation decision received such information as the classification corresponding to the validation decision as well as the confidence score assigned to the classification by the classifier; Paragraph 0080 for reference to the accuracy model being trained continuously as validation decisions are generated and the confidence threshold for a predetermined accuracy; Paragraph 0081 for reference to the accuracy model being trained using all validation decisions, or a random sampling of validation decisions, received in a time interval, e.g. since the last time the accuracy model was trained]
Regarding claim 29, the claim recites limitations already addressed by the rejection of claim 22.
Claims 23 and 30
While the combination of Garera and Kirshenbaum discloses the limitations above, regarding Claim 23, Garera discloses the following:
wherein one or more of the at least one processor circuit is to identify third ones of the classifications as uncertain classifications and refrain from transmitting the one or more class labels associated with the third ones of the classifications to the training data database, the third ones of the classifications different than the first ones of the classifications and the second ones of the classifications [see at least Paragraph 0043 for reference to crowdsourcing forum may return a yes or no response indicating that the classifications was or was not correct; Paragraph 0043 for reference to the crowdsourcing forum may return an “unclear response indicating that the text in a text-classification value output of the classifier is insufficient to accurately judge whether the classification is correct and/or what an accurate classification should be; Figure 3 and related text regarding item 314 ‘Validation Decisions’, item 316a ‘Unclear’, item 316b ‘No’, and item 316c ‘Yes’; Figure 4 and related text regarding item 412 ‘Receive Yes/No/Can’t Tell for Product Records from Crowdsourcing’]
Regarding claim 30, the claim recites limitations already addressed by the rejection of claim 23.
Claims 24 and 31
While the combination of Garera and Kirshenbaum discloses the limitations above, regarding Claim 24, Garera discloses the following:
wherein, one or more of the at least one processor circuit is to transmit the second ones and the third ones of the classifications via the communication network to at least one domain expert for relabeling [see at least Paragraph 0044 for reference to unclear and invalid classifications being further processed; Paragraph 0045 for reference to being submitted to analysts be evaluated for the correctness of the validation decision; Paragraph 0048 for reference to the analyst module may be programmed to select classification values, i.e. values for the classification value fields of the classifications for which additional training data is needed; Paragraph 0049 for reference to the analyst module selecting classification values or categories of classification values on the basis on a percentage of classifications 310 referencing that classification value or category of classification values that were marked as invalid, or either invalid or unclear, by the crowd sourcing forum; Figure 3 and related text regarding item 322 ‘Analyst Module’; Figure 4 and related text regarding item 414 ‘Submit Crowdsourcing Decisions to Analysts for Spotchecking’ and item 416 ‘Identifying Problem Categories (high No. of No/Can’t Tell)’]
Regarding claim 31, the claim recites limitations already addressed by the rejection of claim 24.
Claims 25 and 32
While the combination of Garera and Kirshenbaum discloses the limitations above, regarding Claim 25, Garera discloses the following:
wherein, one or more of the at least one processor circuit is to identify one or more of the second ones of the classifications or the third ones of the classifications as rejected data based on a failure of the domain expert to relabel the one or more of the second ones of the classifications or the third ones of the classifications [see at least Paragraph 0046 for reference to validation decisions found to be incorrect by the analyst may be transmitted to the crowdsourcing forum; Paragraph 0049 for reference to the analyst module selecting classification values or categories of classification values on the basis on a percentage of classifications 310 referencing that classification value or category of classification values that were marked as invalid, or either invalid or unclear, by the crowd sourcing forum; Figure 3 and related text regarding item 320 ‘Feedback’; Figure 4 and related text regarding item 416 ‘Identifying Problem Categories (high No. of No/Can’t Tell)’ and item 418 ‘Request Training Data for Problem Categories from Analysts’]
Regarding claim 32, the claim recites limitations already addressed by the rejection of claim 25.
Claims 26 and 33
While the combination of Garera and Kirshenbaum discloses the limitations above, regarding Claim 26, Garera discloses the following:
one or more of the at least one processor circuit is to select the first ones of classifications based on a sampling strategy and a sample count [see at least Paragraph 0039 for reference to using the confidence score classifications output by the classifier may be divided into high confidence classifications and other classifications; Paragraph 0039 for reference to the number of classifications selected for processing may be chosen in accordance with this capacity; Paragraph 0048 for reference to classification values may be selected individually or as a group; Paragraph 0048 for reference to a category of a taxonomy having one or more classification values as descendants thereof may be selected by the analyst module for the generation of training data; Figure 3 and related text regarding item 310 ‘Classifications’; Figure 4 and related text regarding item 412 ‘Receive Yes/No/Can’t Tell for Product Records from Crowdsourcing’]
Regarding claim 33, the claim recites limitations already addressed by the rejection of claim 26.
Claims 27 and 34
While the combination of Garera and Kirshenbaum discloses the limitations above, regarding Claim 27, Garera discloses the following:
wherein the consensus criterion includes an aggregation algorithm having a sequence of at least a first consensus rule and a second consensus rule [see at least Paragraph 0038 for reference to machine learning algorithm used to implement the classifier ranging from various types of algorithms; Paragraph 0039 for reference to decisions made by the algorithm, e.g. a classification of text, may be assigned a confidence score indicating how much support exists for the decision; Paragraph 0039 for reference to using the confidence score classifications output by the classifier may be divided into high confidence classifications and other classifications; Paragraph 0039 for reference to the number of classifications selected for processing may be chosen in accordance with this capacity]
Regarding claim 34, the claim recites limitations already addressed by the rejection of claim 27.
Claim 42
While the combination of Garera and Kirshenbaum discloses the limitations above, regarding Claim 42, Garrera discloses the following:
wherein the first evaluations are in a first format and the second evaluations are in a second format, the second format different than the first format [see at least Paragraph 0057 for reference to classifications with a confidence score above a specified threshold may be added 408 to the training set wherein a classification may include text->classification value pairs suitable for addition as an entry to the training set; Paragraph 0058 for reference to some or all of the classifications that are not identified as high confidence may then be submitted 410 to a crowdsourcing forum for validation, wherein this may include distributing the some or all of the classifications to participants in a crowdsourcing forum, such as by transmitting classifications to crowdsourcing workstations 106 for display]
Claim(s) 41 and 43 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garera (U.S 2014/0314311 A1) in view of Kirshenbaum (U.S 2003/0191726 A1), as applied in claims 21 and 28, in view of Van Pelt (U.S 2012/0265573 A1).
Claims 41 and 43
While the combination of Garera and Kirshenbaum discloses the limitations above, it does not disclose the first job to include (b) a randomly selected quality control test; compare evaluations of the classifications obtained from the crowdsourcing platform to a quality control metric based on the quality control test to identify quality evaluations.
However, Van Pelt discloses the following:
wherein the first job includes a randomly selected quality control test [see at least Paragraph 0031 for reference to evaluate a workers work quality, reliability in performing crowd sourced tasks via their own user devices 102A-N by intelligently generating evaluation or test tasks with known right answers or known false answers to be presented to a worker or potential worker (e.g., new worker being initially assessed for competency and usability) to see if the worker provides the correct response; Paragraph 0031 for reference to evaluation or test tasks can be generated (e.g., automatically or semi-automatically) from jobs actually submitted to and distributed through the job distribution platform of the host server; Paragraph 0073 for reference to test data/test standards manager 400 can be any combination of software agents and/or hardware modules (e.g., including processors and/or memory units) able to generate test data, test jobs, test or evaluation tasks to assess and evaluate worker competency, ability, and/or usability; Figure 4 and related text regarding components of a test data/test standards manager for use in evaluating worker performing and for use in providing in-task training]
one or more of the at least one processor circuit is to compare evaluations of the classifications obtained from the crowdsourcing platform to a quality control metric based on the quality control test to identify quality evaluations, the quality evaluations including the first evaluations and the second evaluations [see at least Paragraph 0096 for reference to using test jobs/tasks with known responses/answers, the quality/accuracy of a worker's work product can directly be determined and used in computing a quantitative metric indicating the accuracy of a worker's work product; Paragraph 0096 for reference to quality assessment using test jobs/tasks can be performed by the gold standard metric module 242. In addition, the quality assessment can be performed by the peer review metric module 241, where a worker's work product can be evaluated using results generated by other workers (e.g., determine whether they match or not)]
Before the effective filing date, it would have been obvious to one of ordinary skill in the art to modify the crowdsourcing classification apparatus of Garera to include the quality testing of crowdsourcing workers of Van Pelt. Doing so, no additional resources need to be utilized for creation and management of test tasks, as stated by Van Pelt (Paragraph 0031).
Regarding claim 43, the claim recites limitations already addressed by the rejection of claim 41.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Servajean, Maximilien, et al. "Crowdsourcing thousands of specialized labels: A Bayesian active training approach." IEEE Transactions on Multimedia 19.6 (2017): 1376-1391.
DOCUMENT ID
INVENTOR(S)
TITLE
US 9767419 B2
Venanzi et al.
Crowdsourcing system with community learning
WO2013186402A1
Matinez et al.
Classification method and device for large volumes of data
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KRISTIN ELIZABETH GAVIN whose telephone number is (571)270-7019. The examiner can normally be reached M-F 7:30-4:30 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jerry O'Connor can be reached at 571-272-6787. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KRISTIN E GAVIN/Primary Examiner, Art Unit 3624
Prosecution Insights