DETAILED ACTION
This Office Action is in response to Applicant's Communication received on 08/16/2023 for application number 17/910,805.
Claims 1-14 are presented for examination. Claims 1, 13, and 14 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/10/2022 has been considered by the Examiner.
Claim Objections
Claims 1, 13, and 14 are objected to because of the following informalities: in these claims “based reliability information of the workers” should be “based on reliability information of the workers”; “a work including a plurality of unit task” should be “a work including a plurality of unit tasks”; “the first task result inference step”, “the reliability information update step”, and “the task result of the workers” have no antecedent basis. Also, claim 14 recites “a computing-readable medium” in the preamble but subsequently refers to “the computer-readable medium”, and the term should consistently read “computer-readable medium” throughout the claim. Appropriate correction is required.
Claims 2, 4, 5, 6, 7, 9 are objected to because of the following informalities: these claims recite “the unit task”, which has no antecedent basis. Also, claim 4 recites “the worker”, which has no antecedent basis. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 13 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because the claim is directed to software per se.
Claim 13 recites “a system... the system performing steps comprising” followed exclusively by method steps. Claim 13 recites no structural component – no processor, no memory, and no other component. A “system” defined solely by the steps it performs reads on a collection of program instructions - software per se. Thus, claim 13 is directed to a nonstatutory subject matter.
Claim 14 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because the claim is directed to signals per se.
Claim 14 recites “a computing-readable medium” with no “non-transitory” qualifier. Specification [0181] indicates that the software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or in a signal wave to be transmitted; [0182] indicates that an example of the computer-readable medium includes a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specially configured to store and execute a program instruction such as ROM, RAM, and flash memory. According to MPEP 2111, claim terms and phrases must be given their broadest reasonable interpretation in light of specification. Thus, the phrase “computing-readable storage medium” will be reasonably interpreted as a medium including signals. Signal, a form of energy, does not fall within one of the four statutory classes of 35 U.S.C. §101. Thus, claim 14 is directed to a nonstatutory subject matter.
Claims 1-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1
Claims 1-12 are directed to a method and fall within one of the statutory categories (process) and are eligible under Step 1.
Step 2A Prong 1
Independent Claim
Claim 1 recites:
a method for deriving task results by reflecting reliability information of workers processing works collected through crowdsourcing, the method comprising: a task result inference step of deriving a first comprehensive task result for each of a plurality of unit tasks, based reliability information of the workers and the task results of the workers with respect to each of a plurality of unit tasks for deriving a comprehensive task result; a reliability information updating step of updating the reliability information of each of the workers, based on the first comprehensive task result and the task result of the workers; and deriving a final comprehensive task result for each of the unit tasks, based on the updated reliability information of each of the workers and the task results of the workers, wherein the task result inference step and the reliability information update step is sequentially performed N times or more (N is a natural number greater than or equal to 1), the reliability information of the workers in the first task result inference step is determined according to a preset rule, and the reliability information of the workers used in an M times of the task result inference step (M is a natural number greater than or equal to 2) corresponds to reliability information updated in an M-1 times of reliability information update step - these limitations encompass mental processes and mathematical concepts. Evaluating a set of workers' answers, judging which workers are more reliable, forming a consensus answer weighted by that judgement, and then rejudging each worker's reliability by how closely the worker's answers track the consensus are acts of observation, evaluation, and judgement that can be practically performed in a human mind or with pen and paper, and thus recites mental process. Additionally, the same weighted aggregation (comprehensive task result) and iterative re-estimation are also mathematical relationships and calculations.
Accordingly, the claims recites an abstract idea that falls under “mental process” and “mathematical concepts” grouping.
Step 2A Prong 2
Independent Claim
Additional elements
performed in a computing device having at least one memory and at least one processor - these limitations are recited at a high-level of generality such that it amount to no more than using generic computer components to apply the judicial exception (see MPEP § 2106.05(f)).
receiving task results of a plurality of workers for a work including a plurality of unit task - these limitations amount to insignificant extra-solution activity of mere data gathering (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the judicial exception into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to the abstract idea.
Step 2B
Independent Claim
Additional elements
performed in a computing device having at least one memory and at least one processor - these limitations are recited at a high-level of generality such that it amount to no more than using generic computer components to apply the judicial exception (see MPEP § 2106.05(f)).
receiving task results of a plurality of workers for a work including a plurality of unit task - these limitations amount to insignificant extra-solution activity of mere data gathering , which is well-understood, routine, and conventional activity (see MPEP § 2106.05(d), “receiving/ transmitting data”).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claim is patent ineligible.
Step 2A Prong 1
Dependent Claims
Claim 2:
deriving and updating the reliability information for each worker to minimize an error value between a task result for each of the workers and a first comprehensive task result for the unit task corresponding to the task result for each of the workers - these limitations encompass mathematical calculations to minimize error/ objective value.
Claim 3:
the reliability information of the worker includes a plurality of detailed reliability information determined according to a number of a plurality of values that corresponds to a task result for a work including the unit task - these limitations merely furthers the mathematical concepts by specifying the reliability parameters s a set of values.
Claim 4:
the reliability information of the worker includes a plurality of detailed reliability information determined according to a number of a plurality of values that corresponds to a task result for a work including the unit task - these limitations merely furthers the mathematical concepts by specifying the reliability parameters s a set of values.
Claim 5:
when the number of a plurality of values that may correspond to the task result for the work including the unit task is N, the reliability information of the worker includes detailed reliability information about the probability that the worker answers with the j-th value to the task result of the unit task corresponding to the actual i-th value (i, j is a natural number less than or equal to N), that is, total N * N detailed reliability information - these limitations encompass mathematical calculations and relationships.
Claim 6:
the reliability information of the worker when a value of the task result for the work including the unit task is True or False includes: first detailed reliability information about the probability that the worker evaluates the task result of the unit task corresponding to an actual truth as True; second detailed reliability information on the probability that the worker evaluates the task result of the unit task corresponding to an actual truth as False; third detailed reliability information about the probability that the worker evaluates the task result of the unit task corresponding to an actual False as True; and fourth detailed reliability information on the probability that the worker evaluates the task result of the unit task corresponding to an actual False as False - these limitations encompass mathematical calculations and relationships.
Claim 7:
the task result inference step, when a value of the task result for the work including the unit task is True or False, includes assigning a first value when the value of the task result is True, and assigning a second value when the value of the task result is False, so that the first comprehensive task result for each of a plurality of unit tasks is derived by using the following [Equation 1]. [Equation 1] First comprehensive task result for i-th unit task = f(Ej reliability information * task ) (where, task result, is a value of a task result evaluated by the j-th worker for the i-th unit task, reliability informationj is reliability information of the j-th worker, and f is a function representing a value reflecting the reliability informationj in the task result as an interpretable comprehensive transformation value) - these limitations encompass mathematical calculations and relationships.
Claim 8:
the task result inference step, when a value of the task result for the work including the unit task is True or False, includes assigning a first value when the value of the task result is True, and assigning a second value when the value of the task result is False, so that the first comprehensive task result for each of a plurality of unit tasks is derived by using the following [Equation 2].[Equation 2] First comprehensive task result for i-th unit task = f (Ej reliability information * task result i. j) (where, task result,, is a value of a task result evaluated by the j-th worker for the i-th unit task, reliability informationj is values of the first detailed reliability information - the third detailed reliability information of the j-th worker, or values of the fourth detailed reliability information - the second detailed reliability information, and f is a function representing a value reflecting the reliability informationj in the task result as an interpretable comprehensive transformation value) - these limitations encompass mathematical calculations and relationships.
Claim 9:
the task result inference step includes deriving the first comprehensive task result for each of a plurality of unit tasks by using the following [Equation 3] with respect to the task result for the work including the unit task. [Equation 3] First comprehensive task result for i-th unit task = f (Ej reliability information * task resultl,) (where, task result,, is a value of a task result evaluated by the j-th worker for the i-th unit task, reliability informationj is reliability information of the j-th worker, and f is a function representing a value reflecting the reliability informationj in the task result as an interpretable comprehensive transformation value) - these limitations encompass mathematical calculations and relationships.
Claim 10:
the reliability information update step includes updating reliability information to minimize an error value between the first comprehensive task result for each of the unit tasks derived through [Equation 3] in the task result inference step and the task results for each of the unit tasks for each of the workers - these limitations encompass mathematical calculations and relationships.
Claim 11:
an initial reliability information deriving step of deriving initial reliability information of the workers based on the test results of the workers, wherein the task result inference step includes deriving the first comprehensive task result for each of a plurality of unit tasks based on the initial reliability information for each of the workers and the task results of the workers at an initial execution - these limitations encompass mental process and/ or mathematical concepts.
Thus, the claims recite the abstract idea.
Step 2A Prong 2
Dependent Claims
Additional elements
Claim 11:
receiving test results of a plurality of workers for a plurality of initial reliability tests
- these limitations amount to insignificant extra-solution activity of mere data gathering (see MPEP § 2106.05(g)).
Claim 12:
the step of receiving the test result includes receiving test results for a plurality of initial reliability tests performed between works including the unit tasks performed by the workers - these limitations amount to insignificant extra-solution activity of mere data gathering (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the judicial exception into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to the abstract idea.
Step 2B
Dependent Claims
Additional elements
Claim 11:
receiving test results of a plurality of workers for a plurality of initial reliability tests
- these limitations amount to insignificant extra-solution activity of mere data gathering, which is well-understood, routine, and conventional activity (see MPEP § 2106.05(d), “receiving/ transmitting data”).
Claim 12:
the step of receiving the test result includes receiving test results for a plurality of initial reliability tests performed between works including the unit tasks performed by the workers - these limitations amount to insignificant extra-solution activity of mere data gathering, which is well-understood, routine, and conventional activity (see MPEP § 2106.05(d), “receiving/ transmitting data”).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are patent ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, and 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (US 2014/0172767 A1 hereinafter Chen) in view of Drutsa et al. (US 2020/0380410 A1 hereinafter Drutsa).
Regarding Claim 1, Chen teaches a method for deriving task results by reflecting reliability information of workers processing works collected through crowdsourcing (Abstract - optimize the number of correct decisions made by a crowdsourcing system given a fixed budget, tasks for multiple decisions are allocated to workers; [0001] crowdsourcing is a process of providing a task to a large number of individual workers, and using the combined results from the individual workers for that task to make a decision) performed in a computing device having at least one memory and at least one processor ([0073] computing machine 400 typically includes at least one processing unit 402 and memory 404), the method comprising:
receiving task results of a plurality of workers for a work including a plurality of unit task ([0032] the initial assignments are provided to the task processing module, which obtains 304 results from the workers for the assigned tasks; [0039] there are K instances and each one is associated with a true label; each time a label is received from the crowd for the i-th instance (i.e., plurality of unit tasks));
a task result inference step of deriving a first comprehensive task result for each of a plurality of unit tasks, based reliability information of the workers and the task results of the workers with respect to each of a plurality of unit tasks for deriving a comprehensive task result ([0027] a task is allocated to a worker based on results already achieved for that task from other workers; such allocation addresses the different levels of difficulty of decisions; a task also can be allocated to a worker based on results already received for other tasks from that worker; such allocation addresses the different levels of reliability of workers; [0028] the process of allocating tasks to workers is modeled as a Bayesian Markov decision process; a prior distribution, representing the likelihood that an item will be correctly labeled, is defined for each item; if variability in worker reliability is modeled, a prior distribution, representing the likelihood that a worker will label an item correctly; given the information already received for each item and worker, an estimate of the number of correct labels received can be determined by calculating posterior distributions given the data already received and the prior distributions; [0058] workers' reliability also can be modeled; there are M workers, the reliability of the j-th worker can be captured by introducing an extra parameter ρj; let Zij be the label provided by the j-th worker for the i-th instance; given the true label Zi ∈ {−1, 1}, for any instance i, we define ρj=Pr(Zij=Zi|Zi); Pr(Zij=1)=ρjθi+(1-ρj)(1-θi) - thus, the inferred class of instance I is given by the posterior quantity and the posterior quantity computed at an intermediate stage is equivalent to the first comprehensive task result);
a reliability information updating step of updating the reliability information of each of the workers, based on the first comprehensive task result and the task result of the workers ([0036] because different workers have different reliabilities, the underlying reliability of workers can be estimated during the labeling process to avoid spending more of the budget on those unreliable workers; [0058] workers' reliability also can be modeled; there are M workers, the reliability of the j-th worker can be captured by introducing an extra parameter ρj; [0059] this model is often called one-coin model; worker reliability ρj is also drawn from a Beta prior distribution; at each stage t, the system decides on both the next instance i to be labeled and the next worker j to label the instance i; algorithm describing the optimistic knowledge gradient incorporating workers' reliability; we can further extend it to a more complex two-coin model by introducing a pair of parameters (ρj1, ρj2) to model the j-th worker's reliability - thus, worker reliability is re-estimated from the collected results and the current label estimate); and
deriving a final comprehensive task result for each of the unit tasks([0038] when the budget is exhausted, a final inference of the true class for each instance can be determined based on the collected labels; [0042] the true label of each instance is inferred based on the collected labels; a positive set HT is determined, which maximizes the conditional expected accuracy; [0043] the final positive set HT can be determined by the Bayes decision rule), wherein
the task result inference step and the reliability information update step is sequentially performed N times or more (N is a natural number greater than or equal to 1) ([0032] the initial assignments are provided to the task processing module, which obtains 304 results from the workers for the assigned tasks; the task processing module then updates 306 the database with the received results; if the budget has been exhausted, as determined at 308, the process ends; [0033] If the budget has not yet been exhausted, then the optimization engine computes 310 an optimization of the expected number of correct labels given the results of the tasks so far; the optimization engine selects 312 the next task and worker assignment, and provides the assignment to the task processing module, and the steps 304 through 312 repeat until the budget is exhausted; [0038] the labeling process can be decomposed into T stages; updating its posterior distribution each time a new label is collected. See fig. 6 - it shows each iteration performing selection, acquisition of a label, and a posterior update), and the reliability information of the workers used in an M times of the task result inference step (M is a natural number greater than or equal to 2) corresponds to reliability information updated in an M-1 times of reliability information update step ([0060] worker reliability ρj is also drawn from a Beta prior distribution: ρj˜Beta(c0 j, d0 j); [0041] the posterior distribution of θit in the stage t+1 will be updated as Beta(at|1 it, bt|1 it); we put {at i, bt i}K i<1 into a K×2 matrix St, called a state matrix - thus, the reliability/ state used at a later stage is the value updated at the immediately preceding stage).
However, Chen fails to expressly teach wherein deriving a final comprehensive task result based on the updated reliability information of each of the workers and the task results of the workers and the reliability information of the workers in the first task result inference step is determined according to a preset rule.
In the same field of endeavor, Drutsa teaches wherein deriving a final comprehensive task result based on the updated reliability information of each of the workers and the task results of the workers ([0063] the quality score 112 of each given human assessor 106 indicates a reliability of a given result of a task completed by the given human assessor 106; [0094] determining a result for a task executed in a crowd-sourced environment; the crowd-sourcing application 118 executes: a receiving routine 202, a verification routine 204, and an output routine 206; [0105] the receiving routine 202 is configured to transmit a data packet 208 to the verification routine 204; the data packet 208 comprises (i) the result 212; (ii) the quality score 112 associated with the result 212; [0116] the output routine 206 is configured to receive two data packets, namely the data packet 220 received from the verification routine 204 which comprises the quality score 112 associated with the result 212, and a data packet 222 from the trusted assessor 218; [0117] the data packet 222 comprises a label 224 assigned by the trusted assessor 218, which is indicative of the result 212 corresponding to a correct answer to the task, or being an incorrect answer to the task; [0118] in response to the label 224 being indicative that the result 212 corresponds to the correct answer to the task, the output routine 206 is configured to process the task as being completed; the output routine 206 is then configured to calculate and issue a reward to the human assessor 106 who has submitted the result 212; [0119] the output routine 206 is configured to increase the quality score 112 of the human assessor who has submitted the result 212 labelled as the correct answer to the task; [0121] output routine 206 is further configured to decrease the quality score 112 of the human assessor 106 who has submitted the result 212 being the incorrect answer to the task - thus, the result of a future task is determined based on the updated quality score/ reliability information of the worker and the result received from the worker ) and the reliability information of the workers in the first task result inference step is determined according to a preset rule ([0063]the quality score 112 of each given human assessor 106 indicates a reliability of a given result of a task completed by the given human assessor 106; [0064] the quality scores 112 may be determined based on a first plurality of “honeypot tasks” completed by each of the human assessors 106; [0065] the results of the first plurality of honeypot tasks provided by the human assessors 106 are recorded in the database 104; a percentage of the first plurality of honeypot tasks that the given human assessor 106 completes correctly is calculated and recorded in the database 104 as the quality score 112 of the given human assessor 106 (i.e., preset rule)).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein deriving a final comprehensive task result based on the updated reliability information of each of the workers and the task results of the workers and the reliability information of the workers in the first task result inference step is determined according to a preset rule, as taught by Drutsa into Chen. Doing so would be desirable because it would allow for properly assessing whether a given assessor should be considered trustworthy or not in order to determine whether to discard the result provided by the assessor (Drutsa [0009]).
As to dependent Claim 2, Chen and Drutsa teach all the limitations of Claim 1. Drutsa further teaches wherein the reliability information update step includes: deriving and updating the reliability information for each worker to minimize an error value between a task result for each of the workers and a first comprehensive task result for the unit task corresponding to the task result for each of the workers ([0063] the quality score 112 of each given human assessor 106 indicates a reliability of a given result of a task completed by the given human assessor 106; [0094] determining a result for a task executed in a crowd-sourced environment; the crowd-sourcing application 118 executes: a receiving routine 202, a verification routine 204, and an output routine 206; [0105] the receiving routine 202 is configured to transmit a data packet 208 to the verification routine 204; the data packet 208 comprises (i) the result 212; (ii) the quality score 112 (i.e., deriving reliability information for the worker)associated with the result 212; [0107] the verification routine 204 is configured to execute a first machine learning algorithm (MLA) 210 trained to generate an error parameter for the result 212, based on the result 212 and the user activity history associated with the result 212; [0116] the output routine 206 is configured to receive two data packets, namely the data packet 220 received from the verification routine 204 which comprises the quality score 112 associated with the result 212, and a data packet 222 from the trusted assessor 218; [0117] the data packet 222 comprises a label 224 assigned by the trusted assessor 218 (i.e., first comprehensive task result), which is indicative of the result 212 corresponding to a correct answer to the task, or being an incorrect answer to the task; [0118] in response to the label 224 being indicative that the result 212 corresponds to the correct answer to the task, the output routine 206 is configured to process the task as being completed; the output routine 206 is then configured to calculate and issue a reward to the human assessor 106 who has submitted the result 21 2 (i.e., to minimize an error value between a future task result for the worker and a first comprehensive task result/ 224 for the unit task corresponding to the task result for the worker); [0119] the output routine 206 is configured to increase the quality score 112 of the human assessor who has submitted the result 212 labelled as the correct answer to the task; [0121] output routine 206 is further configured to decrease the quality score 112 of the human assessor 106 who has submitted the result 212 being the incorrect answer to the task (i.e., updating the reliability information for the worker)).
As to dependent Claim 4, Chen and Drutsa teach all the limitations of Claim 1. Drutsa further teaches wherein the reliability information of the worker includes a plurality of detailed reliability information determined according to a number of a plurality of values that corresponds to a task result for a work including the unit task ([0063]the quality score 112 of each given human assessor 106 indicates a reliability of a given result of a task completed by the given human assessor 106; [0064] the quality scores 112 may be determined based on a first plurality of “honeypot tasks” completed by each of the human assessors 106; [0065] the results of the first plurality of honeypot tasks (i.e., number of a plurality of values that corresponds to a task result for a work including unit task) provided by the human assessors 106 are recorded in the database 104; a percentage of the first plurality of honeypot tasks that the given human assessor 106 completes correctly is calculated and recorded in the database 104 as the quality score 112 of the given human assessor 106).
As to dependent Claim 11, Chen and Drutsa teach all the limitations of Claim 1. Drutsa further teaches wherein receiving test results of a plurality of workers for a plurality of initial reliability tests ([0015] receiving, by the server, a quality score associated with the user, the quality score being indicative of a reliability of the user; and in response to the assigned label value being indicative of the result being the incorrect answer to the task, lowering the quality score associated with the user; [0063]the quality score 112 of each given human assessor 106 indicates a reliability of a given result of a task completed by the given human assessor 106; [0064] the quality scores 112 may be determined based on a first plurality of “honeypot tasks” completed by each of the human assessors 106; [0065] the results of the first plurality of honeypot tasks provided by the human assessors 106 are recorded in the database 104; a percentage of the first plurality of honeypot tasks that the given human assessor 106 completes correctly is calculated and recorded in the database 104 as the quality score 112 of the given human assessor 106) ; and an initial reliability information deriving step of deriving initial reliability information of the workers based on the test results of the workers, wherein the task result inference step includes deriving the first comprehensive task result for each of a plurality of unit tasks based on the initial reliability information for each of the workers and the task results of the workers at an initial execution ([0064] the quality scores 112 may be determined based on a first plurality of “honeypot tasks” completed by each of the human assessors 106; [0116] the output routine 206 is configured to receive two data packets, namely the data packet 220 received from the verification routine 204 which comprises the quality score 112 associated with the result 212, and a data packet 222 from the trusted assessor 218; [0117] the data packet 222 comprises a label 224 assigned by the trusted assessor 218 (i.e., first comprehensive task result), which is indicative of the result 212 corresponding to a correct answer to the task, or being an incorrect answer to the task).
As to dependent Claim 12, Chen and Drutsa teach all the limitations of Claim 11. Drutsa further teaches wherein the step of receiving the test result includes receiving test results for a plurality of initial reliability tests performed between works including the unit tasks performed by the workers ([0063]the quality score 112 of each given human assessor 106 indicates a reliability of a given result of a task completed by the given human assessor 106; [0064] the quality scores 112 may be determined based on a first plurality of “honeypot tasks” (i.e., initial reliability tests) completed by each of the human assessors 106).
Claim 13 is a system claim corresponding to the method claim 1 above and therefore, rejected for the same reasons. Chen further teaches wherein a system for deriving task results ([0073] example computing environment includes a computing machine; computing machine 400 typically includes at least one processing unit 402 and memory 404).
Claim 14 is a medium claim corresponding to the method claim 1 above and therefore, rejected for the same reasons. Chen further teaches wherein a computing-readable medium for implementing a method, the computer-readable medium stores instructions for allowing the computing device to perform steps ([0073] example computing environment includes a computing machine; computing machine 400 typically includes at least one processing unit 402 and memory 404; computing machine 400 may also include additional storage; computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer program instructions, data structures, program modules or other data; medium which can be used to store the desired information and which can accessed by computing machine 400).
Claims 3 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Drutsa, further in view of Ipeirotis et al. (“Quality Management on Amazon Mechanical Turk”, 2010, hereinafter Ipeirotis).
As to dependent Claim 3, Chen and Drutsa teach all the limitations of Claim 1. Drutsa further teaches wherein the reliability information update step includes: repeatedly updating the reliability information of each of the workers based on the task result for each of the workers and the first comprehensive task result ([0063] the quality score 112 of each given human assessor 106 indicates a reliability of a given result of a task completed by the given human assessor 106; [0094] determining a result for a task executed in a crowd-sourced environment; the crowd-sourcing application 118 executes: a receiving routine 202, a verification routine 204, and an output routine 206; [0105] the receiving routine 202 is configured to transmit a data packet 208 to the verification routine 204; the data packet 208 comprises (i) the result 212; (ii) the quality score 112 associated with the result 212; [0118] in response to the label 224 (i.e., first comprehensive task result) being indicative that the result 212 corresponds to the correct answer to the task, the output routine 206 is configured to process the task as being completed; the output routine 206 is then configured to calculate and issue a reward to the human assessor 106 who has submitted the result 21 2; [0119] the output routine 206 is configured to increase the quality score 112 of the human assessor who has submitted the result 212 labelled as the correct answer to the task; [0121] output routine 206 is further configured to decrease the quality score 112 of the human assessor 106 who has submitted the result 212 being the incorrect answer to the task (i.e., updating the reliability information for the worker)).
However, Chen and Drutsa fail to expressly teach wherein the first comprehensive task result derived just before until the reliability information for each worker converges to a specific value.
In the same field of endeavor, Ipeirotis teaches wherein the first comprehensive task result derived just before until the reliability information for each worker converges to a specific value (page 2, section 2 - the EM algorithm of Dawid and Skene takes as input a set of N objects, each being associated with a latent true class label T(on), picked from one of the L different labels; each object is annotated by one or more of the K workers, each having a varying degree of quality; the algorithm iterates between estimating the correct labels T(on) for each of the objects, and estimating the error rates for each worker. See Algorithm 1 - it shows the EM algorithm where the worker error rates are estimated from the labels, the labels are then re-estimated from the error rates with each worker's vote weighted by that worker's quality, and the procedure returns to the error rate estimation step and iterates until convergence).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein first comprehensive task result derived just before until the reliability information for each worker converges to a specific value, as taught by Ipeirotis into Chen and Drutsa. Doing so would be desirable because it would allow for quality management of the labeling process on crowdsourced environments (Ipeirotis, page 4, section - Conclusions).
As to dependent Claim 5, Chen and Drutsa teach all the limitations of Claim 1. However, Chen and Drutsa fail to expressly teach wherein when the number of a plurality of values that may correspond to the task result for the work including the unit task is N, the reliability information of the worker includes detailed reliability information about the probability that the worker answers with the j-th value to the task result of the unit task corresponding to the actual i-th value (i, j is a natural number less than or equal to N), that is, total N * N detailed reliability information.
In the same field of endeavor, Ipeirotis teaches wherein when the number of a plurality of values that may correspond to the task result for the work including the unit task is N, the reliability information of the worker includes detailed reliability information about the probability that the worker answers with the j-th value to the task result of the unit task corresponding to the actual i-th value (i, j is a natural number less than or equal to N), that is, total N * N detailed reliability information (page 2, section 2 - the EM algorithm of Dawid and Skene takes as input a set of N objects, each being associated with a latent true class label T(on), picked from one of the L different labels; each object is annotated by one or more of the K workers, each having a varying degree of quality; the algorithm iterates between estimating the correct labels T(on) for each of the objects, and estimating the error rates for each worker; the algorithm endows each worker (k) with a latent “confusion matrix” π(k) ij, which gives the probability that worker(k), when presented with an object of true class i, will classify the object into category j - thus, for a task having N possible label values, each worker is characterized by an NxN array whose (I,j) entry is the probability that the worker assigns the j-th label value to an object whose actual value is the i-th).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein when the number of a plurality of values that may correspond to the task result for the work including the unit task is N, the reliability information of the worker includes detailed reliability information about the probability that the worker answers with the j-th value to the task result of the unit task corresponding to the actual i-th value (i, j is a natural number less than or equal to N), that is, total N * N detailed reliability information, as taught by Ipeirotis into Chen and Drutsa. Doing so would be desirable because it would allow for quality management of the labeling process on crowdsourced environments (Ipeirotis, page 4, section - Conclusions).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Drutsa, further in view of Raykar et al. (“Learning From Crowds”, 2010, hereinafter Raykar).
As to dependent Claim 6, Chen and Drutsa teach all the limitations of Claim 1. However, Chen and Drutsa fail to expressly teach wherein the reliability information of the worker when a value of the task result for the work including the unit task is True or False includes: first detailed reliability information about the probability that the worker evaluates the task result of the unit task corresponding to an actual truth as True; second detailed reliability information on the probability that the worker evaluates the task result of the unit task corresponding to an actual truth as False; third detailed reliability information about the probability that the worker evaluates the task result of the unit task corresponding to an actual False as True; and fourth detailed reliability information on the probability that the worker evaluates the task result of the unit task corresponding to an actual False as False.
In the same field of endeavor, Raykar teaches wherein the reliability information of the worker when a value of the task result for the work including the unit task is True or False includes: first detailed reliability information about the probability that the worker evaluates the task result of the unit task corresponding to an actual truth as True (page 5, section 2.1 - let yj ∈{0,1} be the label assigned to the instance x by the jth annotator/expert; the sensitivity (true positive rate) for the jth annotator is defined as the probability that she labels it as one. aj :=Pr[yj = 1|y =1]); second detailed reliability information on the probability that the worker evaluates the task result of the unit task corresponding to an actual truth as False (page 5, section 2.1 - let yj ∈{0,1} be the label assigned to the instance x by the jth annotator/expert; the sensitivity (true positive rate) for the jth annotator is defined as the probability that she labels it as one. aj :=Pr[yj = 1|y =1] - thus, because the annotator's label is binary, the compliment of 1-aj is the actual truth evaluated as False); third detailed reliability information about the probability that the worker evaluates the task result of the unit task corresponding to an actual False as True (page 5, section 2.1 - let yj ∈{0,1} be the label assigned to the instance x by the jth annotator/expert; if the true label is zero, the specificity (1−false positive rate) is defined as the probability that she labels it as zero. bj := Pr[yj = 0|y =0] - thus, because the annotator's label is binary, the compliment of 1-bj is the actual false evaluated as True); and fourth detailed reliability information on the probability that the worker evaluates the task result of the unit task corresponding to an actual False as False (page 5, section 2.1 - if the true label is zero, the specificity (1−false positive rate) is defined as the probability that she labels it as zero. bj := Pr[yj = 0|y =0]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein the reliability information of the worker when a value of the task result for the work including the unit task is True or False includes: first detailed reliability information about the probability that the worker evaluates the task result of the unit task corresponding to an actual truth as True; second detailed reliability information on the probability that the worker evaluates the task result of the unit task corresponding to an actual truth as False; third detailed reliability information about the probability that the worker evaluates the task result of the unit task corresponding to an actual False as True; and fourth detailed reliability information on the probability that the worker evaluates the task result of the unit task corresponding to an actual False as False, as taught by Raykar into Chen and Drutsa. Doing so would be desirable because it would automatically discover the best experts and assigns a higher weight to them (Raykar, page 3, section - 1.2).
Regarding dependent claims 7-10, Examiner would like to note that there are currently no prior art rejections for these claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 CFR § 1.111(c) to consider these references fully when responding to this action.
Dubey et al. (US 2017/0061357 A1) teaches: determine a set of task completion probabilities based on the task information and the worker information; a task completion probability, of the set of task completion probabilities, may identify a likelihood that a particular crowd, of the one or more crowds, will complete the task; the worker information include particular skills associated with the crowd and/or individual workers of the crowd (e.g., proficiency in a programming language, fluency in a spoken language, data entry speed, etc.), past performance of the worker/crowd (e.g., a percentage of past tasks completed by the worker, a reliability indicator indicating how likely the worker is to complete a task, quality of work associated with previously completed tasks, etc.) (see [0004], [0063]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to REJI KARTHOLY whose telephone number is (571)272-3432. The examiner can normally be reached on Monday - Thursday from 7:30 am to 3:30 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch, can be reached at telephone number 571-272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/REJI KARTHOLY/Primary Examiner, Art Unit 2143
Prosecution Insights