DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in responsive to communication(s): original application filed on 02/19/2023, said application claims a priority filing date of 08/19/2020. Claims 1-45 are pending. Claims 1, 22, and 43 are independent.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: 224 in FIG. 2A. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 2, 8, 10-14, 16, 17, 24, 28, 31-35, and 44 are objected to because of the following informalities:
in Claim 2, line 1, "… storing instructions that …" appears to be "… storing further instructions that …";
in Claim 8, line 2, "… verifying the annotation performed by …" appears to be "… verifying annotation performed by …";
in Claim 10, line 2 and Claim 31, lines 1-2, "… within the framework of the game or app" appears to be "… within framework of the game or the app";
in Claims 11, lines 2-3; Claim 32, lines 2-3; and Claim 44, lines 1-2, "… based on one or more of user location, user time zone, and/or user responsiveness" appears to be "… based on one or more of user location, user time zone, or user responsiveness" according to Claims 9 and 30;
in Claims 12 and 33, lines 1-2, "… wherein the same task is assigned to multiple selected users, wherein the annotations of the same task by …" appears to be "… wherein same task is assigned to multiple selected users, wherein annotations of the same task by …"; .
in Claims 13 and 34, line 1, "… wherein annotation tasks are …" appears to be "… wherein the annotation tasks are …";
in Claims 14 and 35, line 1, "… selected from the list including …" appears to be "… selected from a list including …";
in Claim 16, line 1, "… storing instructions that …" appears to be "… storing further instructions that …";
in Claim 17, line 1, "… storing instructions that …" appears to be "… storing further instructions that …";
in Claim 24, lines 1-2, "… wherein the dividing of the dataset is performed by ML models" appears to be "… wherein the dividing of the dataset is performed by the ML models";
in Claim 28, lines 1-2, "… verifying the annotation performed by an ML model" appears to be "… verifying annotation performed by one of the ML models".
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6-8, 19-21, and 22-42 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 6 and 26 recite the limitation "the annotation task" in line 1. There is insufficient antecedent basis for this limitation in the claim. For examination purpose, "… wherein one of the annotation tasks is a qualification task" is considered.
Claims 7 and 27 recites the limitation "the annotation task" in line 1. There is insufficient antecedent basis for this limitation in the claim. For examination purpose, "… wherein one of the annotation tasks is a verification task" is considered.
Claims 8 and 28 are rejected for fully incorporating the deficiency of their respective base claims.
Claims 19 and 40 recite the limitation "the selected user" in line 1. There is insufficient antecedent basis for this limitation in the claim. For examination purpose, "… wherein one of the selected users is rated based on a completed task" is considered.
Claims 20 and 41 recites the limitation "the task" in line 1. There is insufficient antecedent basis for this limitation in the claim. For examination purpose, "… wherein the annotation tasks comprise identifying …" is considered.
Claims 21 and 42 are rejected for fully incorporating the deficiency of their respective base claims.
Claims 21 and 42 recite the limitation "… wherein the identifying one or more visual features comprises … selecting the feature" in lines 1-3, which rendering these claims indefinite because "… identifying one or more of a visual feature in an image, a visual feature in a video …" is also recited in its based claim and it is unclear which "visual feature" ("a visual feature in an image" or "a visual feature in a video") is referred by "the feature" here. Clarification is required.
Claim 22 recites the limitation "and/or" in line 6, which rendering these claims indefinite because it is unclear what is included or excluded by the claim language. For examination purpose, "A and/or B" will be considered as "A or B or both" or "one or more of A or B".
Claims 23-42 are rejected for fully incorporating the deficiency of their respective base claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-45 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea without significantly more.
Independent Claims 1-2, 22, and 43
Step 1: Claims 1-2 and 22 are a system claim, and Claim 43 is a process claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) "dividing a dataset to be annotated into annotation tasks" (i.e., a person can mentally divide a dataset into annotation tasks), "distributing the annotation tasks to [models and/or (Claims 2 and 22)] a plurality of selected users for completion of the annotation tasks, wherein the selected users are playing a game or using an app and wherein the annotation tasks are performed by the selected users as part of in-game or in-app advertising" (i.e., a person can mentally distribute tasks to users), and "reassembling the completed annotation tasks into an annotated dataset" (i.e., a person can mentally reassemble completed annotated tasks into a dataset) which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) recite(s) additional elements/limitations of "dataset annotation system (DAS)" (Claim 1 and 22), "one or more processors" (Claim 1), "at least one non-transitory computer readable medium" (Claim 1), "annotator engine", "distribution server", and "machine learning" (Claims 2 and 22) which only amount to "apply it" with the use of generic computer components or insignificant extra solution activity. None of the additional elements/limitations, taken alone or in combination, integrate the abstract idea into a practical application.
Step 2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements/limitations of "dataset annotation system (DAS)" (Claim 1 and 22), "one or more processors" (Claim 1), "at least one non-transitory computer readable medium" (Claim 1), "annotator engine", and "distribution server" are just generic computer elements and "machine learning" (Claims 2 and 22) is well-understood, routine and conventional (WURC) activity similar to "performing repetitive calculation" (see MPEP 2106.05(d), "Performing repetitive calculations, Flook, 437 U.S. at 594, 198 USPQ2d at 199 (recomputing or readjusting alarm limit values)"). Thus, none of the additional elements/limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 3 and 23
Step 1: Claims 3 and 23 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) "the annotation tasks are performed" (i.e., a person can mentally perform annotated tasks) which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) recite(s) additional element/limitation of ".
Step 2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional element/limitation of .
Claims 4 and 24
Step 1: Claims 4 and 24 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) "the dividing of the dataset is performed by models" which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) recite(s) additional element/limitation of ".
Step 2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional element/limitation of .
Claims 5 and 25
Step 1: Claims 5 and 25 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) "the dividing of the dataset is performed manually by an operator" which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) recite(s) additional element/limitation of ".
Step 2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional element/limitation of .
Claims 6 and 26
Step 1: Claims 6 and 26 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 7 and 27
Step 1: Claims 7 and 27 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 8 and 28
Step 1: Claims 8 and 28 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) recite(s) additional element/limitation of ".
Step 2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional element/limitation of .
Claims 9 and 30
Step 1: Claims 9 and 30 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 10 and 31
Step 1: Claims 10 and 31 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 11, 32, and 44
Step 1: Claims 11 and 32 are a system claim, and Claim 44 is a process claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 12 and 33
Step 1: Claims 12 and 33 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " the same task is assigned to multiple selected users" (i.e., same task can be mentally assigned to multiple users) and "the annotations of the same task by the selected users are evaluated as a group" (i.e., the annotations of the same task by the selected users can be mentally evaluand as a group) which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) recite(s) additional element/limitation of ".
Step 2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional element/limitation of .
Claims 13 and 34
Step 1: Claims 13 and 34 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 14 and 35
Step 1: Claims 14 and 35 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) recite(s) additional element/limitation of ".
Step 2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional element/limitation of .
Claims 15 and 36
Step 1: Claims 15 and 36 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 16 and 37
Step 1: Claims 16 and 37 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 17, 38, and 45
Step 1: Claims 17 and 38 are a system claim, and Claim 45 is a process claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 18 and 39
Step 1: Claims 18 and 39 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 19 and 40
Step 1: Claims 19 and 40 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 20 and 41
Step 1: Claims 19 and 40 are a system claim. These claims are fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) " or organizing human activity.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 6-20, 22-23, 26-28, 30-41, and 43-45 are rejected under 35 U.S.C. 103 as being unpatentable over Welinder et al. (US 2013/0346409 A1, pub. date: 12/26/2013), hereinafter Welinder in view of von Ahn Arellano (US 2005/0014118 A1, pub. date: 01/20/2015), hereinafter von Ahn Arellano.
Independent Claims 1, 22, and 43
Welinder discloses a dataset annotation system [a system comprising a dataset annotation system (DAS) (Claim 22)] (Welinder, ¶ [0038] with 100 and 110 in FIG. 1: distributed data annotation system 100 includes distributed data annotation server system 110; ¶ [0042] with 200 in FIG. 2: distributed data annotation server system 200 ) comprising one or more processors (Welinder, ¶ [0042] with 210 in FIG. 2: processor 210) and at least one non-transitory computer readable medium (Welinder, ¶ [0042] with 230 in FIG. 2: memory 230) having stored thereon instructions (Welinder, ¶ [0042]: memory 230 is any form of storage configured to store a variety of data, including data annotation application 232, source data 234, source data metadata 236, and annotator performance metadata 238; ¶ [0046]: store data or applications on disk or some other form of storage and are loaded into the memory at runtime can be utilized) as appropriate to the requirements of specific applications) that, when executed by the one or more processors, cause the dataset annotation system to perform a method (Welinder, ¶¶ [0037] and [0043]: using the distributed data annotation server system (e.g. as software configuring a processor in the distributed data annotation server system to perform the annotation process), the method comprising [the DAS further including (Claim 22)]:
a. dividing a dataset to be annotated into annotation tasks by an annotator engine [a. an annotator engine configured for dividing a dataset to be annotated into annotation tasks (claim 22)] (Welinder, ¶ [0032]: obtain sets of source data and create training data sets representative of the obtained source data; the training data sets can include pieces of source data from the obtained source data and/or additional pieces of data representative of the set of source data; the pieces of source data in the training data set are analyzed (such as by an expert or previously known information) and the ground truth of the features (or the absence thereof) are identified within the pieces of source data; ¶ [0039] with FIG.1: obtain pieces of source data and store the pieces of source data using source data database 120; source data database 120 includes one or more references (such as a uniform resource locator) to source data that is stored in a distributed fashion; source data database 120 includes one or more sets of source data to be annotated using distributed data annotation server system 110; a set of source data includes one or more pieces of source data including image data, audio data, signal data, and text data; one or more pieces of source data in source data database 120 includes source data metadata describing features of the piece of source data; generate sets of training data representative of the source data and identify the ground truth with respect to one or more features of the pieces of source data in the training data set; ¶ [0043] with FIG. 2: generating training data sets representative of the set of source data 234; the training data sets include pieces of source data and source data metadata describing the ground truth of one or more features within the source data; source data metadata 234 can include annotations provided for the piece of source data, the source of the provided annotations, the time taken to annotate the piece of source data, rewards earned for providing the annotations, and/or one or more properties describing the piece of source data; ¶ [0045]: clustering annotations of a particular feature within a piece of source data; clustering of annotations can be utilized to determine the confidence ( e.g. correctness) of the provided annotations and/or be utilized in the determination of the performance of the annotators providing the annotations; ¶¶ [0047]-[0048] with FIG. 3: obtaining (310) source data and determining (312) features within the source data; the obtained (310) source data contains one or more pieces of source data; the pieces of source data can be image data, audio data, video data, signal data, text data, or any other data appropriate to the requirements of specific applications; the pieces of source data can include source data metadata describing attributes of a piece of source data; determining (312) source data features includes determining the ground truth regarding one or more features within some or all of the obtained (310) source data; the pieces of source data with determined (312) source data features are utilized to create a training data set representative of the obtained (310) source data; the source data features can be determined (312) as appropriate to the requirements of specific applications such as by expert annotators, known from the creation of the training data set, and/or crowdsourced utilizing systems); [and (Claim 22)]
b. distributing the annotation tasks to a plurality of selected users by a distribution server for completion of the annotation tasks [b. a distribution server configured for distributing the annotation tasks to machine learning (ML) models and/or a plurality of selected users for completion of the annotation tasks (Claim 22)] (Welinder, ¶¶ [0031]-[0036]: determine the performance of annotators at specific types of annotation tasks and use the performance information in the allocation of annotation tasks; the training data is then distributed to one or more annotators that are configured to identify the features within the pieces of source data; based on the annotator performance, annotators can be scored based on their performance in various annotation tasks and ranked according to the scores; the annotator rankings can be used in the selection (or targeting) of annotators for particular annotation tasks and the rankings can be updated based upon additional tasks the annotators perform; request the iterative annotation of source data; obtaining an initial set of annotations from a set of annotators, then providing the source data and the initial set of annotations to a second set of annotators that refine, correct, and/or confirm the initial set of annotations; the capabilities of the annotators with respect to the source data can be considered in the allocation of (additional) annotation tasks to refine and/or improve the annotation of the source data; the capabilities of the annotators can be determined utilizing a variety of techniques; each of the initial broad categories can then be transmitted to the annotators by the distributed data annotation server system to further refine the source data metadata associated with each piece of source data in the broad categories and the process repeated until sufficient; based upon the updated annotator performance metadata, the distributed data annotation server system can further refine the selection of annotator (groups) and/or pieces of source data targeted in additional annotation tasks for subsequent annotations of the source data metadata describing the source data is collected; data annotation devices can include human annotators, machine-based annotation devices, and/or a combination of machine and human annotation; identify features within pieces of source data based on a received annotation task; ¶ [0039] with FIG. 1: distribute the training data to one or more data annotation devices 130 and requests annotations of one or more features within the source data; request annotations of pieces of source data not in the training data sets from one or more of the annotator groups based on the annotator performance metadata of distributed data annotation devices 130 within the targeted annotator groups; identify pieces of source data for refinement and iteratively distribute the identified pieces of source data to data annotation devices 130 to further refine, expand, and/or correct the annotations applied to the identified pieces of source data by previous data annotation devices 130; allow for the selection (e.g. targeting) of annotators for particular annotation tasks; ¶ [0040] with FIG.1: data annotation devices 130 include human annotators, machine annotators, and emulations of human annotators performed using machines; human annotators can constitute any human-generated annotators, including users performing human intelligence tasks via a service such as the Amazon Mechanical Turk service provided by Amazon.com, Inc.; allow a user to view the pieces of source data received by the data annotation device 130 and provide annotations (such as identifying features within a piece of source data) for the pieces of source data; ¶¶ [0043]-[0045] with FIG. 2: transmitting the training data sets to one or more annotators; the selection of annotators is based on annotator performance metadata 238; the annotators are configured to generate annotations identifying one or more features within the received source data and generate source data metadata 236 containing the annotations; transmitting pieces of source data to the annotator groups based on their performance identified using the training data sets representing the source data; allow for the targeting of particular annotators and/or groups of annotators for one or more annotation tasks based on the annotator rankings and/or the annotator profiles; a first set of annotators annotates pieces of source data and the first set of annotations is then provided to a second set of annotators that refine the first set of annotations and/or identify additional features within the pieces of source data; ¶¶ [0047]-[0048] with FIG.3: in the crowdsourced annotation of source data, a variety of annotators with a wide range of knowledge and ability will be utilized in the annotation of the source; each annotator has an associated cost such as time, money, or both; annotators can be trained to perform particular annotation tasks using training data sets; these training data sets can also be utilized to determine the relative performance of the annotators; annotators are requested (314) and obtained (316); requesting (314) annotations for the training data set includes creating one or more annotation tasks that configure an annotator (and/or a data annotation device) to annotate one or more features within a piece of source data; requesting (314) annotations includes submitting the annotation task(s) to a human intelligence task marketplace, such as the Amazon Mechanical Turk service provided by Amazon.com, Inc.; requesting (314) annotations can include submitting an annotation task to an annotation device configured to perform machine intelligence tasks based on the annotation tasks; , obtaining (316) annotations includes receiving the annotations from a data annotation device via a network connection), wherein the selected users are (Welinder, ¶ [0015]: the data annotation application configures to obtain sets of annotations from a set of annotators for a portion of the training data set, where an annotation identifies one or more features within a piece of source data in the training data set; ¶ [0031]: in a variety of applications including medical diagnosis, surveillance verification, performing data de-duplication, transcribing audio recordings, or researching data details, a large variety of source data, such as image data, audio data, signal data, and text data, can be generated and/or obtained; by annotating features within these pieces of source data, particular portions of the source data can be identified for particular purposes and/or additional analysis; ¶ [0039] with FIG. 1: data annotation devices 130 generate annotations for one or more features within the source data and transmit annotated source data to distributed data annotation server system 110; ¶ [0043] with 232 in FIG. 2: data annotation application 232 configures to perform a distributed data annotation process for set of source data 234, which includes receiving the annotated pieces of source data); and
c. reassembling the completed annotation tasks into an annotated dataset [wherein the DAS is further configured for reassembling the completed annotation tasks into an annotated dataset (Claim 22)] (Welinder, ¶ [0032]: based on the measured performance, the annotators can be grouped together into annotator groups; the annotator groups (or subsets thereof) can then be tasked with the additional annotation on the pieces of source data that are not in the training data sets; the annotation tasks for the non-training pieces of source data can be targeted to the annotator groups based on their performance in that the better performing groups are more likely to be able to complete difficult annotation tasks quickly and/or in a cost-effective manner; ¶ [0039] with FIG. 1: based on the annotated source data, determine annotator performance metadata describing the performance of the data annotation devices 130 based on the provided annotations and the ground truth for the pieces of source data in the training data set; based on the annotator performance metadata, data annotation devices 130 are grouped into annotator groups; ¶ [0043] with FIG. 2: determining the performance of the annotators based on the source data metadata 236; the annotator performance is stored as annotator performance metadata 238; the annotators are grouped into annotator groups based on their performance; ¶¶ [0047] and [0049]-[0051] with FIG. 3: the annotators can be grouped according to their performance and annotation tasks and/or rewards can be targeted towards the best (or adequate) performing annotators while additional time and/or money need not be allocated toward the poorly performing annotators; annotator performance is determined (318) and the annotators are grouped (320); annotator performance is determined (318) with respect to how well the annotations provided by the annotator correspond to the ground truth of those features as identified within the training data set; this correspondence is determined by measuring the distance between the ground truth of the feature and the annotation of the feature using a thresholding heuristic; the number of correct detections can be determined along with the number of incorrect detections (e.g. false alarms) and the number of missed annotations within the provided annotations; annotator precision can be determined based on the number of correct detections, while annotator recall can be determined based on the total number of features annotated and the total number of features within the annotated portions of the training data set; annotator performance metadata is determined (318) based on the precision and recall of the annotator; annotator performance metadata can also be determined (318) based on the time it took the annotator to provide annotations and/or complete annotation tasks, the cost of the annotations, and/or any other annotator characteristic; annotator performance is determined (318) based on a confidence score calculated based on the clustering of detections across multiple annotators annotating the same piece of source data within the training data set; grouping (320) annotators can be based on the determined (318) annotator performance metadata; the annotators can be grouped (320) based on the similarity of their annotator performance metadata and/or be grouped into groups if the annotator performance metadata exceeds a threshold value; this threshold value can be predetermined and/or determined dynamically based on the determined (318) annotator performance metadata; annotators are grouped (320) based on the error rate of the annotators).
Welinder fails to explicitly disclose wherein the selected users are playing a game or using an app and wherein the annotation tasks are performed by the selected users as part of in-game or in-app advertising.
von Ahn Arellano teaches a system and a method relating to labeling/annotating data (von Ahn Arellano, ¶ [0002]), wherein the selected users are playing a game or using an app and wherein the annotation tasks are performed by the selected users as part of in-game or in-app advertising (von Ahn Arellano, ABSTRACT and ¶¶ [0010]-[0014] and [0023]-[0024] with FIG. 2: determine the contents of an image uses an online game that is played by a large number of people at once; introduces a game that is fun to play, and which can be used to determine the contents of an image; providing access to an online electronic game to a plurality of participants over a communication network; selecting at least two participants from a plurality of participants, presenting an image to each selected participant; requesting each selected participant to provide a description of the image; and receiving from each of the at least two selected participants at least one content-identifying term for the image; ¶¶ [0026]-[0033] with FIG. 3: after a player logs into the host computer 12 (using the player's or participants respective computing unit) offering the image-labeling game (block 26), the game software in the host computer 12 may randomly assign a game partner (block 28) to the player; the partner assignment establishes a pair of participants/players ready to participate in the image labeling game; the game control software may then commence a timer (block 30) for the
current pair of participants; simultaneously with starting the game timer (block 30), the software may select and retrieve (e.g., from the database 22) a new image to be sent to the computer terminals of both players so that the players can view an identical image on their computers (block 32); after retrieving and sending an identical image to both players, the game control Software may monitor the timer to determine whether the
predetermined time period (T) has elapsed (decision block 34); if the predetermined time duration (e.g., T=90 seconds) has elapsed without obtaining an identical match between the content-identifying terms received from both the players, the game control software may check at decision step 36; whether one or both of the players have indicated to stop the game (e.g., by logging out of the game, or by a prolonged period of inactivity); if the game is to be concluded, then the software finishes the game and Sends appropriate game conclusion message to both the players at block 38; if the game is to be continued despite no timely matching responses from both the players, the game control software may instruct the host computer 12 to retrieve another image from the database 22 and Send this new image to the players computer terminals (block 32); the paired players may then continue content identification of the most-recently received new image as indicated at blocks 40, 42, and 44 in FIG. 3; if the timer has not run out at step 34, the game control software may continue to wait for and receive the
content-identifying texts from both the players; at block 40, the software may receive the input text from one of the players and store it in the database 22 in data storage set A so long as the text does not contain any forbidden word(s); similarly, at block 42, the software may receive the image content-identifying entries from the other player and store the entries in the data storage set B so long as the entries do not contain any forbidden word(s); so long as the timer value is less than the predetermined time limit for content identification of the current image, the software may continue receiving text entries from one or both the players; at the decision block 44, the game control software continually compares the input strings stored in sets A and B to check whether any of the string pairs intersect, i.e., whether there is an identical String received from each of the two players; in the absence of the identical match, the software may continue to receive more entries from the players, so long as the timer has not run out (as indicated by the process loop between blocks 44 and 34); on the other hand, if there is an identical entry stored in both sets A and B, then the software may determine that a match has been found and, hence, may conclude the game of content identification of the present image and may continue the game with a new image so long as the “Game Over indication is absent; the online image labeling game may encourage participation of a larger audience, the game hosting website may offer "reward" to participants in terms of, for example, game points redeemable for select online merchandise or souvenirs offered by the host website; furthermore, to maintain participants continued interest in the game, the host website may offer online image-labeling tournaments or team activities to determine winning players who "agree' on the most number of images in the shortest time interval; other business objectives may be used to publicize the game and make it a success).
Welinder and von Ahn Arellano are analogous art because they are from the same field of endeavor, a system and a method relating to labeling/annotating data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of von Ahn Arellano to Welinder. Motivation for doing so would harness human resources to obtain content identification or labeling of a large number of images online in a fun game; i.e., help computers determine the contents of images and maintain users interest with entertainment and rewards at the same time (.
Claim 2
Welinder in view of von Ahn Arellano discloses all the elements as stated in Claim 1 and further discloses to distribute the annotation tasks to machine learning (ML) models (Welinder, ¶ [0036]: annotators utilize data annotation devices to perform the annotation of pieces of source data; data annotation devices can include (but are not limited to) human annotators, machine-based annotation devices, and/or a combination of machine and human annotation; ¶ [0040] with FIG.1: data annotation devices 130 include human annotators, machine annotators, and emulations of human annotators performed using machines; ¶ [0047]: in the crowdsourced annotation of source data, a variety of annotators with a wide range of knowledge and ability will be utilized in the annotation of the source; annotators can be trained to perform particular annotation tasks using training data sets; ¶ [0048] with FIG. 3: requesting (314) annotations can include submitting an annotation task to an annotation device configured to perform machine intelligence tasks based on the annotation tasks; ¶ [0053]: by having a model of different annotators, annotation tasks can be tailored toward the strengths of the annotator, reducing the total cost of obtaining the annotations).
Claims 3 and 23
Welinder in view of von Ahn Arellano discloses all the elements as stated in Claims 1 and 22 respectively and further discloses wherein the annotation tasks are performed on a mobile device (Welinder, ¶ [0040] with FIG.1: data annotation devices 130 are illustrated as personal computers configured using appropriate software; data annotation devices 130 include tablet computers, mobile phone handsets, software running on distributed data annotation server system 110, and/or any of a variety of network-connected devices).
Claims 6 and 26
Welinder in view of von Ahn Arellano discloses all the elements as stated in Claims 1 and 22 respectively and further discloses wherein the annotation task is a qualification task (Welinder, ¶¶ [0033]-[0034]: based on the annotator performance, annotators can be scored based on their performance in various annotation tasks and ranked according to the scores; the annotator rankings can be used in the selection (or targeting) of annotators for particular annotation tasks and the rankings can be updated based upon additional tasks the annotators perform; the rankings can also include annotator profiles including a history of annotation tasks performed, the performance with respect to particular annotation tasks, the time taken for annotations, rewards earned by the annotator, requested rewards for additional annotation tasks, a user rating for the annotator, demographic information, and/or particular types of source data the annotator is skilled in annotating; obtaining an initial set of annotations from a set of annotators, then providing the source data and the initial set of annotations to a second set of annotators that refine, correct, and/or confirm the initial set of annotations; this iterative process can be repeated until the desired level of certainty with respect to the annotations is achieved; ¶ [0039] with FIG. 1: based on the annotated source data, determine annotator performance metadata describing the performance of the data annotation devices 130 based on the provided annotations and the ground truth for the pieces of source data in the training data set; ¶¶ [0047] and [0050]-[0051] with FIG. 3: rewards can be targeted towards the best (or adequate) performing annotators while additional time and/or money need not be allocated toward the poorly performing annotators; annotator performance is determined (318) based on a confidence score calculated based on the clustering of detections across multiple annotators annotating the same piece of source data within the training data set; annotators are grouped (320) based on the error rate of the annotators).
Claims 7 and 27
Welinder in view of von Ahn Arellano discloses all the elements as stated in Claims 1 and 22 respectively and further discloses wherein the annotation task is a verification task (Welinder, ¶ [0032]: annotator performance can be based on precision (e.g. the number of annotations that are correct with respect to the ground truth), recall (the number of features present that are annotated), and any other attributes (including total cost, and time taken); ¶ [0039] with FIG. 1: based on the annotated source data, determine annotator performance metadata describing the performance of the data annotation devices 130 based on the provided annotations and the ground truth for the pieces of source data in the training data set; ¶ [0046]: identifying annotator performance, and calibrating the annotation process to the annotator performance; ¶¶ [0049]-[0050] with FIG. 3: annotator performance is determined (318) with respect to how well the annotations provided by the annotator correspond to the ground truth of those features as identified with