Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see page 28-36, filed 7/17/2026, with respect to Objections for minor informalities and 35 U.S.C. 112(b) have been fully considered and are persuasive. The objection and rejections the claims in the previous office action have been withdrawn.
Applicant's arguments filed 7/17/2026 have been fully considered but they are not persuasive.
Regarding applicant’s argument for 35 U.S.C 101 rejection, applicant argues in page 37-42 “
PNG
media_image1.png
129
742
media_image1.png
Greyscale
” Applicant argues that the office action recites that the application includes significantly more and that “directed to abstract idea without significantly more”. The sections quoted by the applicant means that the application is directed to abstract idea without significantly more.
Applicant further argues in page 37 Step 2A“
PNG
media_image2.png
422
934
media_image2.png
Greyscale
”. The applicant explains that the specification provides a technical improvement. Further applicant states that claim 1 recites a closed loop quantifying and informativeness score, yet claim 1 states “wherein the re-generating of the informativeness score is based on the updated estimation of competency of the expert and the adjusted labels, such that a subsequent prioritization order for a subsequent human annotation is derived from the re-generated informativeness score” only states re-generating the informativeness score and that a subsequent prioritization order is derived from the informativeness score (see MPEP 2106.05(f)) and further argues how the specification identifies the technical improvement. In page 39 applicant further argues “
PNG
media_image3.png
300
924
media_image3.png
Greyscale
” However the claim does not recite any feedback loop and training pipeline. The claim limitations are not interpreted to have a specific order unless mentioned to have a specific order. As explained above MPEP 2106.05(f) recites “(1) Whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished. The recitation of claim limitations that attempt to cover any solution to an identified problem with no restriction on how the result is accomplished and no description of the mechanism for accomplishing the result, does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it”.” Applicant continues on to argue step 2B in page 39-40 “Step 2B - the ordered combination supplies an inventive concept
Should the analysis reach Step 2B, the same ordered combination supplies an inventive concept. An inventive concept may be found in a nonconventional und nongeneric arrangement of elements, even where the individual elements themselves are known, The specification confirms that the claimed arrangement is non-conventional”. The applicant argues how the specification provides the required improvement on the technology as explain in applicant arguments for step 2A prong 2 and 2B. As explained the claims do not reflect the improvement on the technology and the proposed feedback loop the applicant claims that shows an improvement is not claimed (see MPEP 2106.05(a)).
In page 41-42, applicant argues “
PNG
media_image4.png
202
920
media_image4.png
Greyscale
” yet the claims only provide unifying annotations as claimed in claim 6 and generic machine learning training as claimed in claim 7 (See MPEP 2106.05(f)). In claim 12 applicant claims “
PNG
media_image5.png
239
927
media_image5.png
Greyscale
” that uses a computer to output an analysis based on a pattern, a trend, and an association relating to one of human behavior or human-computer interaction. As previously mentioned no particular process or method is claimed to achieve the analyzation required by the claim or how its integrated into a practical application (see MPEP 2106.05(f)).
Further the amended limitations have not been examined thus the argument is moot and not convincing. See updated 35 U.S.C 101 rejection.
Regarding applicant’s argument for 35 U.S.C 103 rejection, applicant argues in page 43-46
“
PNG
media_image6.png
294
731
media_image6.png
Greyscale
PNG
media_image7.png
115
732
media_image7.png
Greyscale
” Applicant argues how the amended claims 1, 13, 19 overcome the prior art rejection based on a feedback-loop limitation. However, based on the amended claim limitation it is no apparent of what is meant by a feedback-loop and how a subsequent prioritization order is derived from the informativeness score. Further applicant argues amend limitations and the amend limitations have not been examined, thus the argument is moot.
In page 46 applicant argues that claim 2-5,14-16, and 20-21 as they depend on claim 1, 13 and 19.
In page 49 applicant argues “
PNG
media_image8.png
204
751
media_image8.png
Greyscale
” Applicant argues how Zhdanov does not teach the limitations of claim 6 as Zhdanov does not teach the most probable set of annotations based on adversarial annotations. Further argues “
PNG
media_image9.png
73
722
media_image9.png
Greyscale
” The claims limitations are given the broadest reasonable interpretation and given the limitations. Also the specification states in paragraph 0010 “The method may compare annotations of the human and/or an another human with each other, and generate a most-probable annotation based on : (1) annotations assigned by human experts and/or their competencies, (2) predictions of machine learning model trained on annotations, (3) substantial dataset (4) historical data, (5) uncertainty of trained machine learning model, (6) annotations generated from the annotation verification task, (7) predictions of a machine learning model for the annotation verification task, and/or (8) uncertainty of the machine learning model for the annotation verification task” Based on the requirements of claim 6 Zhdanov teaches “Col 5 60-67 and Col 6 line 1-7, Once annotations have been received from the WIS 118 and optionally from machine annotation service 114, the annotations can be consolidated into labels by annotation consolidation service 122, as shown at numeral 9. Annotation consolidation may refer to the process of taking annotations from multiple annotators ( e.g., humans and/or machines) and consolidating these together (e.g., using majority-consensus heuristics, removing bias or low-quality annotators, using probabilistic distribution that minimizes a risk function for observed, predicted and true labels, or other techniques. For example, based on each annotators' accuracy history, their annotations can be weighted. If one annotator has a 50% accurate history, their annotations may have a lower weight than another annotator with a 100% accurate history.).
Col 6 line 9-16,
In some embodiments, the annotation consolidation service 122 can maintain a label score and a worker score when performing annotation consolidation scores for each piece of data in the dataset ( e.g., image, video frame, audio utterance, etc.) and current worker scores for the annotator who provided the annotations on that piece of data.
Col 6 line 18-26, The label score can be compared to a specified threshold. If the label score is higher than the threshold then no additional annotations are required. If the label score is lower than the threshold then the data may be passed to additional annotators to be further annotated by annotating service 120. At numeral 10, once the label score is higher than the threshold, then the core engine 110 can be updated to indicate that the subset of the input dataset has been labeled” Zhdanov teaches determine the most probable set of annotations for single piece of data as each annotators label is weighted based on accuracy and the annotations can conflict with the annotations of other annotators. The limitation in question recites “any one of conflicting annotations and adversarial annotations” only requires for either conflicting annotations or adversarial annotations.
In page 50 Applicant argues that claim 7, 8, and 24 are allowable because Zhdanov does not teach claim 6, 17, 23 however as explain above Zhdanov does teach the claims. Applicant also argues claim 8 limitations are not shown after amending out the 112(b) issue identified in the previous OA and argues amend limitations.
In page 50-51, applicant argues that claim 9 and 10 that the cited refence does not teach the amend limitation. However, the amend limitations have not been examined, thus the argument is moot.
In page 52-53, applicant argues that the amend claims overcome the prior art rejection and claim 12 limitations are therefore allowable. However, the argument is not convincing and moot as the claims are amended.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-24 rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea without significantly more. The subject matter eligibility test for products and process is describe below for claim 1 in view of dependent claims.
Regarding claim 1:
Step 1: Is the claim to a process machine manufacture or composition of matter?
Yes – Claim 1 recites a method, which a method that falls under the statutory categories.
Step 2A Prong 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes – The claim recites the following:
“quantifying an informativeness score of data elements in the ” –
The limitation recites a mathematical process of quantifying an informativeness score see MPEP 2106.04(a)(2)I.
The limitation recites a mental process of determining how likely and by what degree the elements lead to model improvement (see MPEP 2106.04(a)(2)III).
“matching the selected data to an expert based on at least one of a competency and a preference of the expert;” The limitation recites a mental process of matching data to an expert based on competency and preference (see MPEP 2106.04(a)(2)III).
“adjusting an estimation of competency of the expert based on annotations obtained from the expert,” The limitation recites a mathematical process of the estimation of competency (see MPEP 2106.04(a)(2)I).
“wherein the annotations to match an expected unified label based on an artificial intelligence algorithm comprising at least one of a weighted voting algorithm, a multiple machine learning models voting algorithm, and an expectation maximization algorithm;” - The limitation recites a mathematical process of performing one of these a weighted voting algorithm, a multiple machine learning models voting algorithm, and an expectation maximization algorithm (see MPEP 2106.04(a)(2)I).
“determining that a confidence in the selected data satisfies a required level by examining at least one of the updated competencies of the expert, consensus among multiple experts, and analysis through a machine learning model;” The limitation recites a mental process of determining that the selected data is certain enough by consensus of among multiple experts (see MPEP 2106.04(a)(2)III).
Step 2 Prong 2: Does the claim recite additional elements that integrate the judicial exception into a particular application? No –
The claim includes the additional element(s):
“A method, comprising: determining which objects from a an increase ”
The additional elements fall under “apply it” as using a generic computer to determine which objects from an a dataset are expected to increase in model quality by applying a samples-selection algorithm. See Mere Instructions to Apply an Exemption (see MPEP 2106.05(f)).
“automatically determining which data elements of the ;”
The additional elements fall under “apply it” as using a generic computer to automatically determine elements in the dataset require human annotation. See Mere Instructions to Apply an Exemption (see MPEP 2106.05(f)).
“wherein data having an informativeness score satisfying a selection criterion ”
The additional elements fall under Insignificant Extra-Solution Activity. See MPEP 2106.5(g).
“choosing a selected data based on the automatically determining which elements of the ”
The additional elements fall under Insignificant Extra-Solution Activity as mere data gathering. See MPEP 2106.5(g).
“generating an annotation view of the selected data tailored to at least one of the preference and the competency of the expert through which the expert is able to annotate the selected data;”
The additional elements fall under “apply it” as using a generic computer to generate an annotation view. See Mere Instructions to Apply an Exemption (see MPEP 2106.05(f)).
“adjusting labels corresponding to annotated elements of the selected data m response to annotations and an update to the estimation of competency of the expert;”
The additional elements fall under “apply it” as using a generic computer adjust labels. See Mere Instructions to Apply an Exemption (see MPEP 2106.05(f)).
“re-generating the informativeness score for the ”
The additional elements fall under “apply it” as using a generic computer to re-generate the informative scores. See Mere Instructions to Apply an Exemption (see MPEP 2106.05(f)).
“wherein the re-generating of the informativeness score is based on the updated estimation of competency of the expert and the adjusted labels, such that a subsequent prioritization order for a subsequent human annotation is derived from the re-generated informativeness score.”
The additional elements fall under “apply it” as using a generic computer to re-generate the informative scores based on the competency of the expert and the adjusted labels. See Mere Instructions to Apply an Exemption (see MPEP 2106.05(f)).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No - The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exemption. As order as a whole, the claim is directed to identifying data in need of expert annotations. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of selecting, generating, adjusting and re-generating fall under using generic computer to apply an exemption and mere data gathering. The method does not improve on the function of a computer, transforms an article into another article, nor is it applied by a particular machine, making the claim not patent eligible.
Regarding claim 2:
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 1 further comprising: determining which objects from the based on one of the following: a score of annotation being trustworthy based on human annotators competence estimation on the examined sample; a measure of conformance to distribution learned by machine learning model; and a difficulty score based on representation of the data element.”
The additional elements fall under “apply it” as using a generic computer to use machine learning model to apply a sample selection algorithm (see MPEP 2106.05(f)).
The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
Regarding claim 3:
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 2 further comprising: expanding a database with annotations assigned by humans to data elements in a manner that the same element may be annotated by multiple human annotators.”
The additional elements fall under Insignificant Extra-Solution Activity as mere data gathering. See MPEP 2106.5(g).
The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application
Regarding claim 4:
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 1 further comprising applying each operation of the method to an annotation verification task to audit whether annotations of other humans are accurate.”
The additional elements fall under “apply it” as using a generic computer to apply a verification task to audit annotations. See Mere Instructions to Apply an Exemption (see MPEP 2106.05(f)). The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
Regarding claim 5:
Step 2A Prong 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes – The claim recites the following:
“comparing annotations of the human and an another human with each other;”
The limitation recites a mental process of comparing annotations of the human with another (see MPEP 2106.04(a)(2)III).
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 4 further comprising: enabling other humans to perform the annotation verification task to audit whether annotations of other humans are accurate;”
The additional elements fall under “apply it” as using a generic computer to enable a human to perform the verification task. See Mere Instructions to Apply an Exemption (MPEP 2106.05(f)). The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
“and generating a most-probable annotation based on at least one of: annotations assigned by human experts and their competencies, predictions of machine learning model trained on at least one of annotations, ”
The additional elements fall under “apply it” as using a generic computer to enable generate the most probable annotation. See Mere Instructions to Apply an Exemption (MPEP 2106.05(f)). The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
Regarding claim 6:
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 5 further comprising: unifying the annotations of at least one of the human, the another human, and the annotation verifications with each other using a label unification module to create a unified subsequent dataset, and wherein the label unification module selects a most probable set of annotations for the single piece of data which was annotated with any one of conflicting annotations and adversarial annotations.”
The additional elements fall under “apply it” as using a generic computer to use unification module to unify annotations into a subsequent data and selecting most probable set of annotations. See Mere Instructions to Apply an Exemption (MPEP 2106.05(f)). The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
Regarding claim 7:
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 6 further comprising: using the unified subsequent dataset as labels for a training dataset, and using composition of those as the input for training the machine learning model.”
The additional elements fall under “apply it” as using a generic computer to us the unified subsequent data set as labels for a training dataset. See Mere Instructions to Apply an Exemption (MPEP 2106.05(f)). The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
Regarding claim 8:
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 7 further comprising: assessing the ununified dataset with an annotation assessment module to determine at least one of: an expert competency, whether designations were applied to the annotated data in an adversarial manner, and whether a designation matching the unified label ”
The additional elements fall under “apply it” as using a generic computer to assess using the assessment module to determine an expert competency, if the designation were applied in an adversarial manner or whether a most correct designation was applied to an unannotated data sample. See Mere Instructions to Apply an Exemption (MPEP 2106.05(f)). The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
Regarding claim 9:
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 8 further comprising: automatically balancing an exploration and an exploitation of expert competencies to optimize obtained annotation quality;”
The additional elements fall under “apply it” as using a generic computer to automatically balance the exploration and exploitation of expert competencies. See Mere Instructions to Apply an Exemption (MPEP 2106.05(f)). The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
“reaching an end condition wherein no further annotated data, subsequent annotated data, and unified subsequent dataset are inputted into the machine learning model.”
The additional elements fall under Insignificant Extra-Solution Activity. See MPEP 2106.5(g). The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
Regarding claim 10:
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 9: wherein expert competencies ocomprises an exploration to assess expert competencies performed by at least one of: creating a new artificial unannotated data sample, a choosing sample from the substantial dataset, choosing sample from selected data”
The additional elements fall under “apply it” as using a generic computer to explore the assess expert competencies by at least one of: creating a new artificial unannotated data sample, choosing sample from the substantial dataset, choosing sample from selected data. The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
Regarding claim 11:
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 10: wherein the that a traditional data-processing method is unable to organize into a structure schema ”
The additional elements fall under Insignificant Extra-Solution Activity. See MPEP 2106.5(g). The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
Regarding claim 12:
Step 2A Prong 2, Step 2B: The additional element(s):
“The method of claim 11 further comprising: analyzing the of a pattern, a trend, and an association relating to at least one of a human behavior and a human-computer interaction.”
The additional elements fall under “apply it” as using a generic computer to analyze the sensational dataset. The judicial exemptions do not integrate into a practical application nor provide an improvement. The process does not provide an inventive concept nor provides a practical application.
Claims 13-18 recite a method and are analogous to the method of claims 1-12. Therefore, the rejections of claim 1-12 above applies to claims 13-18.
Claims 19-24 recite a system and are analogous to the method of claims 1-12. Therefore, the rejections of claim 1-12 above applies to claims 19-24.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-10, and 13-21 are rejected under 35 U.S.C. 103 as being unpatentable over Elisha et al. (US11636389B2) (“Elisha”) in view of Zhu et al. (US20220188575A1) (“Zhu”) and further in view of Welinder, Peter, and Pietro Perona. "Online crowdsourcing: rating annotators and obtaining cost-effective labels." 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. IEEE, 2010 (“Welinder”) and Zhdanov et al. (US11048979) (“Zhdanov”).
Regarding claim 1, Elisha teaches A method, comprising:
determining which objects from a capability comprising a processor and a memory of at least one of a processing system and a graphics processing unit (Elisha Col line 8-22, FIG. 9 shows a flowchart of a method 900 for improving prediction accuracy of an ML model by identifying and eliminating erroneous training samples, according to an example embodiment. Embodiments disclosed herein and other embodiments may operate in accordance with example method 900. Method 900 comprises steps 902-920. However, other embodiments may operate according to other methods. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the foregoing discussion of embodiments. No order of steps is required unless expressly indicated or inherently required. There is no requirement that a method embodiment implement all of the steps illustrated in FIG. 9. FIG. 9 is simply one of many possible embodiments. Embodiments may implement fewer, more or different steps.
Col 23 15-29, In step 916, a removal list may be created by ordering the erroneous samples ( e.g., from a highest to a lowest first variance) followed by suspect samples. For example, as shown in FIG. 2, training evaluator 246 may create a removal list that prioritizes removal of erroneous training samples before suspect training samples.
In step 918, erroneous samples may be selectively removed in order from the removal list to create a revised training set. For example, as shown in FIG. 2, training evaluator 246 may cause data fetcher 208 to selectively remove training samples ( e.g., based on user input provided through portal 212). For example, data fetcher 208 may change an assigned category for targeted training samples so that they are not fetched as training data 204 when classifier model(s) 214b is trained for the category in question.
In step 920, prediction accuracy of the first ML model may be improved by training the first ML model with the revised training set instead of the training set. For example, as shown in FIG. 2, classifier model(s) 214b may be trained by training data 204 that has been revised by removing erroneous and/or suspect training samples [determining which objects from a substantial dataset are expected to lead to the largest increase in model quality by applying a samples-selection algorithm].
Col 25 line 5-28, Data item labeler 102, data fetcher 108, AI engine 110, AI model 114, portal 112, browser 118, data item labeler 200, data fetcher 208, AI engine 210, K-fold validator 216, AI model 214, evaluation model(s) 214a, classifier model(s) 214b, training evaluator 246, matrix generator 218, confusion matrix 228, matrix analyzer 220, action recommender 222, portal 212, confusion matrix 300, confusion matrix 400 (and/or any of the components described therein), and/or flowcharts 500, 900, 1000, 1100, may be implemented in hardware, or hardware combined with one or both of software and/or firmware. For example, data item labeler 102, data fetcher 108, AI engine 110, AI model 114, portal 112, browser 118, data item labeler 200, data fetcher 208, AI engine 210, K-fold validator 216, AI model 214, evaluation model(s) 214a, classifier model(s) 214b, training evaluator 246, matrix generator 218, confusion matrix 228, matrix analyzer 220, action recommender 222, portal 212, confusion matrix 300, confusion matrix 400 (and/or any of the components described therein), and/or flowcharts 500, 900, 1000, 1100 may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer readable storage medium [using computational capability comprising a processor and a memory of at least one of a processing system and a graphics processing unit]);
quantifying an informativeness score of data elements in the (Elisha Col 11 line 2-9, The accuracy and stability (e.g., standard deviation) may be presented to a user via portal 212. After training (e.g., and validation), a user may select 'Train' again, for example, to train a new model with different categories and configuration or re-train the same model. Accuracy may be improved by re-training an AI model, where retraining may occur by user request and/or automatically ( e.g., after a period of time or use of the model).
Col 20 line 11-18, In an example, a removal procedure may begin, for example, with suspected outliers furthest from category mean (e.g., sample!) and may continue removing adjacent samples in the list until a desired objective is achieved, such as predictive accuracy for a category, a category mean score, a number or percentage of outliers permissible in a training set (e.g., which may be specified by a user via portal 212).
Col 24 line 30-67 and col 23 1-14, In step 912, a training sample (e.g., the first training sample) may be identified as a suspect sample based on one of: a determination that the first variance exceeds a first threshold; or a determination that the second variance confidence level exceeds a second threshold. For example, as shown in FIG. 2, training evaluator 246 may identify one or more training samples as suspect training samples if (i) the vector space variance of a training sample exceeds a first threshold or (ii) a predicted category varies from an assigned category for a training sample ( e.g., and if a confidence level exceeds a second threshold).
In step 914, a training sample (e.g., the first training sample) may be identified as an erroneous sample based on both: a determination that the first variance exceeds a first threshold; or a determination that the second variance confidence level exceeds a second threshold. For example, as shown in FIG. 2, training evaluator 246 may identify one or more training samples as erroneous training samples if (i) the vector space variance of a training sample exceeds a first threshold and (ii) a predicted category varies from an assigned category for a training sample (e.g., and if a confidence level exceeds a second threshold).
In step 920, prediction accuracy of the first ML model may be improved by training the first ML model with the revised training set instead of the training set. For example, as shown in FIG. 2, classifier model(s) 214b may be trained by training data 204 that has been revised by removing erroneous and/or suspect training samples [quantifying an informativeness score of data elements in the substantial dataset]);
automatically determining which data elements of the having an informativeness score satisfying a selection criterion t(Elisha Col 15 line 30-39, Action recommender 222 causes a recommendation 242 for resolving the disturbing label to be displayed to a user via portal 212 based on notification 240. For instance, recommendation 242 may be displayed via a web page rendered by browser 118, as described above with reference to FIG. 1. Recommendation 242 may specify that the user is to add additional data items to enrich the data with samples of this disturbing label. For instance, recommendation 242 may instruct the user to manually classify more data items that belong to this category.
Col 18 line 57-67 and col 19 1-7, In an example, AI engine 210 may generate a relatively low category (e.g., label) score for labels with training samples having a relatively high variance. Categories with relatively low scores may be recommended for refinement or elimination. AI model 214 may be retrained, for example, with a refined category. In accordance with an embodiment, AI engine 210 may provide a notification to action recommender 222 which causes a notification to be displayed to a user via portal 212. The notification may enable a user to require use of the category, refine the category (e.g., by relabeling samples in the category) or eliminate the category from training AI model 214. In the event the user decides the label is not to be discarded, a recommendation may be presented to the user to refine the label (e.g., split the label into one or more different categories, which may involve relabeling training samples) [automatically determining which data elements of the substantial dataset are in need of human annotation].
Col 19 line 6-24, Potentially inaccurately tagged messages may ( e.g., also) be identified and managed, for example, based on a distance ---'> of a ( e.g., each) vectorized message ( e.g., sample xik) from a category mean x. An identification procedure may be iterative, e.g., with multiple training and validation cycles that may use different samples as training and validation sets. Samples tagged with a category that are relatively ( e.g., very) far away from the category mean may be outliers ( e.g., inaccurately categorized, such as by human errors in categorization) or accurately categorized, but possibly misinterpreted by AI engine 210. Accurately and inaccurately categorized messages may be distinguished, for example, using a method based on a (e.g., specifically) trained generalization AI model ( e.g., as opposed to an overfit model), such as evaluation model(s) 214. A list of predictions of samples sent for training may be created along with a probability of accurate prediction for each sample using the trained AI model. The distance of a (e.g., each) training sample from the category mean (in the category the sample 25 is tagged with) may be calculated. A threshold may be defined for a distance from a category mean. [based on a prioritization order derived from the informativeness score].
Col 17 line 33-44, In accordance with one or more embodiments, responsive to identifying the one of the plurality of ML model-generated labels as problematic, additional data items associated with the problematic label are provided to the machine learning model and the machine learning model is retrained accordingly. For example, with reference to FIG. 2, action recommender 222 may provide a recommendation 242 based on notification 240 that recommends that additional data items should be provided to AI model 214. The user may provide the additional data items to AI model 214 and cause AI engine 210 to retrain AI model 214 with the additional data items via portal 212
Col 23 line 30-35, In step 920, prediction accuracy of the first ML model may be improved by training the first ML model with the revised training set instead of the training set. For example, as shown in FIG. 2, classifier model(s) 214b may be trained by training data 204 that has been revised by removing erroneous and/or suspect training samples [wherein data having an informativeness score satisfying a selection criterion is prioritized for a human review]);
choosing a selected data based on the automatically determining which elements of the (Elisha Col 18 49-67, AI engine 210 may be configured to prioritize none or more elements in vectors
x
k
→
, for example, based on variable importance (e.g., and prior knowledge). Weights may be implemented by the weighted norm
|
|
.
|
*
according to importance. In an example, the measure of distance from each sample
x
k
→
to the mean x may be a Euclidean distance.
Col 19 line 28-36, A sample that exceeds a threshold may be evaluated further (e.g., during cross validation), such as by K-fold validator 216 and training evaluator 246. During cross validation, a trained AI model (e.g., classifier model(s) 214b) may infer a category for training samples that have been tagged with a category, but were not used to train an AI model. The inferred category may be compared to the tagged category (e.g., by K-fold validator 216), which may be 35 validated (e.g., by manual or automated review) as accurate.
In an example, AI engine 210 may generate a relatively low category (e.g., label) score for labels with training samples having a relatively high variance. Categories with relatively low scores may be recommended for refinement or elimination. AI model 214 may be retrained, for example, with a refined category. In accordance with an embodiment, AI engine 210 may provide a notification to action recommender 222 which causes a notification to be displayed to a user via portal 212. The notification may enable a user to require use of the category, refine the category (e.g., by relabeling samples in the category) or eliminate the category [choosing a selected data].
Col 19 line 56-59, FIG. 7 shows an example list or array of sorted training samples. In example array 700, sample! through sample are suspected outliers while sample+ 1 through sample are not suspected outliers. [based on the automatically determining which elements of the substantial dataset are in need of human annotation based on the prioritization order derived from the informativeness score].);
adjusting labels corresponding to annotated elements of the selected data in response to annotations [and an update to the estimation of competency of the expert] ((Elisha Col 11 line 2-9, The accuracy and stability (e.g., standard deviation) may be presented to a user via portal 212. After training (e.g., and validation), a user may select 'Train' again, for example, to train a new model with different categories and configuration or re-train the same model. Accuracy may be improved by re-training an AI model, where retraining may occur by user request and/or automatically ( e.g., after a period of time or use of the model).
Col 18 line 65 -67 and col 19 line 1, The notification may enable a user to require use of the category, refine the category (e.g., by relabeling samples in the category) or eliminate the category from training AI model 214.
Col 24 line 6-11, In step 1008, an indication may be received from a use to use or to modify the first category. For example, as shown in FIG. 2, a user may provide an input through portal 212 to indicate whether the suspect category should be used as is, modified or removed.) [adjusting labels corresponding to annotated elements of the selected data in response to annotations].);
determining that a confidence in the selected data satisfies a required level by examining at least one of the updated competencies of the expert, consensus among multiple experts, and analysis through a machine learning model (Elisha Col 10 line 63-67, AI engine 210 may evaluate training data 204 (e.g., training set 226 and validation set 224) using evaluation model(s) 214a. AI engine 210 may train classifier model(s) 214b using training set 226. AI engine 210 may provide as input to AI model 214 validation set 224 to test the classification accuracy of the trained version of classifier model(s) 214b. The accuracy and stability (e.g., standard deviation) may be presented to a user via portal 212. After training (e.g., and validation), a user may select 'Train' again, for example, to train a new model with different categories and configuration or re-train the same model. Accuracy may be improved by re-training an AI model, where retraining may occur by user request and/or automatically ( e.g., after a period of time or use of the model [determining that a confidence in the selected data]
Col 29 line 56 , FIG. 7 shows an example list or array of sorted training samples. In example array 700, sample! through sample are suspected outliers while sample+ 1 through sample are not suspected outliers. The list may be constructed, for example
Col 19 28-36, A sample that exceeds a threshold may be evaluated further (e.g., during cross validation), such as by K-fold validator 216 and training evaluator 246. During cross- validation, a trained AI model (e.g., classifier model(s) 214b) may infer a category for training samples that have been tagged with a category, but were not used to train an AI model. The inferred category may be compared to the tagged category (e.g., by K-fold validator 216), which may be validated (e.g., by manual or automated review) as accurate.
Col 20 line 5-18, Sorting may produce an array based on one or more sorting procedures. AI engine 210 may (e.g., based on none or more inputs by a user, such as a number or percentage of allowable suspected outliers) selectively remove samples (e.g., in order) in a sorted list or array.
In an example, a removal procedure may begin, for example, with suspected outliers furthest from category mean (e.g., sample!) and may continue removing adjacent samples in the list until a desired objective is achieved, such as predictive accuracy for a category, a category mean score, a number or percentage of outliers permissible in a training set (e.g., which may be specified by a user via portal 212)[satisfies a required level by examining at least one] [and analysis through a machine learning model]);
and re-generating the informativeness score for the substantial dataset to generate an input data for at least one of a training of the machine learning model, a retraining of the machine learning model, and a business intelligence report of an artificial intelligence application (Elisha Col 10 line 63-67, AI engine 210 may evaluate training data 204 (e.g., training set 226 and validation set 224) using evaluation model(s) 214a. AI engine 210 may train classifier model(s) 214b using training set 226. AI engine 210 may provide as input to AI model 214 validation set 224 to test the classification accuracy of the trained version of classifier model(s) 214b. The accuracy and stability (e.g., standard deviation) may be presented to a user via portal 212. After training (e.g., and validation), a user may select 'Train' again, for example, to train a new model with different categories and configuration or re-train the same model. Accuracy may be improved by re-training an AI model, where retraining may occur by user request and/or automatically ( e.g., after a period of time or use of the model [at least one of a training of the machine learning model, a retraining of the machine learning model,]).
Col 20 line 11-18, In an example, a removal procedure may begin, for example, with suspected outliers furthest from category mean (e.g., sample!) and may continue removing adjacent samples in the list until a desired objective is achieved, such as predictive accuracy for a category, a category mean score, a number or percentage of outliers permissible in a training set (e.g., which may be specified by a user via portal 212) [and re-generating the informativeness score for the substantial dataset to generate an input data].).
Elisha does not explicitly teach matching the selected data to an expert based on at least one of a competency and a preference of the expert;
generating an annotation view of the selected data tailored to at least one of the preference and the competency of the expert through which the expert is able to annotate the selected data;
adjusting an estimation of competency of the expert based on annotations obtained from the expert, wherein the annotations to match an expected unified label based on an artificial intelligence algorithm comprising at least one of a weighted voting algorithm, a multiple machine learning models voting algorithm, and an expectation maximization algorithm;
[adjusting labels corresponding to annotated elements of the selected data in response to annotations] and an update to the estimation of competency of the expert;
wherein the re-generating of the informativeness score is based on the updated estimation of competency of the expert and the adjusted labels, such that a subsequent prioritization order for a subsequent human annotation is derived from the re-generated informativeness score.
However Zhu teaches matching the selected data to an expert based on at least one of a competency and a preference of the expert (Zhu Fig. 4,
PNG
media_image10.png
691
675
media_image10.png
Greyscale
para 0034, In this example, TDSs 304a to 304z have been subject to some pre-screening so that it is known with a decent level of confidence that each TDS includes substantive data in the form of a graphic image of one of the following six (6) animals: (A) snake; (B) cat; (C) house fly; (D) worm; (E) eel; and (F) grub. This is shown in the topmost section of screenshot 400 of FIG. 4. Alternatively, a TDS may contain other kinds of substantive data that is subject to labeling and/or relabeling. As may be discussed in the following subsection of this Detailed Description section, some embodiments of the present invention may be applied to selecting the best answers for other types of questions that are not related to training data sets for machine learning.
Para 0040, Processing proceeds to operation S280, where acceptance mod 312 outputs the accepted subset of candidate labels for further processing by a human or by some type of software algorithm. In this example, the accepted subset of candidate labels are sent through communication network 114 to client subsystem 112 which is used by a human expert. Because of the selective culling of the rejected labels, the human expert has fewer candidate labels to concern herself with when labelling. This can save time and or increase the accuracy of the results from the expert.
Para 0041 As shown in the bottom section of screenshot 400, the expert in this example is Dr. Smart who has determined that the optimal label for the image of TDS 304a is "eel." Alternatively, multiple experts may be consulted [generating an annotation view of the selected data tailored]. Alternatively, there may be several iterative rounds of sending progressively culled sets of candidate labels to sets of experts and/or nonexperts. It is also noted that, in this example, the calling of the rejected candidate labels helps determine to which expert the accepted subset of candidate labels is sent. Dr. Smart is an expert on worms and eels which were two (2) of the choices prominently present in the accepted subset of candidate labels. If the cat and housefly candidate labels had not been culled out by the polling of the nonexperts, then acceptance mod 312 might have sent the accepted subset of candidate labels to Dr. Respected, instead of Dr. Smart, because Dr. Respected, is an expert on cats and house flies [to at least one of the preference and the competency of the expert through which the expert is able to annotate the selected data;] (i.e. the expert only receives data and labels associated to their expertise).);
generating an annotation view of the selected data tailored to at least one of the preference and the competency of the expert through which the expert is able to annotate the selected data (Zhu Fig. 4,
PNG
media_image10.png
691
675
media_image10.png
Greyscale
para 0034, In this example, TDSs 304a to 304z have been subject to some pre-screening so that it is known with a decent level of confidence that each TDS includes substantive data in the form of a graphic image of one of the following six (6) animals: (A) snake; (B) cat; (C) house fly; (D) worm; (E) eel; and (F) grub. This is shown in the topmost section of screenshot 400 of FIG. 4. Alternatively, a TDS may contain other kinds of substantive data that is subject to labeling and/or relabeling. As may be discussed in the following subsection of this Detailed Description section, some embodiments of the present invention may be applied to selecting the best answers for other types of questions that are not related to training data sets for machine learning.
Para 0040, Processing proceeds to operation S280, where acceptance mod 312 outputs the accepted subset of candidate labels for further processing by a human or by some type of software algorithm. In this example, the accepted subset of candidate labels are sent through communication network 114 to client subsystem 112 which is used by a human expert. Because of the selective culling of the rejected labels, the human expert has fewer candidate labels to concern herself with when labelling. This can save time and or increase the accuracy of the results from the expert.
Para 0041 As shown in the bottom section of screenshot 400, the expert in this example is Dr. Smart who has determined that the optimal label for the image of TDS 304a is "eel." Alternatively, multiple experts may be consulted [generating an annotation view of the selected data tailored]. Alternatively, there may be several iterative rounds of sending progressively culled sets of candidate labels to sets of experts and/or nonexperts. It is also noted that, in this example, the calling of the rejected candidate labels helps determine to which expert the accepted subset of candidate labels is sent. Dr. Smart is an expert on worms and eels which were two (2) of the choices prominently present in the accepted subset of candidate labels. If the cat and housefly candidate labels had not been culled out by the polling of the nonexperts, then acceptance mod 312 might have sent the accepted subset of candidate labels to Dr. Respected, instead of Dr. Smart, because Dr. Respected, is an expert on cats and house flies [to at least one of the preference and the competency of the expert through which the expert is able to annotate the selected data;] (i.e. the expert only receives data and labels associated to their expertise).);
Elisha and Zhu are considered to be analogous to the claim invention because they are in the same field of machine learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to have modified Elisha to incorporate the teachings of Zhu of matching data to an expert. Doing so to determine the correct annotation when the correct label when uncertain (Zhu para 0074, As shown in flow chart of FIG. 5, crowd workers are recruited to filter out incorrect options (Filter step, see operation S512) and to select the correct option by comparing examples (Compare step, see operation S516). Tasks that crowd workers are not confident in will be sent to experts (Fix step, see operation S518) In some embodiments, SRLSM generates SRL annotations with very high correctness (95%) while sending only 13% of tasks to experts. Moreover, some embodiments also identify previously undiscovered incorrect "gold" annotations (3%) that existed in a certain database of training data sets. As those of skill in the art will understand, gold annotations are typically considered to be highly reliable and not subject to much error.).
Welinder teaches adjusting an estimation of competency of the expert based on annotations obtained from the expert, wherein the annotations to match an expected unified label based on an artificial intelligence algorithm comprising at least one of a weighted voting algorithm, a multiple machine learning models voting algorithm, and an expectation maximization algorithm (Welinder Page 2 3. Modeling Annotators and Labels para 1
We assume that each image i has an unknown “target value” which we denote by
z
i
. This may be a continuous or discrete scalar or vector. The set of all N images, indexed by image number, is
I
=
{
1
,
…
,
N
}
, and the set of corresponding target values is abbreviated
z
=
z
i
i
=
1
N
. The reliability or expertise of annotator j is described by a vector of parameters,
α
j
. For example, it can be scalar,
a
j
=
a
j
, such as the probability that the annotator provides a correct label; specific annotator parameterizations are discussed in Section 5. There are M annotators in total,
A
=
{
1
,
…
,
M
}
, and the set of their parameter vectors is
a
=
a
j
j
=
1
M
. Each annotator j provides labels
L
i
=
l
i
j
j
∈
I
j
for all or a subset of the images,
PNG
media_image11.png
30
68
media_image11.png
Greyscale
. Likewise, each image i has labels
PNG
media_image12.png
29
122
media_image12.png
Greyscale
provided by a subset of the annotators
PNG
media_image13.png
23
67
media_image13.png
Greyscale
. The set of all labels is denoted L. For simplicity, we will assume that the labels lid belong to the same set as the underlying target values zi; this assumption could, in principle, be relaxed.
Page 3, E-step: Assuming that we have a current estimate
a
^
of the annotator parameters, we compute the posterior on the target values:
PNG
media_image14.png
180
466
media_image14.png
Greyscale
PNG
media_image15.png
440
474
media_image15.png
Greyscale
Page 3 4. Online Estimation The factorized form of the general model in (1) allows for an online implementation of the EM-algorithm. Instead of asking for a fixed number of labels per image, the online algorithm actively asks for labels only for images where the target value is still uncertain. Furthermore, it finds and prioritizes expert annotators and blocks sloppy annotators online. The algorithm is outlined in Figure 4 and discussed in detail in the following paragraphs. [wherein the annotations to match an expected unified label based on an artificial intelligence algorithm comprising at least one] [an expectation maximization algorithm]
Page 4 4. Online Estimation para 4 line 1-8 and 5, Annotator evaluation step: Since posteriors on the image target values ^p(zi) are computed in the label collection step, the annotator parameters can be estimated in the same manner as in the M-step in the EM-algorithm, by maximizing Q(
a
j
,
a
^
j
) in (7). Annotator j is put in either E or B if
a measure of the variance of a is below a threshold,
PNG
media_image16.png
43
296
media_image16.png
Greyscale
where
θ
v
is the threshold on the variance. If the variance is above the threshold we do not have enough evidence to consider the annotator to be an expert or a bot (unreliable annotator).
On Murk the expert- and bot-lists can be implemented by using “qualifications”. A qualification is simply a pair of two numbers, a unique qualification id number and a scalar qualification score, that can be applied to any worker. The qualifications can then be used to restrict (by inclusion or exclusion) which workers are allowed to work on a particular task. [adjusting an estimation of competency of the expert based on annotations obtained from the expert].);
[adjusting labels corresponding to annotated elements of the selected data in response to annotations] and an update to the estimation of competency of the expert (Welinder Page 4 4. Online Estimation para 4 line 1-8 and 5, Annotator evaluation step: Since posteriors on the image target values ^p(zi) are computed in the label collection step, the annotator parameters can be estimated in the same manner as in the M-step in the EM-algorithm, by maximizing Q(
a
j
,
a
^
j
) in (7). Annotator j is put in either E or B if
a measure of the variance of a is below a threshold,
PNG
media_image16.png
43
296
media_image16.png
Greyscale
where
θ
v
is the threshold on the variance. If the variance is above the threshold we do not have enough evidence to consider the annotator to be an expert or a bot (unreliable annotator).
On Murk the expert- and bot-lists can be implemented by using “qualifications”. A qualification is simply a pair of two numbers, a unique qualification id number and a scalar qualification score, that can be applied to any worker. The qualifications can then be used to restrict (by inclusion or exclusion) which workers are allowed to work on a particular task [and an update to the estimation of competency of the expert].));
[wherein the re-generating of the informativeness score] is based on the updated estimation of competency of the expert and the adjusted labels, [such that a subsequent prioritization order for a subsequent human annotation is derived from the re-generated informativeness score] (Welinder page 3. 4. Online estimation, The factorized form of the general model in (1) allows for an online implementation of the EM-algorithm. Instead of asking for a fixed number of labels per image, the online algorithm actively asks for labels only for images where the target value is still uncertain. Furthermore, it finds and prioritizes expert annotators and blocks sloppy annotators online. The algorithm is outlined in Figure 4 and discussed in detail in the following paragraphs [is based on the updated estimation of competency of the expert and the adjusted labels]).
Elisha and Welinder are considered to be analogous to the claim invention because they are in the same field of machine learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to have modified Elisha to incorporate the teachings of Welinder of and determine competency of an expert as they provide labels. Doing so to block sloppy annotators and prioritize experts (Welinder page 2 4. Online Estimation para 1, The factorized form of the general model in (1) allows for an online implementation of the EM-algorithm. Instead of asking for a fixed number of labels per image, the online algorithm actively asks for labels only for images where the target value is still uncertain. Furthermore, it finds and prioritizes expert annotators and blocks sloppy annotators online. The algorithm is outlined in Figure 4 and discussed in detail in the following paragraphs.).
Further Zhdanov teaches wherein the re-generating of the informativeness score [is based on the updated estimation of competency of the expert and the adjusted labels], such that a subsequent prioritization order for a subsequent human annotation is derived from the re-generated informativeness score (Zhdanov Col 5 line 19-23, ALS 112 can perform active learning for unlabeled or partially unlabeled datasets and use machine learning to 20 evaluate unlabeled raw datasets and provide input into the data labeling process by identifying a subset of the input data to be labeled by manual labelers.
Col 6 line 33-45, In some embodiments, the labeled subset of the input dataset can be used to train the active learning service model. As shown at numeral 11, the labeled subset of the input dataset can be provided to the machine annotation service 114. The machine annotation service 114 may include a training service that can generate a new model, or update the previously used model, using the labeled subset of the input dataset. In some embodiments, a separate training service (not shown) may obtain the labeled subset of the input dataset from the annotation consolidation service and may use the labeled subset of the input dataset to further train the model maintained by the machine annotation service 114. The above described process may then be repeated using the updated model. For example, if the updated model has converged, then the remainder of the input dataset can be accurately identified. If the updated model has not converged, then a new subset of the input dataset can be identified for further labeling according to the process described above. In some embodiments, the data labeling service 108 can output one or more of the converged model or the labeled dataset, as described further below.
Col 10 line 32-39, The threshold can then be determined based on a user's choice of acceptable accuracy [wherein the re-generating of the informativeness score]. Features identified with a confidence score with a corresponding accuracy above the threshold can be auto-labeled 404 with the labels generated by the machine learning model 406. The resulting annotations can be stored in label store 412. Features identified with a confidence score below a threshold may be sent by active learning service 402 to be manually labeled 408 [such that a subsequent prioritization order for a subsequent human annotation is derived from the re-generated informativeness score.]).
Elisha and Zhdanov are considered to be analogous to the claim invention because they are in the same field of machine learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to have modified Elisha to incorporate the teachings of Zhdanov and to re-generate an informative score. Doing so to determine accuracy that fall below a threshold to be sent to active learning or manually labeled (Zhdanov Col 10 line 32-39, The threshold can then be determined based on a user's choice of acceptable accuracy. Features identified with a confidence score with a corresponding accuracy above the threshold can be auto-labeled 404 with the labels generated by the machine learning model 406. The resulting annotations can be stored in label store 412. Features identified with a confidence score below a threshold may be sent by active learning service 402 to be manually labeled 408).
Regarding claim 2, Elisha in view of Zhu, Welinder and Zhdanov teach the method of claim 1.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Elisha further teaches further comprising: determining which objects from the (Elisha Col 23 36-51 FIG. 10 shows a flowchart of a method 1000 for improving prediction accuracy of an ML model by identifying suspect categories, according to an example embodiment, according to an example embodiment. Embodiments disclosed herein and other embodiments may operate in accordance with example method 1000. Method 1000 comprises steps 1002-1008. However, other embodiments may operate according to other methods. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the foregoing discussion of embodiments. No order of steps is required unless expressly indicated or inherently required. There is no requirement that a method embodiment implement all of the steps illustrated in FIG. 10. FIG. 10 is simply one of many possible embodiments. Embodiments may implement fewer, more or different steps.
Col 23 line 66-67 and Col 24 line1-5, In step 1006, an indication may be requested (e.g., from a user) whether the first category should be used or modified based on the identification as a suspect category. For example, as shown in FIG. 2, training evaluator 246 may cause portal 212 to display to a user and request user input to indicate whether the suspect category should be used as is, modified or removed.),
Welinder further teaches a score of annotation being trustworthy based on human annotators competence estimation on the examined sample; a measure of conformance to distribution learned by machine learning model; a difficulty score based on representation of the data element (Welinder Page 4 4. Online Estimation para 4 line 1-8 and 5, Annotator evaluation step: Since posteriors on the image target values ^p(zi) are computed in the label collection step, the annotator parameters can be estimated in the same manner as in the M-step in the EM-algorithm, by maximizing Q(
a
j
,
a
^
j
) in (7). Annotator j is put in either E or B if a measure of the variance of aj is below a threshold,
PNG
media_image16.png
43
296
media_image16.png
Greyscale
where
θ
v
is the threshold on the variance. If the variance is above the threshold we do not have enough evidence to consider the annotator to be an expert or a bot (unreliable annotator).
If the variance is above the threshold we do not have enough evidence to consider the annotator to be an expert or a bot (unreliable annotator). If the variance is below the threshold, we place the annotator in E if a satisfies some expert criterion based on the annotation type, otherwise the annotator will be placed in B and excluded labeling in the next iteration.
On Murk the expert- and bot-lists can be implemented by using “qualifications”. A qualification is simply a pair of two numbers, a unique qualification id number and a scalar qualification score, that can be applied to any worker. The qualifications can then be used to restrict (by inclusion or exclusion) which workers are allowed to work on a particular task.
Page 6,
PNG
media_image17.png
408
970
media_image17.png
Greyscale
[score of annotation being trustworthy based on human annotators competence estimation on the examined sample;]).
Regarding claim 3, Elisha in view of Zhu, Welinder and Zhdanov teach the method of claim 2.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Welinder further teaches further comprising: expanding a database with annotations assigned by humans to data elements in a manner that the same element may be annotated by multiple human annotators (Page 2 Figure 2,
PNG
media_image18.png
519
470
media_image18.png
Greyscale
[in a manner that the same element may be annotated by multiple human annotators]
Page 6 7. Experiments and Discussion para 3, Annotator accuracy: Figure 6 shows how the accuracy of Murk annotators varies with the number of images they label for different annotation types. For the Presence-1 dataset, the few annotators that labeled most of the available images had very different DJ . For Attributes-1, on the other hand, the annotators that labeled most images have very similar a . In the case of the bounding box annotations, most annotators provided good labels, except for no. 53 and 58. These two annotators were also the only ones to label all available images. In all three subplots of Figure 6, most workers provide only a few labels, and only some very active annotators label more than 100 images. Our findings in this figure are very similar to the results presented in Figure 6 of [7] [expanding a database with annotations assigned by humans to data elements]).
Regarding claim 4, Elisha in view of Zhu, Welinder and Zhdanov teach the method of claim 1.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Zhu teaches further comprising applying each operation of the method to an annotation verification task to audit whether annotations of other humans are accurate (Zhu para 0073, At operation S518, a list of tasks that will require expert input is generated and stored. This list includes both the original options and the options that remain after the filtering of operations S512 and/or S516. The experts are receiving the original options because, for difficult cases, the workers may have removed the wrong options. Some embodiments will highlight the ones chosen by the crowd along with all other options, so that the work done by the crowd is still taken into consideration.
Para 0074, As shown in flow chart of FIG. 5, crowd workers are recruited to filter out incorrect options (Filter step, see operation S512) and to select the correct option by comparing examples (Compare step, see operation S516). Tasks that crowd workers are not confident in will be sent to experts (Fix step, see operation S518) [applying each operation of the method to an annotation verification task]
Para 0075, Some embodiments may involve analysis of workflow and task design options affecting crowd-expert curation of SRL classifier outputs, including expert involvement ratio, filtering out incorrect options, and notifying when workers' answers are different from the classifiers [to audit whether annotations of other humans are accurate.]).
Regarding claim 5 and analogous claims 16 and 22, Elisha in view of Zhu, Welinder and Zhdanov teach the method of claim 4.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Zhu further comprising:
enabling other humans to perform the annotation verification task to audit whether annotations of other humans are accurate;
comparing annotations of the human and an another human with each other (Zhu para 0004, According to an aspect of the present invention, there is a method, computer program product and/or system, for selecting among a plurality of candidate answers by a plurality of human nonexpert annotators and at least one expert annotator, that performs the following operations (not necessarily in the following order): (i) filtering out one or more rejected candidate answers from the plurality of candidate answers based upon the one or more rejected candidate answers being chosen relatively infrequently as a correct answer by the plurality of human nonexpert annotators to obtain a reduced subset of candidate answers; and (ii) selecting, from the reduced subset of candidate answers, an optimal answer based upon responses received from at least one of the following: the plurality of human nonexpert annotators and the at least one human expert annotator.
Para 0064, As shown in the flowchart of FIG. 5, an embodiment of a method that employs the Filter-Compare-Fix approach for SRL classifier output curation includes the following operations: S502; S504; S506; S508; S510; S512; S514; S516; and S518. Each of these operations will respectively be explained in the following nine (9) paragraphs.
Para 0071, At operation S514, the list of tasks is stored, but with a reduced set of options by virtue of the fact that some options were eliminated in the first round of crowdsource
questioning at operation S512.
Para 0072, At operation S516, all remaining options are provided to the workers who will then select a correct option from them. For some data sets to be identified, the filtering of operation S516 will leave only one answer, which means that the data set has been positively identified with some level of confidence by the second round of crowdsource questioning. For example, this round might resolve, and effectively label, 61 % of the original tasks, meaning that only 13% would still require the expert examination of operation S518.
Para 0073, At operation S518, a list of tasks that will require expert input is generated and stored. This list includes both the original options and the options that remain after the filtering of operations S512 and/or S516. The experts are receiving the original options because, for difficult cases, the workers may have removed the wrong options. Some embodiments will highlight the ones chosen by the crowd along with all other options, so that the work done by the crowd is still taken into consideration [comparing annotations of the human and an another human with each other;]
Para 0074 line 1-6, [0074] As shown in flow chart of FIG. 5, crowd workers are recruited to filter out incorrect options (Filter step, see operation S512) and to select the correct option by comparing examples (Compare step, see operation S516). Tasks that crowd workers are not confident in will be sent to experts (Fix step, see operation S518) [enabling other humans to perform the annotation verification task to audit annotations of other humans are accurate; ]);
and generating a most-probable annotation based on at least one of:
annotations assigned by human experts and their competencies,
predictions of machine learning model trained on at least one of annotations,
and historical data,
uncertainty of trained machine learning model,
annotations generated from the annotation verification task,
predictions of a machine learning model for the annotation verification task,
and uncertainty of the machine learning model for the annotation verification task (Zhu para 0041, As shown in the bottom section of screenshot 400, the expert in this example is Dr. Smart who has determined that the optimal label for the image of TDS 304a is "eel." Alternatively, multiple experts may be consulted. Alternatively, there may be several iterative rounds of sending
progressively culled sets of candidate labels to sets of experts and/or nonexperts. It is also noted that, in this example, the calling of the rejected candidate labels helps determine to which expert the accepted subset of candidate labels is sent. Dr. Smart is an expert on worms and eels which were two (2) of the choices prominently present in the accepted subset of candidate labels. If the cat and housefly candidate labels had not been culled out by the polling of the nonexperts, then acceptance mod 312 might have sent the accepted subset of candidate labels to Dr. Respected, instead of Dr. Smart, because Dr. Respected, is an expert on cats and house flies.
Para 0073, At operation S518, a list of tasks that will require expert input is generated and stored. This list includes both the original options and the options that remain after the filtering of operations S512 and/or S516. The experts are receiving the original options because, for difficult cases, the workers may have removed the wrong options. Some embodiments will highlight the ones chosen by the crowd along with all other options, so that the work done by the crowd is still taken into consideration.
Para 0074 line 1-6, [0074] As shown in flow chart of FIG. 5, crowd workers are recruited to filter out incorrect options (Filter step, see operation S512) and to select the correct option by comparing examples (Compare step, see operation S516). Tasks that crowd workers are not confident in will be sent to experts (Fix step, see operation S518)) [and generating a most-probable annotation based on at least one of: annotations assigned by human experts and their competencies,]).
Regarding claim 6 and analogous claims 17 and 23, Elisha in view of Zhu, Welinder and Zhdanov teach the method of claim 5.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Elisha does not explicitly teach further comprising:
unifying the annotations of at least one of the human, the another human, and the annotation verifications with each other using a label unification module to create a unified subsequent dataset,
and wherein the label unification module selects a most probable set of annotations for the single piece of data which was annotated with any one of conflicting annotations and adversarial annotations.
However Zhdanov teaches further comprising:
unifying the annotations of at least one of the human, the another human, and the annotation verifications with each other using a label unification module to create a unified subsequent dataset,
and wherein the label unification module selects a most probable set of annotations for the single piece of data which was annotated with any one of conflicting annotations and adversarial annotations (Zhdanov Col 5 60-67 and Col 6 line 1-7, Once annotations have been received from the WIS 118 and optionally from machine annotation service 114, the annotations can be consolidated into labels by annotation consolidation service 122, as shown at numeral 9. Annotation consolidation may refer to the process of taking annotations from multiple annotators ( e.g., humans and/or machines) and consolidating these together (e.g., using majority-consensus heuristics, removing bias or low-quality annotators, using probabilistic distribution that minimizes a risk function for observed, predicted and true labels, or other techniques. For example, based on each annotators' accuracy history, their annotations can be weighted. If one annotator has a 50% accurate history, their annotations may have a lower weight than another annotator with a 100% accurate history.).
Col 6 line 9-16,
In some embodiments, the annotation consolidation service 122 can maintain a label score and a worker score when performing annotation consolidation scores for each piece of data in the dataset ( e.g., image, video frame, audio utterance, etc.) and current worker scores for the annotator who provided the annotations on that piece of data.
Col 6 line 18-26, The label score can be compared to a specified threshold. If the label score is higher than the threshold then no additional annotations are required. If the label score is lower than the threshold then the data may be passed to additional annotators to be further annotated by annotating service 120. At numeral 10, once the label score is higher than the threshold, then the core engine 110 can be updated to indicate that the subset of the input dataset has been labeled [wherein the label unification module selects a most probable set of annotations for the single piece of data which was annotated with any one of conflicting annotations and adversarial annotations].
Col line 29-33, The annotation consolidation service can output the labeled subset of the input dataset to an output location, as discussed further below.
PNG
media_image19.png
701
756
media_image19.png
Greyscale
[unifying the annotations of at least one of the human, the another human, and the annotation verifications with each other using a label unification module to create a unified subsequent dataset].).
Regarding claim 7 and analogous 24, Elisha in view of Zhu, Welinder and Zhdanov teach the method of claim 6.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Zhdanov further teaches further comprising: using the unified subsequent dataset as labels for a training dataset, and using composition of those as the input for training the machine learning model (Zhdanov Col 6 line 33-45, In some embodiments, the labeled subset of the input dataset can be used to train the active learning service model [using the unified subsequent dataset as labels for a training dataset,]. As shown at numeral 11, the labeled subset of the input dataset can be provided to the machine annotation service 114. The machine annotation service 114 may include a training service that can generate a new model, or update the previously used model, using the labeled subset of the input dataset. In some embodiments, a separate training service (not shown) may obtain the labeled subset of the input dataset from the annotation consolidation service and may use the labeled subset of the input dataset to further train the model maintained by the machine annotation service 114 [composition of those as the input for training the machine learning model].).
Regarding claim 8, Elisha in view of Zhu, Welinder and Zhdanov teach the method of claim 7.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Zhdanov teaches further comprising: assessing the ununified dataset with an annotation assessment module to determine at least one of: an expert competency, whether designations were applied to the annotated data in an adversarial manner, and whether a designation matching a unified label (Zhdanov Col 5 line 19-23, ALS 112 can perform active learning for unlabeled or partially unlabeled datasets and use machine learning to 20 evaluate unlabeled raw datasets and provide input into the data labeling process by identifying a subset of the input data to be labeled by manual labelers.
Col 6 line 33-45, In some embodiments, the labeled subset of the input dataset can be used to train the active learning service model. As shown at numeral 11, the labeled subset of the input dataset can be provided to the machine annotation service 114. The machine annotation service 114 may include a training service that can generate a new model, or update the previously used model, using the labeled subset of the input dataset. In some embodiments, a separate training service (not shown) may obtain the labeled subset of the input dataset from the annotation consolidation service and may use the labeled subset of the input dataset to further train the model maintained by the machine annotation service 114. The above described process may then be repeated using the updated model. For example, if the updated model has converged, then the remainder of the input dataset can be accurately identified. If the updated model has not converged, then a new subset of the input dataset can be identified for further labeling according to the process described above. In some embodiments, the data labeling service 108 can output one or more of the converged model or the labeled dataset, as described further below.
Col 10 line 36-52, The resulting annotations can be stored in label store 412. Features identified with a confidence score below a threshold may be sent by active learning service 402 to be manually labeled 408. As discussed above, manual labeling 408 can include sending a subset of the unlabeled dataset to a workforce interface service which may interface with one or more types of manual annotators. The manual annotators may return annotations for one or more features of the unlabeled dataset [assessing the ununified dataset with an annotation assessment module to determine at least one of: ]. These annotations may then be consolidated into labels by annotation consolidation service 410. For example, a feature depicted in an image may receive a plurality of annotations from a plurality of annotators. These annotations may be consolidated based on, e.g., the accuracy of the annotator and other factors, as discussed above. Once the annotations have been consolidated into labels, the labeled dataset can be output to label store 412 [whether designations were applied to the annotated data in an adversarial manner]).
Regarding claim 9, Elisha in view of Zhu, Welinder and Zhdanov teach the method of claim 8.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Zhdanov further comprising:
further comprising:
automatically balancing an exploration and an exploitation of expert competencies to optimize obtained annotation quality (Zhdanov Col 5 60-67 and Col 6 line 1-7, Once annotations have been received from the WIS 118 and optionally from machine annotation service 114, the annotations can be consolidated into labels by annotation consolidation service 122, as shown at numeral 9. Annotation consolidation may refer to the process of taking annotations from multiple annotators ( e.g., humans and/or machines) and consolidating these together (e.g., using majority-consensus heuristics, removing bias or low-quality annotators, using probabilistic distribution that minimizes a risk function for observed, predicted and true labels, or other techniques. For example, based on each annotators' accuracy history, their annotations can be weighted. If one annotator has a 50% accurate history, their annotations may have a lower weight than another annotator with a 100% accurate history.);
reaching an end condition wherein no further annotated data, subsequent annotated data, and unified subsequent dataset are inputted into the machine learning model (Zhdanov Col 6 line 18-32, The label score can be compared to a specified threshold. If the label score is higher than the threshold then no additional annotations are required. If the label score is lower than the threshold then the data may be passed to additional annotators to be further annotated by annotating service 120. At numeral 10, once the label score is higher than the threshold, then the core engine 110 can be updated to indicate that the subset of the input dataset has been labeled. The active learning loop may continue to execute with the core engine invoking the active learning service 112 to label a new subset of the input dataset that is still unlabeled or partially labeled. The annotation consolidation service can output the labeled subset of the input dataset to an output location, as discussed further below [reaching an end condition wherein no further annotated data,].
Col 6 line 33-45, In some embodiments, the labeled subset of the input dataset can be used to train the active learning service model. As shown at numeral 11, the labeled subset of the input dataset can be provided to the machine annotation service 114. The machine annotation service 114 may include a training service that can generate a new model, or update the previously used model, using the labeled subset of the input dataset. In some embodiments, a separate training service (not shown) may obtain the labeled subset of the input dataset from the annotation consolidation service and may use the labeled subset of the input dataset to further train the model maintained by the machine annotation service 114. The above described process may then be repeated using the updated model. For example, if the updated model has converged, then the remainder of the input dataset can be accurately identified. If the updated model has not converged, then a new subset of the input dataset can be identified for further labeling according to the process described above. In some embodiments, the data labeling service 108 can output one or more of the converged model or the labeled dataset, as described further below [subsequent annotated data, and unified subsequent dataset are inputted into the machine learning model.].).
Regarding claim 10, Elisha in view of Zhu, Welinder and Zhdanov teach the method of claim 9.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Welinder further teaches wherein to automatically balance the experts competency exploration of expert competencies
comprises an exploration to assess expert competencies performed by at least one of: creating a new artificial unannotated data sample, a choosing sample from the substantial dataset, choosing sample from selected data (Welinder Page 2,
PNG
media_image20.png
509
473
media_image20.png
Greyscale
] [choosing sample from the substantial dataset, choosing sample from selected data]
Page 3 4. Online Estimation The factorized form of the general model in (1) allows for an online implementation of the EM-algorithm. Instead of asking for a fixed number of labels per image, the online algorithm actively asks for labels only for images where the target value is still uncertain. Furthermore, it finds and prioritizes expert annotators and blocks sloppy annotators online. The algorithm is outlined in Figure 4 and discussed in detail in the following paragraphs.
Page 4 4. Online Estimation para 4 line 1-8 and 5, Annotator evaluation step: Since posteriors on the image target values ^p(zi) are computed in the label collection step, the annotator parameters can be estimated in the same manner as in the M-step in the EM-algorithm, by maximizing Q(
a
j
,
a
^
j
) in (7). Annotator j is put in either E or B if a measure of the variance of aj is below a threshold,
PNG
media_image16.png
43
296
media_image16.png
Greyscale
where
θ
v
is the threshold on the variance. If the variance is above the threshold we do not have enough evidence to consider the annotator to be an expert or a bot (unreliable annotator) [wherein the exploration of expert competencies comprises an exploration to assess expert competencies performed by at least one of]).
Regarding Claims 13 and 14 method are analogous to claim 1 as they a similar to claim one in function and have in combination the same limitations. Thus the same rejection for claim 1 is applicable to claims 13 and 14.
Regarding Claims 15 method is analogous to claims 3 and 4 as they a similar to claim 15 in function and have in combination the same limitations. Thus the same rejection for claim 3 and 4 is applicable to claims 15.
Regarding Claims 18 method is analogous to claims 7, 8 and 9 as they a similar to claim 18 in function and have in combination the same limitations. Thus the same rejection for claim 7, 8 and 9 is applicable to claims 18.
Regarding Claims 19 and 20 are system claims that analogous to claim 1 as they a similar to claim one in function and have in combination the same limitations. Thus the same rejection for claim 1 is applicable to claims 19 and 20.
Regarding Claims 21 system is analogous to claims 2, 3 and 4 as they a similar to claim 21 in function and have in combination the same limitations. Thus the same rejection for claim 2, 3 and 4 is applicable to claims 21.
Claim(s) 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Elisha in view of Zhu Welinder and further in view of Zhdanov and Kumaran, Rajesh. "Etl techniques for structured and unstructured data." International Research Journal of Engineering and Technology (IRJET) 8 (2021): 1727-1735 (“Kumaran”).
Regarding claim 11, Elisha in view of Zhu, Welinder and Zhdanov teach the method of claim 10.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Elisha does not explicitly teach wherein the substantial dataset is an unstructured data that is so voluminous that traditional data processing methods are unable to discreetly organize the data in a structured schema.
However Kumaran teaches wherein the substantial dataset is an unstructured data that a traditional data-processing method is unable to organize into a structure schema t(Kumaran page 1730, 5. ETL Techniques for Unstructured Data ,Unstructured data, with its diverse and often complex nature, requires specialized ETL (Extract, Transform, Load) techniques to effectively process and integrate it into usable formats. This section covers the tools, methods, and best practices for handling unstructured data, including preprocessing, transformation, scalability, and common challenges.
5.1. ETL Tools for Unstructured Data Apache Hadoop: An open-source framework designed for distributed storage and processing of large datasets across clusters of computers. Hadoop’s Hadoop Distributed File System (HDFS) provides scalable storage, while its MapReduce programming model allows for parallel processing of unstructured data.
Ideal for storing and processing vast amounts of unstructured data, such as log files, social media content, or large text corpora [wherein the substantial dataset is an unstructured data].
Page 1732, 6.2. Data Lakes vs. Data Warehouses Data Lakes: Storage Capabilities: Data lakes are designed to store vast amounts of data in its raw form, including both structured and unstructured data. They use distributed storage systems like Hadoop HDFS or cloud-based storage solutions (e.g., Amazon S3, Azure Data Lake).
Elisha and Kumaran are considered to be analogous to the claim invention because they are in the same field of data processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to have modified Elisha to incorporate the teachings of Kumaran to incorporate best practices to process unstructured data. Doing so to would allow data to be used in NLP and proper storage (Kumaran page 1727 abstract line 7-19, data, typically organized in relation databases, requires SQL-based ETL tools and schema-based transformation, while unstructured data, such as text, images, and videos, demands more flexible, AI-driven method like Natural Language Processing (NLP) and big data frameworks like Hadoop or Spark. Additionally, hybrid ETL pip are discussed, highlighting strategies to integrate both data types and ensure scalable, high-performance processing. The article concludes with best practices for managing mixed data ETL workflows, addressing challenges such as data governance, automation, and scalability, while also anticipating future trends in ETL driven by advancements in machine learning and cloud computing.).
Regarding claim 12, Elisha in view of Zhu, Welinder, Zhdanov, and Kumaran teach the method of claim 12.
Elisha, Zhu, Welinder and Zhdanov are combine in the same rational as set forth above with respect to claim 1.
Elisha and Kumaran are combine in the same rational as set forth above with respect to claim 11.
Welinder further teaches further comprising: analyzing the substantial dataset computationally to reveal at least one a pattern, a trend, and an association relating to at least one of a human behavior and a human-computer interaction (Welinder Page 5 5. Annotation Types para 5 line 4-7, The annotator behavior is assumed to be governed by a single parameter a 2 [0; 1], which is the probability that the annotator attempts to provide an honest label.
page 6 Figure 6,
PNG
media_image21.png
416
976
media_image21.png
Greyscale
[analyzing the substantial dataset computationally to reveal at least one pattern] [a human behavior] (Examiner Note: The data shows a trend of annotator 53 and 58 consistently performed worse than all other participants as shown in (c) graph.)).
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Akbik et al. (US10783328B2) teaches routing labeling task to expert for labeling.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALFREDO CAMPOS whose telephone number is (571)272-4504. The examiner can normally be reached 7:00 - 4:00 pm M - F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J. Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALFREDO CAMPOS/Examiner, Art Unit 2129
/IMAD KASSIM/Primary Examiner, Art Unit 2129
Prosecution Insights