Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/02/2025 has been entered.
Status of Claims
This action is a responsive to the application filed on 12/02/2025.
Claims 1-2, 4-6, 7-9, 11-12, 14-18, and 21-26 are pending.
Claims 1, 5, 7-8, 12, 14-15, 18, and 22 have been amended.
Claims 3, 6, 10, 13, and 19-20 have been canceled.
Claims 24-26 have been added.
Response to Arguments
Applicant’s arguments, with respect to the rejection(s) of claim(s) 1-2, 4-6, 7-9, 11-12, 14-18, and 21-26 under 35 U.S.C. 101, have been considered but they are not persuasive. The applicant argues that the claims overcome the 101 rejections, in light of the specification “paragraphs [0017] - [0026]”, since they are directed to “an improvement to at least the field of data annotation by enhancing the synergy between machine learning models and annotators in concurrent labeled data set creations”. The examiner respectfully disagrees.
The recitations of the “machine learning models” and outputting results to a user interface operations are recited at a high level and do not integrate the judicial exceptions into a practical application since the steps are mere “black-box” operations without further details of the inner workings/architecture of the algorithms and how the learning specifically affects the algorithms for outputting desired predictions; thus are maintained as generally link the use of the judicial exception to a particular technological environment or field of use and insignificant extra-solution activity. See 35 U.S.C 101 section for full, updated analysis of claim limitations necessitated by applicant amendments.
Applicant’s arguments, with respect to the rejection(s) of claim(s) 19, 26, and 33 under 35 U.S.C. 102, have been considered but they are not persuasive. Applicant argues that no reference teaches the amended limitations, since Edgar “clearly does not disclose the use of ‘a plurality of data instance selectors’ which are depicted in FIG. 6 and which at least two of the plurality of data instance selectors utilize ‘different machine learning operations’”, or “machine learning models” learning annotator preferences with continuous training. The examiner respectfully disagrees.
Due to the broadness of the claim language, Edgar has been found to teach the argued limitations. Edgar, paragraphs 0003; 0004; 0041; 0052; 0063-0064, and 0071 teach multiple types of machine learning operations for automatic data sample prioritization and annotation, and further manual operations for data sample prioritization and annotation; thus, reading on the claimed language. The models are also taught to be trained on training samples from user labeled data and historical data.
Applicant is encouraged to add clarifying amendments to the claim elements cannot be read as broadly.
See 35 U.S.C 102 section for full mapping of claim limitations necessitated by applicant amendments.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, 4-6, 7-9, 11-12, 14-18, and 21-26 are rejected under 35 USC 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1
Step 1
Claim 1 along with dependent claims are method type claims.
Step 2A Prong I
wherein at least two of the plurality of data instance selectors utilize different machine learning operations (mental process of choosing between two ML operations)
coordinating … annotation tasks between one or more annotators … based on one or more annotator preferences …; (This is directed to an abstract idea of a human performing the steps mentally with the aid of a pen and paper; recites a mental process. The coordination of tasks based on preferences may be performed manually by a human.)
generating…one or more annotation strategies based on the learning of the one or more annotator preferences and the data annotation requirements for annotating an unlabeled data set in one or more queues; (This is directed to an abstract idea of a human performing the steps mentally with the aid of a pen and paper; recites a mental process. A human could generate strategies using their own judgement, taking into account preferences and requirements.)
…wherein each one of the plurality of queues are generated based on the annotation strategy that applies the one or more annotator preferences. (This is directed to an abstract idea of a human performing the steps mentally with the aid of a pen and paper; recites a mental process. A human could generate a queue with the aid of a pen and paper and use their own judgement to strategize the content of the queue based on preferences and lists of annotations.)
Step 2A Prong II
receiving, by a plurality of data instance selectors, unlabeled data requiring annotation by one or more annotators (insignificant extra solution activity – data gathering),
in a user interface (field of use - generic computer implementation)
between one or more … machine learning models based on one or more … data annotation requirements of a machine learning model; (Examiner notes that his is a high-level recitation of a machine learning model. This is directed to mere instructions to apply an exception, MPEP 2106.05(f). Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea.)
learning, by the one or more machine learning models, the one or more annotator preferences and the data annotation requirements for coordinating the annotation tasks (This is a machine learning step, but as recited, it’s very high-level and doesn’t specify a particular improvement to technology or a technical solution. This is directed to mere instructions to apply an exception, MPEP 2106.05(f). Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea.)
…by the one or more machine learning models… (This is directed to field of use, MPEP 2106.05(h).)
providing, in the user interface, an unlabeled data set in a plurality of queues for the one or more annotators to perform the annotation tasks based on the one or more annotator preferences, (This is directed to insignificant extra-solution activity, MPEP 2106.05(g).)
Therefore, Claim 1 does not include additional elements, individually or in combination, that integrate the judicial exception into a practical application.
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because when considered separately and in combination, they do not add significantly more (also known as an “inventive concept”) to the exception.
The additional elements are:
receiving, by a plurality of data instance selectors, unlabeled data requiring annotation by one or more annotators (MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network, e.g., using the Internet to gather data” – Berkheimer for Well-Understood, Routine, and Conventional Activity),
in a user interface (field of use - generic computer implementation)
between one or more … machine learning models based on one or more … data annotation requirements of a machine learning model; (Examiner notes that his is a high-level recitation of a machine learning model. This is directed to mere instructions to apply an exception, MPEP 2106.05(f). Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea.)
learning, by the one or more machine learning models, the one or more annotator preferences and the data annotation requirements for coordinating the annotation tasks (This is a machine learning step, but as recited, it’s very high-level and doesn’t specify a particular improvement to technology or a technical solution. This is directed to mere instructions to apply an exception, MPEP 2106.05(f). Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea.)
…by the one or more machine learning models… (This is recited at a high-level and doesn’t specify a particular structure or operations of the model. This is directed to generally linking to a particular technological environment or field of use, MPEP 2106.05(h).)
providing, in the user interface, the unlabeled data set in each of the plurality of queues for the one or more annotators to perform the annotation tasks based on the one or more annotator preferences, (The extra solution activity identified is transmitting data, which is well-understood, routine, conventional activity in accordance with the court cases listed in 2106.05(D)(II(i), receiving or transmitting data over a network and Presenting offers and gathering statistics – for UI).
Claim 2
Step 2A Prong I
See the rejection of Claim 1 above, which Claim 2 depends on.
creating one or more annotator queues of unlabeled data. (This is directed to an abstract idea of a human performing the steps mentally with the aid of a pen and paper; recites a mental process. Creating one or more queues of unlabeled data for one or more annotators may be performed manually by a human.)
Step 2A Prong II
Claim 2 does not include additional elements, individually or in combination, that integrate the judicial exception into a practical application, as discussed here and in the rejection of Claim 1.
Step 2B
Claim 2 does not include additional elements that are sufficient to amount to significantly more than the judicial exception because when considered separately and in combination, they do not add significantly more (also known as an “inventive concept”) to the exception, as discussed above and in the rejection of Claim 1.
Claim 4
Step 2A Prong I
See the rejection of Claim 1, which Claim 4 depends on.
generating a plurality of unlabeled data sets requiring one or more of the annotation tasks; and ranking, by the one or more machine learning models, the plurality of unlabeled data sets requiring the one or more of the annotation tasks based on the one or more annotator preferences. (This is directed to an abstract idea of a human performing the steps mentally with the aid of a pen and paper; recites a mental process. Ranking a plurality of unlabeled data sets that require one or more annotation tasks based on preferences can be performed by a human using their own judgement.)
Step 2A Prong II
Claim 4 does not include additional elements, individually or in combination, that integrate the judicial exception into a practical application, as discussed here and in the rejection of Claim 1.
Step 2B
Claim 4 does not include additional elements that are sufficient to amount to significantly more than the judicial exception because when considered separately and in combination, they do not add significantly more (also known as an “inventive concept”) to the exception, as discussed above and in the rejection of Claim 1.
Claim 5
Step 2A Prong I
predicted class of instances (mental process of predicting)
Step 2A Prong II
providing, in the user interface, a queue of unlabeled data sets for the one or more annotators to perform the annotation tasks based on the one or more annotator preferences (This is directed to insignificant extra-solution activity, MPEP 2106.05(g). and field of use)
wherein each of the unlabeled data sets includes a predicted class of instances determined by the one or more machine learning models and This is directed to mere instructions to apply an exception, MPEP 2106.05(f). Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. – high level use of ML)
presented to the one or more annotators in the user interface as a system-estimated class distribution (This is directed to insignificant extra-solution activity, MPEP 2106.05(g)
These additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea as discussed above and in the rejection of Claim 1.
Step 2B
Claim 5 does not include additional elements that are sufficient to amount to significantly more than the judicial exception because when considered separately and in combination, they do not add significantly more (also known as an “inventive concept”) to the exception, as discussed above and in the rejection of Claim 1.
The additional elements are:
providing, in the user interface, a queue of unlabeled data sets for the one or more annotators to perform the annotation tasks based on the one or more annotator preferences (The extra-solution activity identified is transmitting data, which is well-understood, routine, conventional activity in accordance with the court cases listed in 2106.05(d)(II)(i), receiving or transmitting data over a network.)
wherein each of the unlabeled data sets includes a predicted class of instances determined by the one or more machine learning models and This is directed to mere instructions to apply an exception, MPEP 2106.05(f). Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. – high level use of ML)
presented to the one or more annotators in the user interface as a system-estimated class distribution (MPEP 2106.05(d)(II) – “Presenting offers and gathering statistics.” – Berkheimer for Well-Understood, Routine, and Conventional Activity)
Claim 7
Step 2A Prong I
See the rejection of Claim 1, which Claim 6 depends on.
wherein the plurality of queues are ranked. (This is directed to an abstract idea of a human performing the steps mentally with the aid of a pen and paper; recites a mental process. A human is capable of ranking a queue using their own judgement.)
Step 2A Prong II
Claim 7 does not include additional elements, individually or in combination, that integrate the judicial exception into a practical application, as discussed here in in the rejection of Claim 1.
Step 2B
Claim 7 does not include additional elements that are sufficient to amount to significantly more than the judicial exception because when considered separately and in combination, they do not add significantly more (also known as an “inventive concept”) to the exception, as discussed above and in the rejection of Claim 1.
Claim 8
Step 1
Claim 8 and its dependent claims are system type claims.
Step 2A Prong I
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation by mathematical calculation but for the recitation of generic computer components, then it falls within the “Mathematical Concepts” grouping of abstract ideas.
Independent Claim 8 recites substantially the same limitations as Claim 1, in the form of a system. The claim is also directed to performing mental processes without significantly more.
Step 2A Prong II
one or more computers with executable instructions that when executed cause the system to: (Examiner notes that this is a high-level recitation of using a computer to execute instructions. This is directed to mere instructions to apply an exception, MPEP 2106.05(f). adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea.)
Step 2B
Independent Claim 8 recites substantially the same limitations as Claim 1, in the form of a system. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
For the reasons above, Claim 8 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to the dependent claims. The additional limitations are addressed below.
The additional elements are:
a processor set: one or more computer-readable storage media: and program instructions stored on the one or more computer-readable media to cause the processor set to perform operations comprising: (Examiner notes that this is a high-level recitation of using a computer to execute instructions. This is directed to mere instructions to apply an exception, MPEP 2106.05(f). adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea.)
Claim 9
Claim 9 recites substantially the same limitations as Claim 2. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 11
Claim 11 recites substantially the same limitations as Claim 4. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 12
Claim 12 recites substantially the same limitations as Claim 5. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 14
Claim 14 recites substantially the same limitations as Claim 7. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 15
Step 1
Claim 15 and its dependent claims are rejected under 35 USC 101 because the claimed invention is directed to non-statutory subject matter. As per Claim 15, the claim limitation recites “one or more computer readable storage media, and program instructions collectively store on the one or more computer readable storage media”. However, the usage of the phrase “computer readable storage media” is broad enough to include both “non-transitory” and “transitory” media. The specification further does not limit the utilization of a non-transitory computer-usable storage media (Specification, paragraphs 00113 and 00120 both limit storage medium but not storage media). Also, extrinsic evidence suggests that computer-usable storage media covers a signal per se. Therefore, when the broadest reasonable interpretation of a claim covers a signal per se, the claim must be rejected under 35 USC 101 as covering non-statutory subject matter. See In re Nuijten, 500 F.3d 1346, 1356-57 (Fed Cir. 2007) (transitory embodiments are not directed to statutory subject matter). Therefore, Claim 15 and its dependent claims are rejected as being directed to non-statutory subject matter. A suggestion is made the Applicant to amend the claim to recite non-transitory computer readable storage media.
Step 2A Prong I
Even though Claim 15 fails Step 1, Claim 15 and its dependent claims are still shown to be rejected under abstract idea assuming Applicant addresses the above issue.
See the rejections of claims 1 and 8 for rationale.
Step 2A Prong II
Independent Claim 15 recites substantially the same limitations as Claim 1, in the form of a product. The claim is also directed to performing mental processes without significantly more.
Step 2B
Independent Claim 15 recites substantially the same limitations as Claim 1, in the form of a product. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
For the reasons above, Claim 15 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 16-19. The additional limitations are addressed below.
Claim 16
This Claim is directed to non-statutory subject matter as discussed in Claim 15, but is still shown to be rejected under abstract idea assuming Applicant addresses the issue in Claim 15. Claim 16 recites substantially the same limitations as Claim 2. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 17
This Claim is directed to non-statutory subject matter as discussed in Claim 15, but is still shown to be rejected under abstract idea assuming Applicant addresses the issue in Claim 15. Claim 17 recites substantially the same limitations as Claim 3 and Claim 4. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 18
This Claim is directed to non-statutory subject matter as discussed in Claim 15, but is still shown to be rejected under abstract idea assuming Applicant addresses the issue in Claim 15. Claim 18 recites substantially the same limitations as Claim 5. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 21
The method of claim 1, wherein at least one of the plurality of data instance selectors utilizes active learning operations (additional element of “apply it” under prong 2 and step 3B),
and wherein at least one of the plurality of data instance selectors utilizes manual selection for labeling and annotations tasks associated with the data requiring annotation (mental process – manual tasks for labeling data – prong 1).
Claim 22
The method of claim 1, wherein the learning of the one or more annotator preferences further comprises: monitoring, by a strategy learner, activities and context of the one or more annotators (mental process – manually tracking activity – prong 1);
prioritizing, in the user interface (additional element – field of use), the annotation tasks for each of the one or more annotators based on the one or more annotator preferences, the activities and the context of the one or more annotators, and time constraints of each of the one or more annotators (mental process – manually prioritization based on various factors – prong 1).
Claim 23
The method of claim 22, further comprising: storing labeled data from the annotation tasks completed by each of the one or more annotators (additional element – insignificant extra solution activity under prong 2 and Berkheimer evidence under step 2B - MPEP 2106.05(d)(II) - Storing and retrieving information in memory.); and
retraining the one or more machine learning models of the plurality of data instance selectors based on the labeled data (additional element of “apply it” under prong 2 and step 3B).
Claim 24
The method of claim 7, wherein each of the plurality of queues are ranked according to a degree of relevance associated with a set of instances (This is directed to an abstract idea of a human performing the steps mentally with the aid of a pen and paper; recites a mental process. A human is capable of ranking a queue using their own judgement when comparing how similar values are.), wherein a ranking indicates those of a plurality of dimensions that are included within an interactive representation for visualization and exploration within the user interface (additional element – field of use).
Claim 25
The method of claim 1, wherein the unlabeled data set in each of the plurality of queues includes images and text that require annotation for a machine learning model (This is directed to mere instructions to apply an exception, MPEP 2106.05(f). Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. – high level use of ML), and wherein the user interface includes label options for each of the one or more annotators for the unlabeled data (This is directed to insignificant extra-solution activity, MPEP 2106.05(g)).
Claim 26
The method of claim 1, wherein the annotation tasks for each of the one or more annotators considers at least priority, expertise, and deadlines in addition to the one or more annotator preferences (mental process – manually considering parameters, importance, knowledge level, and time limit for response – prong 1).
As shows above for the newly presented claims 21-26. This judicial exception is not integrated into a practical application. The claims do not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 4-6, 7-9, 11-12, 14-18, and 21-25 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Edgar et al. (hereinafter Edgar) (US PG-PUB 20210034920).
Claim 1
A method, by a processor, for facilitating enhanced synergy between machine learning models and annotators in a computing environment, (Edgar; Specification paragraph 0094; Edgar teaches the use of a processor for machine learning models and annotation, “a system comprising a processor … can collect … unannotated data samples … for input to a machine learning model configured to generate inferences based on the unannotated data samples….”) comprising:
receiving, by a plurality of data instance selectors, unlabeled data requiring annotation by one or more annotators, wherein at least two of the plurality of data instance selectors utilize different machine learning operations (Edgar; Specification paragraphs 0003; 0004; 0041; 0052; 0071; unannotated data samples are collected; different annotation techniques read on different machine learning operations - at least two involve different machine learning operations: supervised learning and metadata extraction; also see manual annotation for annotator);
coordinating, in a user interface (Edgar; Specification paragraphs 0052 – see rendering which suggests a UI);, annotation tasks between the one or more annotators and one or more machine learning models of the plurality of data instance selectors based on one or more annotator preferences and data annotation requirements of the one or more machine learning model; (Edgar; Specification paragraph 0051; Edgar teaches the use of different annotator based on the annotator, “… evaluate the unannotated data samples … collected in the annotation queue 114 to determine how to prioritize annotating the unannotated data samples and/or to determine the most appropriate mechanism or mechanisms for annotating each … unannotated data sample based on one or more prioritization criteria … in one implementation, the different types of annotation techniques can include a manual annotation technique, a metadata extraction annotation technique and a semi-supervised machine learning technique. With the manual annotation technique, an unannotated data sample can be manually reviewed and labeled (e.g., by a radiologist viewing and interacting with the actual medical image). With the metadata extraction annotation technique, an unannotated data sample can be automatically annotated based on machine analysis of the associated metadata (e.g., the additional, non-image-based clinical information associated with a medical image) that identifies or indicates the classification of the unannotated data sample that the machine learning model (e.g., M1) is configured to infer.” Here the examiner is interpreting the preferences of the annotator to be the equivalent of the capability of the annotator in Edgar. For example, the metadata extraction annotation technique ‘prefers’ unlabeled data that contains metadata and the manual annotation technique ‘prefers’ unlabeled data such as images that can be viewed and interacted with. Further in the Specification paragraph 0006; Edgar teaches the coordination of annotation tasks based on the requirements of a machine learning model, “In other implementations, the priority evaluation component can determine the annotation priority levels based on attributes associated with the respective unannotated data samples and correlations between the attributes and accuracy of performance of the machine learning model on previous data samples comprising the attributes.” Here it is shown that annotation tasks can be prioritized, or ranked, based on the level of accuracy the model currently has for certain attributes and prioritize attributes where the model does not perform as well. For example, if the model is failing to consistently and accurately make the proper inference it is required to make, data that would improve its accuracy would then be prioritized, or ranked, higher. And 0052 and the response to arguments section above.)
learning, by the one or more machine learning models, the one or more annotator preferences and the data annotation requirements for coordinating the annotation tasks; (Edgar; Specification paragraph 0063, Edgar teaches that the preferences and requirements can be learned for the purpose of coordination, “the prioritization criteria can include predefined or learned information regarding what types of data samples are most important/relevant for annotating and/or annotating with manual annotation based on the goals and needs of the entity applying the machine learning model so as to facilitate tailoring generating of accurate training examples for tailoring the performance of the machine learning model in accordance with those goals and needs.”; see 0074, 0088 and then 0051 and 0052 and the response to arguments section above; 0090)
generating, by the one or more machine learning models, one or more annotation strategies based on the learning of the one or more annotator preferences and the data annotation requirements for annotating an unlabeled data set in one or more queues; and (Edgar; Specification paragraphs 0061-0063: “the annotation management component 204 can be configured to determine how to prioritize annotating the unannotated data samples and/or which annotation technique or techniques to apply based on the annotation priority levels associated with the respective unannotated data samples … the prioritization criteria can include predefined or learned information regarding what types of data samples are most important/relevant for annotating and/or annotating with manual annotation based on the goals and needs of the entity applying the machine learning model”. Here the examiner is interpreting generating…one or more annotation strategies as the application of Edgar’s priority evaluation component and annotation management component in the prioritization scheme as it routes unlabeled data through the pipeline in a way that specifically benefits the model data is being annotated for, and with consideration for the methods of annotation)
providing, in the user interface, the unlabeled data set in each of the plurality of queues for the one or more annotators to perform the annotation tasks based on the one or more annotator preferences, wherein each one of the plurality of queues are generated based on the annotation strategy that applies the one or more annotator preferences. (Specification paragraph 0051; Edgar teaches the use of different annotator based on the annotator preferences along with sub queues within the main queue, “…determine how to prioritize annotating the unannotated data samples and/or to determine the most appropriate mechanism or mechanisms for annotating each … unannotated data sample based on one or more prioritization criteria … in one implementation, the different types of annotation techniques can include a manual annotation technique, a metadata extraction annotation technique and a semi-supervised machine learning technique. With the manual annotation technique, an unannotated data sample can be manually reviewed and labeled (e.g., by a radiologist viewing and interacting with the actual medical image). With the metadata extraction annotation technique, an unannotated data sample can be automatically annotated based on machine analysis of the associated metadata (e.g., the additional, non-image-based clinical information associated with a medical image) that identifies or indicates the classification of the unannotated data sample that the machine learning model (e.g., M1) is configured to infer.” Here the examiner is interpreting the preferences of the annotator to be the equivalent of the capability of the annotator in Edgar. The multiple levels of prioritization within the main queue creates sub queues on a per annotator basis.; Specification paragraph 0034; Edgar teaches annotations within the main and sub queues being a basis for an annotation strategy alongside the annotator preferences. “The active learning component can also identify annotated data samples associated with low confidence levels and tag these data samples for additional review. In this regard, the active learning component can identify incorrect annotations based on association with a low confidence level (e.g., relative to a threshold confidence level). In some implementations, the active learning component can provide real-time feedback to a manual annotator identifying incorrect annotations to facilitate correcting the annotations in real-time. In other implementations, the active learning component can send annotated data samples associated with a low confidence level back to the annotation queue for re-annotating using a different annotation technique or a different entity in implementations in which the low confidence annotation was manually applied.”)
Claim 2
The method of claim 1, wherein the coordinating of the annotation tasks between the one or more annotators, further comprises: creating one or more annotator queues of the unlabeled data. (Edgar; Specification paragraph 0030; Edgar teaches the use of a queue for unlabeled data which the examiner is interpreting to be equivalent to unannotated data samples, “In this regard, in one or more embodiments, the advanced annotation pipeline can include an annotation queue that collects unannotated data samples.”)
Claim 4
The method of claim 1, further comprising: generating a plurality of unlabeled data sets requiring one or more of the annotation tasks (Edgar; Specification paragraph 0003; 0041; see unannotated data samples); and ranking, by the one or more machine learning models, the plurality of unlabeled data sets requiring the one or more of the annotation tasks based on the one or more annotator preferences. (Edgar; Specification paragraph 0051; Edgar teaches the prioritization which the examiner is interpreting as ranking, based on the annotator preferences which the examiner is interpreting as the capabilities of the annotator, “The annotation management component 204 can evaluate the unannotated data samples … collected in the annotation queue 114 to determine how to prioritize annotating the unannotated data samples and/or to determine the most appropriate mechanism or mechanisms for annotating each … unannotated data sample based on one or more prioritization criteria. In this regard, the annotation pipeline module 112 can leverage different types of annotation techniques to facilitate annotating the data samples, wherein the different types of annotation techniques can vary with respect to the amount of time and resources involved. For example, in one implementation, the different types of annotation techniques can include a manual annotation technique, a metadata extraction annotation technique and a semi-supervised machine learning technique.” The different techniques for annotation have different ‘preferences’, for example the metadata extraction annotator prefers unlabeled data that has metadata.)
Claim 5
The method of claim 1, wherein providing the unlabeled data set in each of the plurality of queues, further comprises: generating each of the plurality of queues for the unlabeled data set for the one or more annotators to perform the annotation tasks based on the one or more annotator preferences, (Edgar; Specification paragraph 0051; Edgar teaches the queue of unlabeled data for the annotation tasks performed by the annotators, “In this regard, in one or more embodiments, the annotation management component 204 can evaluate the unannotated data samples 104 included in the annotation queue 114 to determine which annotation technique or techniques from among the different annotation technique options to apply to each (or in some implementations one or more) unannotated data sample included in the annotation queue 114 based on one or more prioritization criteria.” )
wherein each of the unlabeled data sets includes a predicted class of instances determined by the one or more machine learning models and presented to the one or more annotators in the user interface as a system-estimated class distribution (Edgar; Specification paragraph 0030; 0045; 0052 - ML model predicts class (and confidence), which is presented to annotators via UI as system-estimated class/distribution).
Claim 7
The method of claim 1, wherein the plurality of queues are ranked. (Edgar; Edgar teaches the ranking of the queue datasets based on the data annotation requirements; Specification paragraph 0061: “the priority evaluation component 206 can determine a percentage of the amount annotated training data samples included in the annotated training data set 106 that correspond to the unannotated data sample … The annotation management component 204 can further prioritize annotating unannotated data samples that are associated with a lower representation percentage … over unannotated data samples that are associated with a higher representation percentage”; Specification paragraph 0063: “the annotation management component 204 can be configured to determine how to prioritize annotating the unannotated data samples and/or which annotation technique or techniques to apply based on the annotation priority levels associated with the respective unannotated data samples … the prioritization criteria can include predefined or learned information regarding what types of data samples are most important/relevant for annotating and/or annotating with manual annotation”. Here the examiner is interpreting ranking and prioritization to be equivalent methods of measurement or ordering such that data that is ranked/prioritized will be routed appropriately as it pertains to the timing or method of its annotation.)
Claim 8
Edgar teaches A system for facilitating enhanced synergy between machine learning models and annotators in a computing environment, comprising: a processor set; one or more computer-readable storage media; and program instructions stored on the one or more computer-readable media to cause the processor set to perform operations comprising (Edgar; Figure 15, Specification paragraph 0029; “The disclosed subject matter provides systems, computer-implemented methods, apparatus and/or computer program products that facilitate enhancing the efficiency and accuracy of annotating data samples …”; also see 0003; 0107; 0112)
The rest of the claim language in Claim 8 recites substantially the same limitations as Claim 1, in the form of a system comprising one or more computers with executable instructions, therefore it is rejected under the same rationale.
Claim 9
Claim 9 recites substantially the same limitations as Claim 2 therefore it is rejected under the same rationale.
Claim 11
Claim 11 recites substantially the same limitations as Claim 4 therefore it is rejected under the same rationale.
Claim 12
Claim 12 recites substantially the same limitations as Claim 5 therefore it is rejected under the same rationale.
Claim 13
Claim 13 recites substantially the same limitations as Claim 6 therefore it is rejected under the same rationale.
Claim 14
Claim 14 recites substantially the same limitations as Claim 7 therefore it is rejected under the same rationale.
Claim 15
Claim 15 recites substantially the same limitations as Claim 8 therefore it is rejected under the same rationale.
Claim 16
Claim 16 recites substantially the same limitations as Claim 2 therefore it is rejected under the same rationale.
Claim 17
Claim 17 recites substantially the same limitations as Claim 4 therefore it is rejected under the same rationales.
Claim 18
Claim 18 recites substantially the same limitations as Claim 5 therefore it is rejected under the same rationale.
Claim 19
Claim 19 recites substantially the same limitations as Claim 6 therefore it is rejected under the same rationale.
Claim 21
The method of claim 1, wherein at least one of the plurality of data instance selectors utilizes active learning operations, and wherein at least one of the plurality of data instance selectors utilizes manual selection for labeling and annotations tasks associated with the data requiring annotation (Edgar; for active learning see 0008; 0033; 0090 - a selector using active learning operations for data selection/annotation; for manila selection see 0004; 0066 – see manual annotation).
Claim 22
The method of claim 1, wherein the learning of the one or more annotator preferences further comprises: monitoring, by a strategy learner, activities and context of the one or more annotators (Edgar; 0074; 0088 - monitors annotator activity, context, and performance via accuracy tracking and feedback);
prioritizing, in the user interface, the annotation tasks for each of the one or more annotators based on the one or more annotator preferences, the activities and the context of the one or more annotators, and time constraints of each of the one or more annotators (Edgar; 0052 “generate annotation prioritization information… rendering… at a device associated with a user… The user can then choose whether to accept and implement… based on their domain knowledge.”; for time constants see 0003; 0041 - data samples are stored in a queue, implying a temporal workflow; annotation occurs as resources become available; 0052 - annotators can choose when to act, implying timing may depend on user availability).
Claim 23
The method of claim 22, further comprising: storing labeled data from the annotation tasks completed by each of the one or more annotators (Edgar; 0003; 0041 - teaches storing labeled data after annotation tasks); and
retraining the one or more machine learning models of the plurality of data instance selectors based on the labeled data (Edgar; 0003; 0041 - teaches retraining ML models using stored labeled data).
Claim 24
The method of claim 7, wherein each of the plurality of queues are ranked according to a degree of relevance associated with a set of instances, wherein a ranking indicates those of a plurality of dimensions that are included within an interactive representation for visualization and exploration within the user interface (Edgar; 0064 – teaches “priority evaluation component 206 can determine a priority level for an unannotated data sample based on the estimated degree of confidence (relevance) in the inference output that would be generated based on application of the machine learning model to the unannotated data sample. The priority evaluation component 206 can also determine the priority level based on the amount (e.g., a percentage) (alternate relevance) of training data samples (instances) included in the annotated training data set 106 (alternate instances) that correspond to the unannotated data sample”; paragraphs 0087-0088 – teach sending low confidence annotated samples to a “system administrator” or “manual annotator” for correcting the annotation).
Claim 25
The method of claim 1, wherein the unlabeled data set in each of the plurality of queues includes images and text that require annotation for a machine learning model, and wherein the user interface includes label options for each of the one or more annotators for the unlabeled data (Edgar; 0049-0051 – teaches machine learning model annotation for queued unlabeled images and the “non-image-based” data including “text”; and paragraph 0066 –“annotation component 208 can include and/or interface with a manual annotation application that presents one or more manual annotators (humans) with unannotated (or in some implementations previously annotated) data samples for annotation…that identifies or indicates a desired evaluation of the medical image (e.g., a diagnosis, a severity level, a disease or condition classification, etc.)”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Edgar et al. (hereinafter Edgar) (US PG-PUB 20210034920), in view of Xu et al (US Pub 20230033075) hereinafter Xu.
Claim 26
The method of claim 1, wherein the annotation tasks for each of the one or more annotators considers at least priority, expertise, and deadlines in addition to the one or more annotator preferences (Edgar; 0054, 0063-0064 – teach the model prioritizing amount of time (deadlines) for annotating one or more samples, determining priority for the samples, and degree of confidence in the predicted annotation (expertise)).
Edgar at least implies wherein the annotation tasks for each of the one or more annotators considers at least…deadlines in addition to the one or more annotator preferences (see mappings above); however, Xu teaches wherein the annotation tasks for each of the one or more annotators considers at least…deadlines in addition to the one or more annotator preferences (paragraphs 0050-0053 teach “a person can utilize a user interface to draw or specify a geometric shape or construct…this information (preferences) can be provided as input to a segmentation neural network 304” and for “machine learning methods…there may be tumors that are not identifiable to an annotator within time limit (deadlines), which may have a label applied such as ‘no significant finding,’”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to implement Xu’s teachings of a machine learning annotator model accounting for user input areas that are relevant and a processing time limit for execution into Edgar‘s teaching of machine learning annotation for prioritizing samples and outputting a degree of confidence in an amount of time in order to “save significant reading time” for a user and increase accuracy predictions of tumors (Xu, paragraphs 0053-0055).
Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Peccoud et al (US Pub 20210286604) teaches utilizing machine learning for assisting in laboratory prediction processes and consider project requirement data including deadlines.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CLINT MULLINAX whose telephone number is 571-272-3241. The examiner can normally be reached on Mon - Fri 8:00-4:30 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.M./Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights