DETAILED ACTION
This office action is responsive to the above identified application filed 12/20/2023. The application contains claims 1-20, all examined and rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Claim 1 is rejected under 35 USC 101 because the claimed inventions are directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
While independent claims 1, 2 and 15 are each directed to a statutory category, it recites a series of steps which appears to be directed to an abstract idea (mental process, mathematical concept).
Claims 1-20 are rejected under 35 U.S.C. § 101 because the instant application is directed to non-patentable subject matter. Specifically, the claims are directed toward at least one judicial exception without reciting additional elements that amount to significantly more than the judicial exception. The rationale for this determination is in accordance with the guidelines of USPTO, applies to all statutory categories, and is explained in detail below.
When considering subject matter eligibility under 35 U.S.C. 101, (1) it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter. If the claim does fall within one of the statutory categories, (2a) it must then be determined whether the claim is directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), and if so (2b), it must additionally be determined whether the claim is a patent-eligible application of the exception. If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim amounts to significantly more than the abstract idea itself. Examples of abstract ideas include certain methods of organizing human activities; a mental processes; and mathematical concepts, (2019 PEG)
STEP 1.
Per Step 1, the claims are determined to include process, manufacture, and machine as in independent Claims 1, 2 and 15, and in the therefrom dependent claims. Therefore, the claims are directed to a statutory eligibility category.
At step 2A, prong 1, The invention is directed to Mental Process (see Alice), As such, the claims include an abstract idea. When considering the limitations individually and as a whole the limitations directed to the abstract idea are:
“generating a first labeled dataset based on the first portion, wherein the first labeled dataset comprises a first plurality of labeled samples, and wherein the first plurality of labeled samples is generated using a first data labeling routine”, (Mental process, observation, evaluation and judgment)
“generating, using the second data labeling routine, a second labeled dataset based on the second portion, wherein the second labeled dataset comprises a second plurality of labeled samples; determining a first confidence metric for a first labeled sample from the second plurality of labeled samples; comparing the first confidence metric to a threshold confidence metric; in response to the first confidence metric not corresponding to the threshold confidence metric, assigning the first labeled sample to a third data labeling routine; in response to assigning the first labeled sample to a third data labeling routine, deleting the first labeled sample from the second plurality of labeled samples; and generating a new labeled sample for the first labeled sample based on the third data labeling routine” (Mental process, observation, evaluation and judgment)
The claim recites additional elements as
“A system for improvements to data labeling using a hybrid artificial intelligence labeling approach that is ambiguous to training data requirements, the system comprising: one or more processors; and one or more non-transitory, computer-readable mediums comprising instructions that when executed by the one or more processors causes operations” (“Using a computer as a tool to perform a mental process”, MPEP 2106.04(a)(2)(III)(C));
retrieving a first portion from an unlabeled dataset, wherein the unlabeled dataset comprises a plurality of unlabeled samples (insignificant extra-solution activity, MPEP 2106.05(g));
wherein the plurality of unlabeled samples is based on unstructured text based on linguistic inputs (data description, which is directed to generally linking the use of a judicial exception to a particular technological environment or type or source of data or field of use MPEP 2106.05(h));
wherein the first data labeling routine comprises a first artificial intelligence model (merely indicates a field of use or technological environment in which the judicial exception is performed and fails to add an inventive concept to the claims. See MPEP 2106.05(h));
wherein the first artificial intelligence model comprises a weakly supervised learning(data description, which is directed to generally linking the use of a judicial exception to a particular technological environment or type or source of data or field of use MPEP 2106.05(h));
training, using the first labeled dataset, a second artificial intelligence model for a second data labeling routine, wherein the second artificial intelligence model comprises active learning (This limitation is directed to training a system which is a high-generic computer software process of training data. This limitation does not amount to significantly more than the judicial exception, see MPEP 2106.05 (f))
retrieving a second portion from the unlabeled dataset (insignificant extra-solution activity, MPEP 2106.05(g));
wherein the third data labeling routine comprises a user input via a user interface (data description, which is directed to generally linking the use of a judicial exception to a particular technological environment or type or source of data or field of use MPEP 2106.05(h)).
This judicial exception is not integrated into a practical application. The elements are recited at a high level of generality, i.e. a generic computing system performing generic functions including generic processing of data. Accordingly the additional elements do not integrate the abstract into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Therefore the claims are directed to an abstract idea. (2019 Revised Patent Subject Matter Eligibility Guidance ("2019 PEG"). Thus, under Step 2A of the Mayo framework, the Examiner holds that the claims are directed to concepts identified as abstract.
STEP 2B.
Because the claims include one or more abstract ideas, the examiner now proceeds to Step 2B of the analysis, in which the examiner considers if the claims include individually or as an ordered combination limitations that are "significantly more" than the abstract idea itself. This includes analysis as to whether there is an improvement to either the "computer itself," "another technology," the "technical field," or significantly more than what is "well-understood, routine, or conventional" (WURC) in the related arts.
The instant application includes in Claim 1 additional steps to those deemed to be abstract idea(s).
When taken the steps individually, these steps are:
“A system for improvements to data labeling using a hybrid artificial intelligence labeling approach that is ambiguous to training data requirements, the system comprising: one or more processors; and one or more non-transitory, computer-readable mediums comprising instructions that when executed by the one or more processors causes operations” (“Using a computer as a tool to perform a mental process”, MPEP 2106.04(a)(2)(III)(C));
retrieving a first portion from an unlabeled dataset, wherein the unlabeled dataset comprises a plurality of unlabeled samples (well-understood, routine, or conventional activity, sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i));
wherein the plurality of unlabeled samples is based on unstructured text based on linguistic inputs (data description, which is directed to generally linking the use of a judicial exception to a particular technological environment or type or source of data or field of use MPEP 2106.05(h));
wherein the first data labeling routine comprises a first artificial intelligence model (merely indicates a field of use or technological environment in which the judicial exception is performed and fails to add an inventive concept to the claims. See MPEP 2106.05(h));
wherein the first artificial intelligence model comprises a weakly supervised learning(data description, which is directed to generally linking the use of a judicial exception to a particular technological environment or type or source of data or field of use MPEP 2106.05(h));
training, using the first labeled dataset, a second artificial intelligence model for a second data labeling routine, wherein the second artificial intelligence model comprises active learning (This limitation is directed to training a system which is a high-generic computer software process of training data. This limitation does not amount to significantly more than the judicial exception, see MPEP 2106.05 (f))
retrieving a second portion from the unlabeled dataset (well-understood, routine, or conventional activity, sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i));
wherein the third data labeling routine comprises a user input via a user interface (data description, which is directed to generally linking the use of a judicial exception to a particular technological environment or type or source of data or field of use MPEP 2106.05(h)).
In the instant case, Claim 1 is directed to above mentioned abstract idea. Technical functions such as receiving, and extracting are common and basic functions in computer technology. The individual limitations are recited at a high level and do not provide any specific technology or techniques to perform the functions claimed.
In addition, when the claims are taken as a whole, as an ordered combination, the combination of steps does not add "significantly more" by virtue of considering the steps as a whole, as an ordered combination. The instant application, therefore, still appears only to implement the abstract idea to the particular technological environments using what is well-understood, routine, and conventional in the related arts. The steps are still a combination made to the abstract idea. The additional steps only add to those abstract ideas using well understood and conventional functions, and the claims do not show improved ways of, for example, an unconventional non-routine functions for analyzing model operations or updating the model that could then be pointed to as being "significantly more" than the abstract ideas themselves.
Moreover, Examiner was not able to identify any "unconventional" steps, which, when considered in the ordered combination with the other steps, could have transformed the nature of the abstract idea previously identified. The instant application, therefore, still appears to only implement the abstract ideas to the particular technological environments using what is well-understood, routine, and conventional (WURC) in the related arts.
Further, note that the limitations, in the instant claims, are done by the generically
recited computing devices. The limitations are merely instructions to implement the abstract idea on a computing device that is recited in an abstract level and require no more than a generic computing devices to perform generic functions.
Claim 15 recites a “One or more non-transitory, computer-readable mediums comprising instructions that when executed by one or more processors” causes operations” configured to perform the same method as set forth in claim 1, the added element of “One or more non-transitory, computer-readable mediums comprising instructions that when executed by one or more processors causes operations” do not transform the judicial exception into a practical application because they are tantamount to a mere instruction to apply the judicial exception to a generic computer. The additional elements are also not sufficient to amount to significantly more than the judicial exception because the action of implementing the method on a general purpose computer with at least one processor and at least one memory is tantamount to a mere instruction to apply the judicial exception to a computer.
Claim 15 is therefore rejected according to the same findings and rationale as provided above.
Independent claims 2 and 15 are the same analogy and rejected using similar analysis as claim 1.
CONCLUSION
It is therefore determined that the instant application not only represents an abstract idea identified as such based on criteria defined by the Courts and on USPTO examination guidelines, but also lacks the capability to bring about "Improvements to another technology or technical field" (Alice), bring about "Improvements to the functioning of the computer itself" (Alice), "Apply the judicial exception with, or by use of, a particular machine" (Bilski), "Effect a transformation or reduction of a particular article to a different state or thing" (Diehr), "Add a specific limitation other than what is well-understood, routine and conventional in the field" (Mayo), "Add unconventional steps that confine the claim to a particular useful application" (Mayo), or contain "Other meaningful limitations beyond generally linking the use of the judicial exception to a particular technological environment" (Alice), transformed a traditionally subjective process performed by humans into a mathematically automated process executed on computers (McRO), or limitations directed to improvements in computer related technology, including claims directed to software (Enfish). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 3 disclose “assigning the first labeled sample to the third data labeling routine further comprises: in response to assigning the first labeled sample to a third data labeling routine, deleting the first labeled sample from the second plurality of labeled samples; and generating a new labeled sample for the first labeled sample based on the third data labeling routine” (Mental process), It does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claims 4 disclose “wherein retrieving the first portion from the unlabeled dataset further comprises: generating a plurality of identifiers to the plurality of unlabeled samples by assigning a respective identifier to each of the plurality of unlabeled samples; determining a sample size; and selecting, using a random number generator, a subset of the plurality of identifiers, wherein a number of the identifiers in the subset corresponds to the sample size” (Mental process). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 5 disclose “determining the sample size further comprises: determining a number of features of the second model; and selecting the sample size based on the number of features” (Mental process). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 6 disclose “determining the sample size further comprises: determining a required performance of the second model; and selecting the sample size based on the required performance” (Mental process). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 7 disclose “determining the sample size further comprises: determining a number of hyperparameters requiring training for the second model; and selecting the sample size based on the number of hyperparameters requiring training” (Mental process). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 8 disclose “determining the sample size further comprises: determining a data variability of the first labeled dataset (Mental process, Mathematical concept); and selecting the sample size based on the data variability (Mental process). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 9 disclose “wherein determining the sample size further comprises: determining respective processing power requirements for training the second model; and selecting the sample size based on the respective processing power requirements for training the second model.” (Mental process). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 10 disclose “determining the sample size further comprises: determining a validation type of the second model; and selecting the sample size based on the validation type” (Mental process). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 11 disclose “comparing the first confidence metric to the threshold confidence metric further comprises: determining available resources for the third data labeling routine; and determining the threshold confidence metric based on the available resources” (Mental process). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 12 disclose “comparing the first confidence metric to the threshold confidence metric further comprises: determining a user identifier for the third data labeling routine; and determining the threshold confidence metric based on the user identifier.” (Mental process). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 13 disclose “determining the threshold confidence metric based on the user identifier further comprises: determining a user accuracy rating attributed to the user identifier; and determining the threshold confidence metric based on the user accuracy rating” (Mental process). The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim. claims 14 “wherein assigning the first labeled sample to the third data labeling routine further comprises: generating for display, in a user interface, first metadata corresponding to the first labeled sample; and receiving a user input confirming the first labeled sample based on the first metadata” (insignificant extra-solution activity, MPEP 2106.05(g) that is well-understood, routine, or conventional activity, sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i))). It does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea.
The dependent claims which impose additional limitations also fail to claim patent eligible subject matter because the limitations cannot be considered statutory. The dependent claim(s) have been examined individually and in combination with the preceding claims, however they do not cure the deficiencies of claim 1 ; where all claims are directed to the same abstract idea, "addressing each claim of the asserted patents [is] unnecessary." Content Extraction &. Transmission LLC v, Wells Fargo Bank, Natl Ass'n, 776 F.3d 1343, 1348 (Fed. Cir. 2014). If applicant believes the dependent claims are directed towards patent eligible subject matter, they are invited to point out the specific limitations in the claim that are directed towards patent eligible subject matter. Claims for the other statutory classes are similarly analyzed.
For at least these reasons, the claimed inventions of each of dependent claims 3-14, 16-20 that are similar in scope to claims 1, 4-7 are directed or indirect to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more and are rejected under 35 USC 101.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 2-3, and 14-15 are rejected under 35 U.S.C. 102(a)(1) and 35 U.S.C. 102(a)(2) as being anticipated by Martin et al. [US 2021/0042577 A1, hereinafter Martin].
With regard to Claim 2,
Martin teach a method for improvements to data labeling using a hybrid artificial intelligence labeling approach that is ambiguous to training data requirements (¶25, “provide mechanisms that can combine machine learning based data labeling with human specialist data labeling”, ¶95), the method comprising:
retrieving a first portion from an unlabeled dataset, wherein the unlabeled dataset comprises a plurality of unlabeled samples (¶93, “selects all unlabeled records (or some specified number thereof) for a use case (records that have not yet been labeled by the ML labeler)”);
generating a first labeled dataset based on the first portion, wherein the first labeled dataset comprises a first plurality of labeled samples, and wherein the first plurality of labeled samples is generated using a first data labeling routine, and wherein the first data labeling routine uses a first model (¶93, “selects all unlabeled records (or some specified number thereof) for a use case (records that have not yet been labeled by the ML labeler) and has those labeled by the ML model 620”, ¶84, “Output labels from ML labeler 500 are the result of running a conditioned label request through a deployed ML model 501 to obtain an inferred answer”, ¶¶51-53, “Over time, the image classifier can be retrained using medical images labeled by the CDW or other training data to become increasingly accurate. As the confidence in the image classifier's accuracy increases, CDW may dynamically change to rely more heavily on the ML image classifier and reduce or eliminate human involvement in labeling training data”);
training, using the first labeled dataset, a second model for a second data labeling routine (¶121, “Results that meet the configured target confidence threshold (or other training data) can be used to retrain ML labeler”, ¶93, “Active learning record selector 630 evaluates the confidences in the results and forwards some subset of the results to the other labelers in the graph and/or an oracle labeler for augmented labeling. These records then come back as training data for the ML labeler”, ¶¶51-53, “Over time, the image classifier can be retrained using medical images labeled by the CDW or other training data to become increasingly accurate. As the confidence in the image classifier's accuracy increases, CDW may dynamically change to rely more heavily on the ML image classifier and reduce or eliminate human involvement in labeling training data”);
retrieving a second portion from the unlabeled dataset; generating, using the second data labeling routine, a second labeled dataset based on the second portion, wherein the second labeled dataset comprises a second plurality of labeled samples (¶29, ¶¶92-93, “active learning record selector 630 selects all unlabeled records (or some specified number thereof) for a use case (records that have not yet been labeled by the ML labeler) and has those labeled by the ML model 620”, ¶52, “an ML model behind an ML labeler may be retrained”);
determining a first confidence metric for a first labeled sample from the second plurality of labeled samples (¶93, “The ML model 620 evaluates its results (e.g., provides a confidence in its results)”, ¶84, “result output by ML labeler 500, includes the input label request, the inferred label, and a self-reported confidence measure”);
comparing the first confidence metric to a threshold confidence metric (¶209, “If the confidence estimate returned for the label by QMS 750 meets the confidence threshold target”, ¶210, “If the confidence estimate returned by QMS 750 does not meet the confidence threshold”); and
in response to the first confidence metric not corresponding to the threshold confidence metric, assigning the first labeled sample to a third data labeling routine ¶210, “If the confidence estimate returned by QMS 750 does not meet the confidence threshold … workflow orchestrator 710 routes the labeling request to the next labeler”, ¶25, “routing requests to human specialists when the machine learning component produces a low confidence result”).
With regard to Claim 3,
Martin teach the method of claim 2, wherein assigning the first labeled sample to the third data labeling routine further comprises:
in response to assigning the first labeled sample to a third data labeling routine, deleting the first labeled sample from the second plurality of labeled samples (¶214, “the workflow orchestrator 710 can be configured to throw away the result and retry with a different labeler instance”); and
generating a new labeled sample for the first labeled sample based on the third data labeling routine (¶214, “the workflow orchestrator 710 can be configured to throw away the result and retry with a different labeler instance”, ¶71, “a number of task templates can be defined for human labeler specialists with each task template expressing a user interface to use for presenting a labeling request to a human for labeling and receiving a label assigned by the human to the labeling request. Task UI configuration 412 can specify which template to use and the labeling options to be made available in the task UI”, ¶72, “the configured browser-based task UI 420, then accepts the task result from the specialist and validates it before sending it back to the labeler”, ¶68, “A human labeler acts as a gateway to a human specialist workforce”).
With regard to Claim 14,
Martin teach the method of claim 2, wherein assigning the first labeled sample to the third data labeling routine further comprises: generating for display, in a user interface, first metadata corresponding to the first labeled sample (¶71, “ a number of task templates can be defined for human labeler specialists with each task template expressing a user interface to use for presenting a labeling request to a human for labeling and receiving a label assigned by the human to the labeling request. Task UI configuration 412 can specify which template to use and the labeling options to be made available in the task UI”, ¶72, “ labeling platform 104 (e.g., dispatcher service 409) serves the configured browser-based task UI 420, then accepts the task result from the specialist”, ¶110, “workflow orchestrator 710 passes the labels determined by ML labeler 712 and blind judgement human labeler 714 for the labeling request to human labeler 716 so that the human specialist can see the previously determined labels”); and
receiving a user input confirming the first labeled sample based on the first metadata (¶168, ¶119, “if the answer output by human labeler 714 agrees with the answer output by ML labeler 712, this may result in that answer being considered a high-confidence answer that exceeds the target confidence threshold”, ¶133, “ a labeling task that requires one labeler to assess another labeler's output would be treated as conditional”).
With regard to Claim 15,
Claim 15 is similar in scope in to claim 2 ; therefore it is rejected under similar rationale. Martin further teach one or more non-transitory, computer-readable mediums comprising instructions that when executed by one or more processors causes operations (¶31, ¶240, “computer-readable program code may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium. The computer-readable program code can be operated on by a processor to perform steps, operations, methods, routines or portions thereof described herein”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Martin et al. [US 2021/0042577 A1, hereinafter Martin] in view of Smith et al. [US 2024/0160900 A1, hereinafter Smith].
With regard to Claim 1,
Martin teach a system for improvements to data labeling using a hybrid artificial intelligence labeling approach that is ambiguous to training data requirements, the system comprising:
one or more processors; and one or more non-transitory, computer-readable mediums comprising instructions that when executed by the one or more processors causes operations (2021, ¶31, ¶¶226-227)comprising:
retrieving a first portion from an unlabeled dataset, wherein the unlabeled dataset comprises a plurality of unlabeled samples, wherein the plurality of unlabeled samples is based on unstructured text based on linguistic inputs (2021, ¶¶48-50, “ data item to be labeled (e.g., image, video, word, document, or other discrete unit to be labeled)”);
generating a first labeled dataset based on the first portion, wherein the first labeled dataset comprises a first plurality of labeled samples, and wherein the first plurality of labeled samples is generated using a first data labeling routine, wherein the first data labeling routine comprises a first artificial intelligence model (¶93, “selects all unlabeled records (or some specified number thereof) for a use case (records that have not yet been labeled by the ML labeler) and has those labeled by the ML model 620”, ¶84, “Output labels from ML labeler 500 are the result of running a conditioned label request through a deployed ML model 501 to obtain an inferred answer”, ¶¶51-53, “Over time, the image classifier can be retrained using medical images labeled by the CDW or other training data to become increasingly accurate. As the confidence in the image classifier's accuracy increases, CDW may dynamically change to rely more heavily on the ML image classifier and reduce or eliminate human involvement in labeling training data”);
training, using the first labeled dataset, a second artificial intelligence model for a second data labeling routine, wherein the second artificial intelligence model comprises active learning (¶121, “Results that meet the configured target confidence threshold (or other training data) can be used to retrain ML labeler 712”, ¶92, “an active learning record selector 630 to select records for active learning”);
retrieving a second portion from the unlabeled dataset, generating, using the second data labeling routine, a second labeled dataset based on the second portion, wherein the second labeled dataset comprises a second plurality of labeled samples (¶93, “active learning record selector 630 selects all unlabeled records (or some specified number thereof) for a use case (records that have not yet been labeled by the ML labeler) and has those labeled by the ML model 620”);
determining a first confidence metric for a first labeled sample from the second plurality of labeled samples (¶93, “The ML model 620 evaluates its results (e.g., provides a confidence in its results)”, ¶124, ““confidence estimate” refers to an estimate of confidence in the result (probability of accuracy). In other words, the “confidence estimate” is an amount of confidence the labeling system has in a result”);
comparing the first confidence metric to a threshold confidence metric (¶26, “ collection of labelers which are consulted in sequence, and their individual results are incorporated into an overall result, until a configured confidence threshold for an overall result is reached”, ¶209, “If the confidence estimate returned for the label by QMS 750 meets the confidence threshold target);
in response to the first confidence metric not corresponding to the threshold confidence metric, assigning the first labeled sample to a third data labeling routine (¶25, “ routing requests to human specialists when the machine learning component produces a low confidence result”);
in response to assigning the first labeled sample to a third data labeling routine, deleting the first labeled sample from the second plurality of labeled samples (¶214, “the workflow orchestrator 710 can be configured to throw away the result and retry with a different labeler instance”); and
generating a new labeled sample for the first labeled sample based on the third data labeling routine, wherein the third data labeling routine comprises a user input via a user interface (¶214, “the workflow orchestrator 710 can be configured to throw away the result and retry with a different labeler instance”, ¶210, “workflow orchestrator 710 routes the labeling request to the next labeler”, ¶71, “a number of task templates can be defined for human labeler specialists with each task template expressing a user interface to use for presenting a labeling request to a human for labeling and receiving a label assigned by the human to the labeling request. Task UI configuration 412 can specify which template to use and the labeling options to be made available in the task UI”, ¶72, “the configured browser-based task UI 420, then accepts the task result from the specialist and validates it before sending it back to the labeler”, ¶68, “A human labeler acts as a gateway to a human specialist workforce”).
Martin does not disclose the first artificial intelligence model comprises a weakly supervised learning.
Smith teach a system for improvements to data labeling using a hybrid artificial intelligence labeling approach that is ambiguous to training data requirements, the system comprising:
one or more processors; and one or more non-transitory, computer-readable mediums comprising instructions that when executed by the one or more processors causes operations (¶335, ¶342, ¶52-53)comprising:
retrieving a first portion from an unlabeled dataset, wherein the unlabeled dataset comprises a plurality of unlabeled samples, wherein the plurality of unlabeled samples is based on unstructured text based on linguistic inputs (¶8, “ development process to create a high-quality set of labeling functions. This can be especially time-intensive when working with large data sets consisting of unstructured data (such as plain text, PDF documents, or HTML web pages) as the characteristics of the data cannot be meaningfully summarized without further processing”, ¶¶100-108);
generating a first labeled dataset based on the first portion, wherein the first labeled dataset comprises a first plurality of labeled samples, and wherein the first plurality of labeled samples is generated using a first data labeling routine, wherein the first data labeling routine comprises a first artificial intelligence model, wherein the first artificial intelligence model comprises a weakly supervised learning (¶7, “The labeling function or functions are applied to unlabeled data examples, and the outputs are aggregated into a final set of training labels using an algorithm or ruleset. This process is referred to as “weak supervision””, ¶87, “generative model outputs a set of probabilistic training labels, which can be used to train a flexible discriminative model “).
Martin and Smith are analogous art to the claimed invention because they are from a similar field of endeavor of semi-automatically generate labels for data. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin resulting in resolutions as disclosed by Smith with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Martin as described above to generate labels for large quantities of data at lower cost and quicker than manual labeling using programmatic labeling function to accelerate the process (Smith, ¶¶8-9). This simply combining prior art elements according to known methods to yield predictable results; use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 16,
Claim 16 is similar in scope in to claim 1; therefore it is rejected under similar rationale.
Claims 4 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Martin et al. [US 2021/0042577 A1, hereinafter Martin] in view of Ivanov et al. [US 2018/0129663 A1, hereinafter Ivanov].
With regard to Claim 4,
Martin teach the method of claim 2, wherein retrieving the first portion from the unlabeled dataset (¶93, “selects all unlabeled records (or some specified number thereof) for a use case (records that have not yet been labeled by the ML labeler)”).
Martin does not explicitly teach generating a plurality of identifiers to the plurality of unlabeled samples by assigning a respective identifier to each of the plurality of unlabeled samples; determining a sample size ; and selecting, using a random number generator, a subset of the plurality of identifiers, wherein a number of the identifiers in the subset corresponds to the sample size.
Ivanov teach generating a plurality of identifiers to the plurality of unlabeled samples by assigning a respective identifier to each of the plurality of unlabeled samples (¶7, “ each random number is determined based on a unique ID associated with each data element”, ¶38, “each data element in a dataset can be associated with a unique ID”, ¶67, “When a new object of a particular type is created … assigns a unique object identifier to it”);
determining a sample size (¶9, “first sample set comprises the top k data elements in the dataset based on the first priority score ranking at the first time, k being a predetermined number”, ¶40, “the top k data elements in a dataset can be selected based on priority score. If a sample set of 10,000 data elements is desired, for example, then the data elements having the top 10,000 priority scores can be selected”); and
selecting, using a random number generator, a subset of the plurality of identifiers, wherein a number of the identifiers in the subset corresponds to the sample size (¶6, “each data element of the dataset is associated with a random number”, ¶38, “random number generator module 206 can be configured to generate a random number for each data element in a dataset. As will be described in greater detail below, the random number associated with a data element can be utilized in conjunction with the weight of the data element to determine a priority score for the data element”. ¶40, “the top k data elements in a dataset can be selected based on priority score. If a sample set of 10,000 data elements is desired, for example, then the data elements having the top 10,000 priority scores can be selected”).
Martin and Ivanov are analogous art to the claimed invention because they are from a similar field of endeavor of data sampling and specifically to determine optimal sample size. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin resulting in resolutions as disclosed by Ivanov with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Martin as described above to generate data sampling can be a useful way to make conclusions about a dataset based on a subset, or a sample set, that is generally representative of the dataset as a whole (Ivanov, ¶3). This simply combining prior art elements according to known methods to yield predictable results; use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 17,
Claim 17 is similar in scope in to claim 4; therefore it is rejected under similar rationale.
Claims 5 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Martin et al. [US 2021/0042577 A1, hereinafter Martin] in view of Ivanov et al. [US 2018/0129663 A1, hereinafter Ivanov] in view of “Optimal number of features as a function of sample size for various classification rules“ Published 2004 [hereinafter D1].
With regard to Claim 5,
Martin-Ivanov teach method of claim 4, wherein determining the sample size (Ivanov, ¶9, “first sample set comprises the top k data elements in the dataset based on the first priority score ranking at the first time, k being a predetermined number”, ¶40, “the top k data elements in a dataset can be selected based on priority score. If a sample set of 10,000 data elements is desired, for example, then the data elements having the top 10,000 priority scores can be selected”). The same motivation to combine for claim 4 equally applies for current claim
Martin-Ivanov does not explicitly teach determining a number of features of the second model; and selecting the sample size based on the number of features.
D1 teach determining a number of features of the second model (P. 3, 2, “Feature size (d): Except for the regular histogram, all classifiers are tested on 29 different feature sizes from 2 to 30’, P. 1, 1, Col. 2, “Determining the optimal number of features is complicated by the fact that if we have D potential features, then there are C(D,d) feature sets of size d”, P. 3, 3, “the optimal feature size is around n−1”; and
selecting the sample size based on the number of features (P. 3, 3, “Note that the sample size must exceed the number of features to avoid degeneracy”, P. 3, 2, “Sample size (n): Sample sizes run from 10 to 200, increased by steps of 10, for a total of 20 sample sizes”, P. 7, 5, “the performance of a designed classifier can be greatly influenced by the number of features and therefore one should attempt to use a number close to the optimal number”).
Martin-Ivanov and D1 are analogous art to the claimed invention because they are from a similar field of endeavor of data sampling and specifically to determine optimal sample size. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin-Ivanov resulting in resolutions as disclosed by D1 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Martin-Ivanov as described above to determine the minimum or optimal sample size sufficient to reliably train and evaluate a model without spending unnecessary resources. This simply combining prior art elements according to known methods to yield predictable results; use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 18,
Claim 18 is similar in scope in to claim 5; therefore it is rejected under similar rationale.
Claims 6 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Martin et al. [US 2021/0042577 A1, hereinafter Martin] in view of Ivanov et al. [US 2018/0129663 A1, hereinafter Ivanov] in view of Segura et al. [US 2019/0142965 A1, hereinafter Segura].
With regard to Claim 6,
Martin-Ivanov teach method of claim 4, wherein determining the sample size (Ivanov, ¶9, “first sample set comprises the top k data elements in the dataset based on the first priority score ranking at the first time, k being a predetermined number”, ¶40, “the top k data elements in a dataset can be selected based on priority score. If a sample set of 10,000 data elements is desired, for example, then the data elements having the top 10,000 priority scores can be selected”). The same motivation to combine for claim 4 equally applies for current claim
Martin-Ivanov does not explicitly teach determining a number of features of the second model; and selecting the sample size based on the number of features.
Segura teach determining a number of features of the second model (P. 3, 2, “Feature size (d): Except for the regular histogram, all classifiers are tested on 29 different feature sizes from 2 to 30’, P. 1, 1, Col. 2, “Determining the optimal number of features is complicated by the fact that if we have D potential features, then there are C(D,d) feature sets of size d”, P. 3, 3, “the optimal feature size is around n−1”); and
selecting the sample size based on the number of features (P. 3, 3, “Note that the sample size must exceed the number of features to avoid degeneracy”, P. 3, 2, “Sample size (n): Sample sizes run from 10 to 200, increased by steps of 10, for a total of 20 sample sizes”, P. 7, 5, “the performance of a designed classifier can be greatly influenced by the number of features and therefore one should attempt to use a number close to the optimal number”.
Martin-Ivanov and Segura are analogous art to the claimed invention because they are from a similar field of endeavor of data sampling and specifically to determine optimal sample size. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin-Ivanov resulting in resolutions as disclosed by Segura with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Martin-Ivanov as described above to determine the minimum or optimal sample size sufficient to reliably train and evaluate a model without spending unnecessary resources. This simply combining prior art elements according to known methods to yield predictable results; use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 19,
Claim 19 is similar in scope in to claim 6; therefore it is rejected under similar rationale.
Claims 7 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Martin et al. [US 2021/0042577 A1, hereinafter Martin] in view of Ivanov et al. [US 2018/0129663 A1, hereinafter Ivanov] in view of “HyperNP: Interactive Visual Exploration of Multidimensional Projection Hyperparameters“ Published 2021 [hereinafter D2].
With regard to Claim 7,
Martin-Ivanov teach method of claim 4, wherein determining the sample size (Ivanov, ¶9, “first sample set comprises the top k data elements in the dataset based on the first priority score ranking at the first time, k being a predetermined number”, ¶40, “the top k data elements in a dataset can be selected based on priority score. If a sample set of 10,000 data elements is desired, for example, then the data elements having the top 10,000 priority scores can be selected”). The same motivation to combine for claim 4 equally applies for current claim
Martin-Ivanov does not explicitly teach determining a number of hyperparameters requiring training for the second model; and selecting the sample size based on the number of hyperparameters requiring training.
D2 teach determining a number of hyperparameters requiring training for the second model (P. 9, 6.3, “the complexity of training a traditional network is O(K), the complexity of training HyperNP with one hyperparameter is O(K×|H|) with |H| as described in Eqn. 3. For |h| hyperparameters, this becomes O(K×|H||h|)”; and selecting the sample size based on the number of hyperparameters requiring training (P. 7, 5.2, “the neural network will be trained on the number of sample dinstances times the number of hyperparameters. For our experiments we chose what we expect an average case would be, examining hyperparameter values from 2 to 50. It should be noted that Figure 10 shows overall training time for HyperNP as a function of dataset size; the number of training instances is determined not just by this value, but the number of hyperparameter values and fraction of this number used”, P. 3, 3.2, “where D’ _D is the training set and H’ _H is the set of sampled values of the hyperparameters h, respectively”).
Martin-Ivanov and D2 are analogous art to the claimed invention because they are from a similar field of endeavor of data sampling and specifically to determine optimal sample size. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin-Ivanov resulting in resolutions as disclosed by D2 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Martin-Ivanov as described above to determine the minimum or optimal sample size sufficient to reliably train and evaluate a model without spending unnecessary resources. This simply combining prior art elements according to known methods to yield predictable results; use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 20,
Claim 20 is similar in scope in to claim 7; therefore it is rejected under similar rationale.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Martin et al. [US 2021/0042577 A1, hereinafter Martin] in view of Ivanov et al. [US 2018/0129663 A1, hereinafter Ivanov] in view of Ukil et al. [US 2016/0371228 A1, hereinafter Ukil].
With regard to Claim 8,
Martin-Ivanov teach method of claim 4, wherein determining the sample size (Ivanov, ¶9, “first sample set comprises the top k data elements in the dataset based on the first priority score ranking at the first time, k being a predetermined number”, ¶40, “the top k data elements in a dataset can be selected based on priority score. If a sample set of 10,000 data elements is desired, for example, then the data elements having the top 10,000 priority scores can be selected”). The same motivation to combine for claim 4 equally applies for current claim
Martin-Ivanov does not explicitly teach determining the sample size further comprises: determining a data variability of the first labeled dataset; and selecting the sample size based on the data variability.
Ukil teach determining the sample size further comprises: determining a data variability of the first labeled dataset (¶33, “ The standard deviation (σ), computed by the system 202”); and
selecting the sample size based on the data variability (¶28, Eq. 1, ¶33, “The system 202 may determine the critical sample size … and the standard deviation (σ)”, ¶¶40-41, “optimal sample block size of the plurality of dataset may be computed by using the standard deviation”).
Martin-Ivanov and Ukil are analogous art to the claimed invention because they are from a similar field of endeavor of data sampling and specifically to determine optimal sample size. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin-Ivanov resulting in resolutions as disclosed by Ukil with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Martin-Ivanov as described above to determine the minimum or optimal sample size sufficient to reliably train and evaluate a model without spending unnecessary resources. This simply combining prior art elements according to known methods to yield predictable results; use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over Martin et al. [US 2021/0042577 A1, hereinafter Martin] in view of Ivanov et al. [US 2018/0129663 A1, hereinafter Ivanov] in view of Boren et al. [US 2025/0245564 A1, hereinafter Boren].
With regard to Claim 9,
Martin-Ivanov teach method of claim 4, wherein determining the sample size (Ivanov, ¶9, “first sample set comprises the top k data elements in the dataset based on the first priority score ranking at the first time, k being a predetermined number”, ¶40, “the top k data elements in a dataset can be selected based on priority score. If a sample set of 10,000 data elements is desired, for example, then the data elements having the top 10,000 priority scores can be selected”). The same motivation to combine for claim 4 equally applies for current claim
Martin-Ivanov does not explicitly teach determining respective processing power requirements for training the second model; and selecting the sample size based on the respective processing power requirements for training the second model.
Boren teach determining respective processing power requirements for training the second model; and selecting the sample size based on the respective processing power requirements for training the second model (¶14, “the performance indicators of a trainer may include at least one of: available memory space of each processor in the trainer, computing power of each processor in the trainer, total memory space of the processors in the trainer, frequency of past dynamic adjustments of the size of the micro-batch assigned to the trainer, specifications of each processor of the trainer and sample processing rate of each processor in the trainer”, ¶99, “Orchestrator 120 and trainers 130, 132, and 134 may have the ability to dynamically adjust the size of the micro-batch in accordance to the size of the currently available memory space of each of the processors (e.g. GPU's) in each of the trainers 130, 132 and 134, the total memory space of the processors in each of the trainers 130, 132 and 134, and the computing power of each of the processors and of all processors combined, of each of trainers”, ¶107, “Orchestrator 120 may determine or calculate a maximal micro-batch size for each of the processors (e.g. GPUs) in trainer 711, 713 and 715 based on the performance indicator of the respective trainer, and parameters of the mini-task, such that this maximal micro-batch size is small enough to enable at least one of processors in trainer 711, 713 and 715 to perform or execute the mini-task 701”).
Martin-Ivanov and Boren are analogous art to the claimed invention because they are from a similar field of endeavor of data sampling and specifically to determine optimal sample size. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin-Ivanov resulting in resolutions as disclosed by Boren with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Martin-Ivanov as described above to determine the minimum or optimal sample size sufficient to reliably train and evaluate a model without spending unnecessary resources. This simply combining prior art elements according to known methods to yield predictable results; use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Claims 10 is rejected under 35 U.S.C. 103 as being unpatentable over Martin et al. [US 2021/0042577 A1, hereinafter Martin] in view of Ivanov et al. [US 2018/0129663 A1, hereinafter Ivanov] in view of “Tissue Classification with Gene Expression Profiles“ published 2000 [hereinafter D3].
With regard to Claim 10,
Martin-Ivanov teach method of claim 4, wherein determining the sample size (Ivanov, ¶9, “first sample set comprises the top k data elements in the dataset based on the first priority score ranking at the first time, k being a predetermined number”, ¶40, “the top k data elements in a dataset can be selected based on priority score. If a sample set of 10,000 data elements is desired, for example, then the data elements having the top 10,000 priority scores can be selected”). The same motivation to combine for claim 4 equally applies for current claim
Martin-Ivanov does not explicitly teach determining the sample size further comprises: determining a validation type of the second model; and selecting the sample size based on the validation type.
D3 teach determining the sample size further comprises: determining a validation type of the second model; and selecting the sample size based on the validation type (P. 5-6, 3.2, “A common method to test accuracy m such situations is cross-validation. To apply this method, we partition the data into k sets of samples, C1,..., Ck (typically, these will be of roughly the same size). Then, we construct a dataset D, = D - C,, and test the accuracy of fD, () on the samples in C,. Having done this for all 1 < , < k we estimate the accuracy of the method by averaging the accuracy in each one of the cross-validation trials. Cross-validation has several important properties. First, the training set and the test set in each trial are disjoint. Second, the classifier is tested on each sample exactly once. Finally, the training set for each trial is (k - l)/k of the original data set. Thus, for large k, we get a relatively unbiased estimate of the classifier behavior gwen a training set of size There are several possible choices of k. A common approach is to set k = m. In this case, every trial removes a single sample and trains on the rest. This method is known as leave one out cross validation (LOOCV). Other common choices are k = 10 or k = 5. LOOCV has been in use since early days of pattern recognition (e.g, [10]). In some situations, using larger partitions reduces the variance of the estimators (see [18]). In this work, since the number of samples is small, we use LOOCV”).
Martin-Ivanov and D3 are analogous art to the claimed invention because they are from a similar field of endeavor of data sampling and specifically to determine optimal sample size. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin-Ivanov resulting in resolutions as disclosed by D3 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Martin-Ivanov as described above to determine the minimum or optimal sample size sufficient to reliably train and evaluate a model without spending unnecessary resources. This simply combining prior art elements according to known methods to yield predictable results; use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Claims 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Martin et al. [US 2021/0042577 A1, hereinafter Martin] in view of ”Active Learning With Drifting Streaming Data” Published on 2014 [hereinafter D4].
With regard to Claim 11,
Martin teach the method of claim 2, wherein comparing the first confidence metric to the threshold confidence metric (¶209, “If the confidence estimate returned for the label by QMS 750 meets the confidence threshold target”, ¶210, “If the confidence estimate returned by QMS 750 does not meet the confidence threshold”).
Martin does not explicitly teach determining available resources for the third data labeling routine; and determining the threshold confidence metric based on the available resources.
D4 teach determining available resources for the third data labeling routine (P. 3, IIII.A, Col. 1, “the budget is a fraction of all the incoming instances. The budget is motivated by online learning applications with limited resources, e.g., laboratory tests in the chemical industry”); and determining the threshold confidence metric based on the available resources (P. 6, Eq. (13), “Thus we introduce a time-variable threshold, which adjusts itself depending on the incoming data to align with the budget. If a classifier becomes more certain (stable situations), the threshold expands to be able to capture the most uncertain instances. If a change happens and suddenly many labeling requests appear, then the threshold is contracted to query the most uncertain instances first”).
Martin and D4 are analogous art to the claimed invention because they are from a similar field of endeavor of active learning. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin resulting in resolutions as disclosed by D4 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Martin as described above to provide a system that to adopt and react well to changes that can occur anywhere in the instance space and unexpectedly rather than a system that have a fixed threshold and eventually would stop learning and fail to react to changes (D4, P. 1, Abstract Introduction). This simply combining prior art elements according to known methods to yield predictable results; use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Claims 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Martin et al. [US 2021/0042577 A1, hereinafter Martin] in view of Zhdanov et al. [US 11501210 B1, hereinafter Zhdanov].
With regard to Claim 12,
Martin teach the method of claim 2, wherein comparing the first confidence metric to the threshold confidence metric (¶209, “If the confidence estimate returned for the label by QMS 750 meets the confidence threshold target”, ¶210, “If the confidence estimate returned by QMS 750 does not meet the confidence threshold”).
Martin does not explicitly teach determining a user identifier for the third data labeling routine; and determining the threshold confidence metric based on the user identifier.
Zhdanov determining a user identifier for the third data labeling routine (Col. 17, lines 6064, “The content review service 106 may maintain a database of reviewers or reviewers utilized by the content review service 106 when reviewing content. In some instances, each of the reviewers may be experts or trained within specific fields to identify certain subject matter within the content“, Col. 18, lines 8-15, “select the reviewer 104 based on their field of expertise, the content 114, the request of the user 102, the condition(s) 116, and the confidence of the ML model(s) 126. In some instances, selecting a specific reviewer may assist in accurately fulfilling the request of the user”); and
determining the threshold confidence metric based on the user identifier (Col. 22, lines 14-20, “the process 300 may determine the accuracy of certain reviewers. This accuracy, or results of the reviewers, may be used to generate model(s) indicative of the accuracy of the reviewer. The similarly between reviewers and/or the accuracy of the reviewers may be used to determine a confidence of the ML model(s) and/or the confidence of the results of the ML model(s)”).
Martin and Zhdanov are analogous art to the claimed invention because they are from a similar field of endeavor of labeling/annotating data for training machine learning models. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin resulting in resolutions as disclosed by Zhdanov with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Martin as described above to provide a realistic degree of confidence in the confidence of the ML model(s) and/or the confidence of the results of the ML model(s) (Zhdanov, Col. 22, lines 14-20, “the process 300 may determine the accuracy of certain reviewers. This accuracy, or results of the reviewers, may be used to generate model(s) indicative of the accuracy of the reviewer. The similarly between reviewers and/or the accuracy of the reviewers may be used to determine a confidence of the ML model(s) and/or the confidence of the results of the ML model(s)”). This simply combining prior art elements according to known methods to yield predictable results; use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 13,
Martin-Zhdanov teach the method of claim 12, wherein determining the threshold confidence metric based on the user identifier further comprises: determining a user accuracy rating attributed to the user identifier (Zhdanov, Col. 22, lines 12-20, “Based on an agreement and consistency over time, or whether the reviewers agree (e.g., reviews indicating the same results), the process 300 may determine the accuracy of certain reviewers. This accuracy, or results of the reviewers, may be used to generate model(s) indicative of the accuracy of the reviewer. The similarly between reviewers and/or the accuracy of the reviewers may be used to determine a confidence of the ML model(s) and/or the confidence of the results of the ML model(s)”, Col. 16, lines 59-55, “ Audits may also be performed based on experience levels”); and determining the threshold confidence metric based on the user accuracy rating (Zhdanov, Col. 22, lines 12-20, “Based on an agreement and consistency over time, or whether the reviewers agree (e.g., reviews indicating the same results), the process 300 may determine the accuracy of certain reviewers. This accuracy, or results of the reviewers, may be used to generate model(s) indicative of the accuracy of the reviewer. The similarly between reviewers and/or the accuracy of the reviewers may be used to determine a confidence of the ML model(s) and/or the confidence of the results of the ML model(s)”). The same motivation to combine for claim 12 equally applies for current claim
Conclusion
The prior art made of record and not relied upon is considered pertinent to the applicant’s disclosure.
US Patent Application Publication No. 20240054390 filed by Wendt et al. that disclose to the improvement of label identification and/or quality in training and/or testing data for machine-learning models See at least Abstract, ¶1-5
Examiner has pointed out particular references contained in the prior arts of record in the body of this action for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and Figures may apply as well. It is respectfully requested from the applicant, in preparing the response, to consider fully the entire references as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior arts or disclosed by the examiner. It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED ABOU EL SEOUD whose telephone number is (303)297-4285. The examiner can normally be reached Monday-Thursday 9:00am-6:00pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED ABOU EL SEOUD/Primary Examiner, Art Unit 2148
Prosecution Insights