Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
This Office Action is responsive to Applicants' Amendment filed on March 02, 2026, in which claims 1, 4-6, 8, 11, 12, 16, and 19 are amended. No claims have been newly cancelled or added. Claims 1-20 are currently pending.
Response to Arguments
With regards to the objection to claims 1, 8 and 16, Applicant has amended the claims to correct the previously noted informalities. However, due to Applicant’s amendment additional informalities have been introduced and thus claims 1, 8, and 16 continue to be objected to, albeit on different grounds.
With regards to the rejections of claims 1-5, 7-9, 11-13, and 16-19 under 35 U.S.C 103 as being unpatentable over Li et al. “DivideMix: Learning with Noisy Labels as Semi-supervised Learning”, further in view of Liu et al. “Co-Correcting: Noise-Tolerant Medical Image Classification via Mutual Label Correction”, further in view of Quader et al. (U.S. Patent Application Publication No. 2021/0357776), further in view of Bai et al. “Understanding and Improving Early Stopping for Learning with Noisy Labels”, Applicant’s arguments that the claims as amended overcome the rejections are persuasive, however the arguments are moot in view of a new grounds of rejection, necessitated by Applicant’s amendments to the claims, as presented below. Examiner also notes that there does not appear to be support in the original disclosure for a limitation amended into claims 1, 8, and 16, necessitating new matter rejections under 35 U.S.C 112(a).
Claim Objections
Claims 1, 8, and 16 objected to because of the following informality: swapping…data samples between the first and second machine learning model should read “swapping…data samples between the first and second machine learning models”. Appropriate correction is required.
Claim Rejections - 35 USC § 112(a)
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claims contain subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claim 1,
Claim 1, as amended, recites the limitation determining, based on the relabeling, that a number of corrections to labels in the training data set has been performed. This limitation does not appear to be supported by the specification as originally filed. On page 13 of the Remarks, Applicant cites the following for support: “’the computer system may swap the training data examples after a set number of cross-label-corrections S (e.g., operations at blocks 110-116) have been performed.’ See paragraph [0051] of the application”. However, it appears that “a number of cross-label-corrections” does not refer to a number of labels that are corrected, but rather a number of iterations of operations which have the effect of correcting an indeterminate amount of labels. See Fig. 1 from the Drawings, where it can be seen that a group of operations at blocks 110-116, which the specification refers to as a cross-label correction, have the effect of predicting, using first and second machine learning models, cross-feeding noisy data samples between the first and second machine learning models, and then correcting at least one (but possibly more) noisy label. Blocks 110-116 may then optionally iterate an additional number of times before proceeding to block 118, which is the swapping operation.
PNG
media_image1.png
607
381
media_image1.png
Greyscale
This use of “cross-label-correction” to refer to a group of operations that correct labels rather than a number of labels that are corrected is consistent throughout the specification. See [0049] “After the relabeling has been performed, the computer system may repeat the operations at blocks 106 and 108 until C half-epochs have been performed (e.g., eh mod C = = 0), indicating the cross-label-correction at blocks 110-116 should be performed again” and paragraphs [0020]-[0023], which describe the various operations that are part of each “cross-label-correction”. Therefore the inclusion of the limitation determining, based on the relabeling, that a number of corrections to labels in the training data set has been performed within the claim constitutes new matter not supported by the original disclosure. Examiner notes that in the case that “determining…that a number of corrections to labels…has been performed” does not to refer to “a number of corrections” specifically but rather “any number of corrections”, which is a range of numbers of corrections, which would have support within the specification, the limitation would be obvious in view of the combination of prior art of Li, Liu, Quader, and Bai, without the need to include the newly cited Zhang-2018.
In reference to dependent claims 2-7, claims 2-7 do not cure the deficiencies noted in the rejection of independent claim 1. Therefore, these claims are rejected under the same rationale as claim 1.
Regarding claim 8,
Claim 8, as amended, recites the limitation determining, by the computer system, that a number of corrections to labels in the training data set has been performed. The inclusion of this limitation within the claim within the claim constitutes new matter not supported by the original disclosure, with the same rationale as given for the rejection of claim 1.
In reference to dependent claims 9-15, claims 9-15 do not cure the deficiencies noted in the rejection of independent claim 8. Therefore, these claims are rejected under the same rationale as claim 8.
Regarding claim 16,
Claim 16, as amended, recites the limitation determining, based on the relabeling, that a number of corrections to labels in the training data set has been performed. The inclusion of this limitation within the claim within the claim constitutes new matter not supported by the original disclosure, with the same rationale as given for the rejection of claim 1.
In reference to dependent claims 17-20, claims 17-20 do not cure the deficiencies noted in the rejection of independent claim 16. Therefore, these claims are rejected under the same rationale as claim 16.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-9, 11-13, and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. “DivideMix: Learning with Noisy Labels as Semi-supervised Learning”, hereinafter Li, further in view of Liu et al. “Co-Correcting: Noise-Tolerant Medical Image Classification via Mutual Label Correction”, hereinafter Liu, further in view of Quader et al. (U.S. Patent Application Publication No. 2021/0357776), hereinafter Quader, further in view of Bai et al. “Understanding and Improving Early Stopping for Learning with Noisy Labels”, hereinafter Bai, further in view of Zhang et al. “Improving Crowdsourced Label Quality Using Noise Correction”, hereinafter Zhang-2018.
Regarding claim 1,
Li teaches A computer system comprising:
performing a cross-label correction of a training data set using a first machine learning model and a second machine learning model, wherein the performing the cross-label correction comprises: (Li Pg. 3, Fig. 1 shows a cross-label correction being performed using two different neural networks, which are first and second machine learning models)
PNG
media_image2.png
443
1265
media_image2.png
Greyscale
splitting the training data set into a first portion and a second portion; ((Li Pg. 3) “At each epoch, we perform co-divide, where one network divides the noisy training dataset into a clean labeled set (X) and a noisy unlabeled set (U), which are then used by the other network”, broadest reasonable interpretation of two portions of the training data set includes two versions of the same dataset split differently into a clean portion and a noisy portion)
determining that [the training data set includes] a plurality of noisy data samples that increase the likelihood of overfitting the first and second machine learning models on the training data set; ((Li Pg. 1) “A recent study (Zhang et al., 2017) shows that DNNs can easily overfit to noisy labels and results in poor generalization performance”, Li does not explicitly teach determining that the training set includes noisy data)
determining one or more first noisy data samples from the first portion of the training data set and one or more second noisy data samples from the second portion of the training data set based on a first set of loss function values associated with the first portion and a second set of loss function values associated with the second portion; ((Li Pg. 3, Fig. 1 Caption) “At each epoch, a network models its per-sample loss distribution with a GMM to divide the dataset into a labeled set (mostly clean) and an unlabeled set (mostly noisy), which is then used as training data for the other network (i.e. co-divide)”, Li Pg. 3, Fig. 1 shows that both portions of the training data set, portion A used to train network A, and portion B used to train network B, contain noisy unlabeled data samples, which is determined by a modeled loss distribution for the respective network)
inputting one or more first noisy data samples from the first portion of the training data set to the second machine learning model to be classified ((Li Pg. 5) “Having acquired X-hat (and U-hat) which consists of multiple augmentations of labeled (unlabeled) samples and their refined (guessed) labels, we follow MixMatch to “mix” the data, where each sample is interpolated with another sample randomly chosen from the combined mini-batch of X-hat and U-hat”, Li Pg. 3, Fig. 1 shows that mixed data including noisy data samples from a first dataset portion is given to network B during training, training of a classification model using a training dataset includes classifying the dataset) based on the first set of loss function values ((Li Pg. 3, Fig. 1 Caption) “At each epoch, a network models its per-sample loss distribution with a GMM to divide the dataset into a labeled set (mostly clean) and an unlabeled set (mostly noisy), which is then used as training data for the other network (i.e. co-divide)”, creating divided portions for another network based on the per-sample loss distribution corresponds to inputting noisy data samples from one training data set to a different machine learning model based on a first set of loss function values)
inputting one or more second noisy data samples from the second portion of the training data set to the first machine learning model to be classified ((Li Pg. 5) “Having acquired X-hat (and U-hat) which consists of multiple augmentations of labeled (unlabeled) samples and their refined (guessed) labels, we follow MixMatch to “mix” the data, where each sample is interpolated with another sample randomly chosen from the combined mini-batch of X-hat and U-hat”, Li Pg. 3, Fig. 1 shows that mixed data including noisy data samples from a second dataset portion is given to network A during training, training of a classification model using a training dataset includes classifying the dataset) based on the second set of loss function values ((Li Pg. 3, Fig. 1 Caption) “At each epoch, a network models its per-sample loss distribution with a GMM to divide the dataset into a labeled set (mostly clean) and an unlabeled set (mostly noisy), which is then used as training data for the other network (i.e. co-divide)”, creating divided portions for another network based on the per-sample loss distribution corresponds to inputting noisy data samples from one training data set to a different machine learning model based on a second set of loss function values)
performing a number of iterations of the cross-label correction of the training data set, (Li Pg. 4, Algorithm 1 shows iterations of cross-label correction being performed at multiple points, such as a number of training epochs (at line 3), a number of iterations on two training sets (at line 9), and a number of iterations on mini-batches (at line 12))
PNG
media_image3.png
780
833
media_image3.png
Greyscale
and swapping, based on the determining that the number of corrections has been performed, data samples between the first and second machine learning model in a subsequent iteration of the training the first and second machine learning models, (Li Pg. 4, Algorithm 1 shows that application of MixMatch, in which swapping of data samples between two data sets which are used to train two different machine learning models occurs, takes place only after it is determined that a number of mini-batch correction iterations from lines 12 to 22 (specifically B iterations) have been performed)
Liu teaches a non-transitory memory storing instructions; and one or more hardware processors configured to execute the instructions and cause the computer system to perform operations comprising ((Liu Pg. 6) “Our experiments were carried out on the software platform of Ubuntu 18.04 LTS and PyTorch 1.4.0, using the hardware of Intel i7-8700k CPU, NVIDIA RTX2080Ti GPU, and 32GB RAM”), as well as the following further limitations that Liu teaches more explicitly than Li:
a non-transitory memory storing instructions; and one or more hardware processors configured to execute the instructions and cause the computer system to perform operations comprising: ((Liu Pg. 6) “Our experiments were carried out on the software platform of Ubuntu 18.04 LTS and PyTorch 1.4.0, using the hardware of Intel i7-8700k CPU, NVIDIA RTX2080Ti GPU, and 32GB RAM”)
relabeling at least one noisy data sample of the one or more [first] noisy data samples based on a classification of the one or more [first] noisy data samples outputted by the second machine learning model; (Liu Pg. 4, Fig. 4 shows how the loss from the classification of the second machine learning model (backbone CNN-2) is used to compute the distribution used to correct noisy labels, (Liu Pg. 4, Fig. 4 Caption) “We use the label distribution yd to replace the noisy label y-hat”, Li teaches two sets of noisy data samples)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li and Liu to use the method of training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, taught by Li, with the system including noisy label cross-correction taught by Liu, as Liu teaches (Liu Pg. 7) “the methods that do not update the label have a significant decrease in accuracy as the proportion of noises increases. These methods try to avoid learning noisy samples. Therefore as the noise ratio increases, the samples actually used for training will be reduced accordingly”, that is, noisy label correction allows for high classification accuracy even on very noisy datasets. Such a combination would be obvious.
Quader teaches the following further limitations that neither Li nor Liu teaches explicitly:
determining that the training data set includes a plurality of noisy data samples … ((Quader Abstract) “The processor(s) applies ensemble machine-learning and a generative model to the training dataset to detect noisy labeled datapoints in the training dataset”)
in response to completing the number of iterations of the cross-label correction, outputting the training data set with the cross label correction to a machine learning classifier; ((Quader [0058]) “Program code repeats the datapoint label rectification process to gradually create a labeled dataset (DL) 423 (FIG. 4) as the union of the cleaned dataset Dclean and the actively-learned dataset DAL 806. Note that the classifier f 428 can be retrained for each iteration through phase 2 310. The phase outputs the completely cleaned dataset DL 432 as the completely cleaned version of the original noisy and incomplete dataset 402, along with the classifier f 428 (i.e., machine-learning model) trained using completely cleaned dataset DL 432”)
and training the machine learning classifier of a machine learning system based on the training data set with the cross label correction ((Quader [0058]) “The phase outputs the completely cleaned dataset DL 432 as the completely cleaned version of the original noisy and incomplete dataset 402, along with the classifier f 428 (i.e., machine-learning model) trained using completely cleaned dataset DL 432”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, and Quader to use the system of training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, and the system including noisy label cross-correction, taught jointly by Li and Liu, with detecting the presence of noisy labels, outputting the cleaned dataset, and using it to train a machine learning classifier, as taught by Quader, as it is well-known within the art that machine learning models trained on data with more accurate labels will themselves be more accurate when deployed. Such a combination would be obvious.
Bai teaches the following further limitation that neither Li, nor Liu, nor Quader teaches:
and training the [first] machine learning model [based on the first portion and the second machine learning model based on the second portion] utilizes a number of training epochs selected to minimize a likelihood of overfitting the [first and second] machine learning models on the training data set, ((Bai Pg. 3) “to alleviate the impact of noisy labels for latter layers, we reinitialize and progressively train latter DNN layers by using smaller numbers of epochs with preceding DNN layers fixed”, (Bai Pg. 2) “the memorization effect−if clean labels are of majority within the noisy labels for each class, deep networks tend to first memorize and fit majority (clean) patterns and then overfit minority (noisy) patterns”, Li and Liu teach first and second machine learning models with first and second portions of the training data set)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, and Bai to use the system of training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, noisy label cross-correction, and detecting the presence of noisy labels and outputting a corrected training data set, taught jointly by Li, Liu, and Quader, and adding the use of selecting a number of training epochs that likely do not overfit the training data, taught by Bai, as Bai teaches: (Bai Pg. 2) “As deep networks have large model capacities, they can easily memorize and eventually overfit the noisy labels, leading to poor generalization performance [38]. Therefore, it is of great importance to develop a methodology that is robust to noisy annotations”. Such a combination would be obvious.
Zhang-2018 teaches the following further limitation that neither Li, nor Liu, nor Quader, nor Bai teaches:
wherein the performing the number of iterations includes: determining, based on the relabeling, that a number of corrections to labels in the training data set has been performed, ((Zhang-2018 Pg. 7) “The BC baseline noise correction algorithm adopts the framework in [43], namely, self-training semisupervised noise correction. The potential noises are first filtered out from the original training set as unlabeled instances. The remaining training set is used to build a learning model. All unlabeled instances (noises) are predicted by this learning model, but only a small portion are confirmed as repaired with high confidence. The repaired instances are removed from the unlabeled set and merged into the training set for the next iteration of correction. The remaining unlabeled instances will be further corrected in the next iteration. The correction loop stops when the number of iterations reaches its predefined maximum value or all noises are confirmed”, iterative label correction that stops when all noisy labels are confirmed to be repaired corresponds to determining that a number of corrections to labels have been performed)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, Bai, and Zhang-2018 to use the system of training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, noisy label cross-correction, detecting the presence of noisy labels and outputting a corrected training data set, and selecting a number of training epochs that likely do not overfit the training data, taught jointly by Li, Liu, Quader, and Bai, and including, during performance of a number of iterations, that a number of corrections to labels have been performed, taught by Zhang-2018, as doing so allows for the iterative process to be stopped once a satisfactory amount of labels have relabeled, such as all of the noisy labels, which provides the predictable benefit of improving computational efficiency by not performing unnecessary iterations. Such a combination would be obvious.
Regarding claim 2,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The computer system of claim 1,
Li further teaches:
wherein the operations further comprise: selecting the one or more first noisy data samples from the first portion of the training data set ((Li Pg. 3) “At each epoch, we perform co-divide, where one network divides the noisy training dataset into a clean labeled set (X) and a noisy unlabeled set (U), which are then used by the other network”, Li Pg. 3, Fig. 1 shows a first network A receiving first noisy data samples from a first portion), based on a corresponding loss function value of the first set of loss function values for each of the one or more first noisy data samples determined in a classification of the first portion using the first machine learning model ((Li Pg. 3) “Following Arazo et al. (2019), we aim to find the probability of a sample being clean by fitting a mixture model to the per-sample loss distribution…Given a model with parameters θ, the cross-entropy loss l(θ) reflects how well the model fits the training samples”, fitting a classification model to training samples includes classification of the training samples, Li Pg. 3, Fig. 1 shows a first network A receiving first noisy data samples from a first portion, including a first per-sample loss distribution for the network)
and selecting the one or more second noisy data samples from the second portion of the training data set ((Li Pg. 3) “At each epoch, we perform co-divide, where one network divides the noisy training dataset into a clean labeled set (X) and a noisy unlabeled set (U), which are then used by the other network”, Li Pg. 3, Fig. 1 shows a second network B receiving second noisy data samples from a second portion), based on a corresponding loss function value of the second set of loss function values for each of the one or more second noisy data samples determined in a classification of the second portion using the first machine learning model ((Li Pg. 3) “Following Arazo et al. (2019), we aim to find the probability of a sample being clean by fitting a mixture model to the per-sample loss distribution…Given a model with parameters θ, the cross-entropy loss l(θ) reflects how well the model fits the training samples”, fitting a classification model to training samples includes classification of the training samples, Li Pg. 3, Fig. 1 shows a second network B receiving second noisy data samples from a second portion, including a second per-sample loss distribution for the network)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 2, claim 1. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 3,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The computer system of claim 2,
Li further teaches:
wherein the operations further comprise: classifying the first portion using the first machine learning model (Li Pg. 3, Fig. 1 shows a portion of the training data is used to train network A after the co-divide step, broadest reasonable interpretation of training a classifier includes classification)
wherein the one or more first noisy data samples are selected as a percent of samples having a largest loss function value of the first set of loss function values in the classifying using the first machine learning model ((Li Pg. 3) “For each sample, its clean probability wi is the posterior probability p(g|i), where g is the Gaussian component with smaller mean (smaller loss). We divide the training data into a labeled set and an unlabeled set by setting a threshold τ on wi”, a threshold on the probability based on loss means that the samples selected as unlabeled and noisy are those with the highest loss, Li Pg. 3, Fig. 1 shows model A’s GMM selects a noisy data portion and the caption states each model has its own per-sample loss distribution)
and classifying the second portion using the second machine learning model (Li Pg. 3, Fig. 1 shows a different portion of the training data is used to train network B after the co-divide step, broadest reasonable interpretation of training a classifier includes classification)
wherein the one or more second noisy data samples are selected as a percent of samples having a largest loss function value of the second set of loss function values in the classifying using the second machine learning model ((Li Pg. 3) “For each sample, its clean probability wi is the posterior probability p(g|i), where g is the Gaussian component with smaller mean (smaller loss). We divide the training data into a labeled set and an unlabeled set by setting a threshold τ on wi”, a threshold on the probability based on loss means that the samples selected as unlabeled and noisy are those with the highest loss, Li Pg. 3, Fig. 1 shows model B’s GMM selects a noisy data portion and the caption states each model has its own per-sample loss distribution)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 3, claim 2. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 4,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The computer system of claim 1,
Li further teaches:
wherein the performing the number of iterations of the cross-label correction includes: retraining the first machine learning model using a third portion of the training data set; (Li Pg. 3, Fig. 1 shows network A undergoes at least two training epochs (e-1 and e), and that every time a portion of the training data set is used, which is remade in each epoch)
and retraining the second machine learning model using a fourth portion of the training data set, (Li Pg. 3, Fig. 1 shows network B undergoes at least two training epochs (e-1 and e), and that every time a portion of the training data set is used, which is remade in each epoch)
wherein the [third portion or the fourth portion] have the at least one noisy sample relabeled for the retraining ((Liu Pg. 2) “3) A label correction curriculum learned by unsupervised clustering on deep features is proposed, according to which Co-Correcting can gradually update the labels, from easy to difficult, to improve the stability of the learning”, Li teaches splitting the training data into portions for each model)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 4, claim 1. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 5,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The computer system of claim 4,
Li further teaches:
wherein the retraining of the first machine learning model and the retraining of the second machine learning model are iteratively repeated for the number of iterations during which additional portions of the training data are selected (Li Pg. 4, Algorithm 1 shows the first and second machine learning models being retrained via updating weight parameters (at line 23), which is repeated for a number of iterations (at line 9), with additional portions of training data being selected as mini-batches at lines 10 and 11 during the same iteration)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 5, claim 4. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 7,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The computer system of claim 1,
Liu further teaches:
wherein the operations further comprise: identifying one or more first corrective data samples from the classification outputted by the first machine learning model ((Liu Pg. 4) “we also calculate the losses of potentially noisy samples, though the gradients are set to zero when updating the classification models. These losses are used to update the label distribution”, Liu Pg. 4, Fig. 4 shows backbone CNN-1 outputs its own losses, (Liu Pg. 4, Fig. 4 Caption) “We use the label distribution yd to replace the noisy label y-hat”, broadest reasonable interpretation of corrective data samples includes samples with losses that are used to update the distribution used for label correction)
and identifying one or more second corrective data samples from the classification outputted by the second machine learning model ((Liu Pg. 4) “we also calculate the losses of potentially noisy samples, though the gradients are set to zero when updating the classification models. These losses are used to update the label distribution”, Liu Pg. 4, Fig. 4 shows backbone CNN-2 outputs its own losses, (Liu Pg. 4, Fig. 4 Caption) “We use the label distribution yd to replace the noisy label y-hat”, broadest reasonable interpretation of corrective data samples includes samples with losses that are used to update the distribution used for label correction)
wherein the relabeling comprises: replacing a noisy label for at least one of the one or more [first noisy data samples] with a corrective label from the second corrective data samples ((Liu Pg. 4 Fig. 4 Caption) “We use the label distribution yd to replace the noisy label y-hat”, Liu Pg. 5 Equations 4 and 5 show yd is computed based off the loss of the first machine learning model, Li teaches a set of noisy data samples for the first machine learning model)
PNG
media_image4.png
104
433
media_image4.png
Greyscale
and replacing a noisy label for at least one of the one or more [second noisy data samples] with a corrective label from the first corrective data samples ((Liu Pg. 4 Fig. 4 Caption) “We use the label distribution yd to replace the noisy label y-hat”, Liu Pg. 5 Equations 4 and 5 show yd is computed based off the loss of the second machine learning model, Li teaches a set of noisy data samples for the second machine learning model)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 7, claim 1. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 8,
Li teaches A method comprising:
performing, by a computer system, a cross-label correction of a training data set using a first machine learning model and a second machine learning model, wherein the performing the cross-label correction comprises: (Li Pg. 3, Fig. 1 shows a cross-label correction being performed using two different neural networks, which are first and second machine learning models)
splitting, by the computer system, the training data set into a first portion and a second portion; ((Li Pg. 3) “At each epoch, we perform co-divide, where one network divides the noisy training dataset into a clean labeled set (X) and a noisy unlabeled set (U), which are then used by the other network”, broadest reasonable interpretation of two portions of the training data set includes two versions of the same dataset split differently into a clean portion and a noisy portion)
determining that [the training data set includes] a plurality of noisy data samples that increase the likelihood of overfitting the first and second machine learning models on the training data set; ((Li Pg. 1) “A recent study (Zhang et al., 2017) shows that DNNs can easily overfit to noisy labels and results in poor generalization performance”, Li does not explicitly teach determining that the training set includes noisy data)
determining one or more first noisy data samples from the first portion of the training data set and one or more second noisy data samples from the second portion of the training data set based on a first set of loss function values associated with the first portion and a second set of loss function values associated with the second portion; ((Li Pg. 3, Fig. 1 Caption) “At each epoch, a network models its per-sample loss distribution with a GMM to divide the dataset into a labeled set (mostly clean) and an unlabeled set (mostly noisy), which is then used as training data for the other network (i.e. co-divide)”, Li Pg. 3, Fig. 1 shows that both portions of the training data set, portion A used to train network A, and portion B used to train network B, contain noisy unlabeled data samples, which is determined by a modeled loss distribution for the respective network)
classifying, by the computer system and using the first machine learning model, the first portion of the training data set (Li Pg. 3, Fig. 1 shows that a first portion of the training data set is used to train network A, training a classification model includes classification of the training data)
selecting, by the computer system, the one or more first noisy data samples from the first portion based on the first set of loss function values ((Li Pg. 5) “Having acquired X-hat (and U-hat) which consists of multiple augmentations of labeled (unlabeled) samples and their refined (guessed) labels, we follow MixMatch to “mix” the data, where each sample is interpolated with another sample randomly chosen from the combined mini-batch of X-hat and U-hat”, broadest reasonable interpretation of selecting noisy data samples includes random sampling of a noisy data set, U-hat is a set of noisy data samples, Li Pg. 3, Fig. 1 shows that MixMatch selects from the dataset portion used with network B with a per-sample loss distribution for the network)
classifying, by the computer system and using the second machine learning model, the second portion of the training data set (Li Pg. 3, Fig. 1 shows that a second portion of the training data set is used to train network B, training a classification model includes classification of the training data)
selecting, by the computer system, one or more second noisy data samples from the second portion based on the second set of loss function values ((Li Pg. 5) “Having acquired X-hat (and U-hat) which consists of multiple augmentations of labeled (unlabeled) samples and their refined (guessed) labels, we follow MixMatch to “mix” the data, where each sample is interpolated with another sample randomly chosen from the combined mini-batch of X-hat and U-hat”, broadest reasonable interpretation of selecting noisy data samples includes random sampling of a noisy data set, U-hat is a set of noisy data samples, Li Pg. 3, Fig. 1 shows that MixMatch selects from the dataset portion used with network B with a per-sample loss distribution for the network)
classifying, by the computer system and using the first machine learning model, the one or more second noisy data samples (Li Pg. 3, Fig. 1 shows that the second noisy data samples are mixed into the input for the first machine learning model for training, which includes classification)
classifying, by the computer system and using the second machine learning model, the one or more first noisy data samples (Li Pg. 3, Fig. 1 shows that the first noisy data samples are mixed into the input for the second machine learning model for training, which includes classification)
performing, by the computer system, a number of iterations of the cross-label correction of the training data set, (Li Pg. 4, Algorithm 1 shows iterations of cross-label correction being performed at multiple points, such as a number of training epochs (at line 3), a number of iterations on two training sets (at line 9), and a number of iterations on mini-batches (at line 12))
and swapping, by the computer system based on the determining that the number of corrections has been performed, data samples between the first and second machine learning model in a subsequent iteration of the training the first and second machine learning models, (Li Pg. 4, Algorithm 1 shows that application of MixMatch, in which swapping of data samples between two data sets which are used to train two different machine learning models occurs, takes place only after it is determined that a number of mini-batch correction iterations from lines 12 to 22 (specifically B iterations) have been performed)
Liu teaches the following further limitation more explicitly than what is taught in Li:
relabeling at least one noisy sample of the [first] noisy data samples and at least one noisy sample of the [second] noisy data based on the classifying of the one or more [first and second] noisy data samples ((Liu Pg. 4, Fig. 4 shows how the losses from the classification of the first and second machine learning model (backbone CNN-1 and backbone CNN-2) are used to compute the distribution used to correct noisy labels, (Liu Pg. 4, Fig. 4 Caption) “We use the label distribution yd to replace the noisy label y-hat”, Li teaches two sets of noisy data samples)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li and Liu to use the method of training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, taught by Li, with the method including noisy label cross-correction taught by Liu, as Liu teaches (Liu Pg. 7) “the methods that do not update the label have a significant decrease in accuracy as the proportion of noises increases. These methods try to avoid learning noisy samples. Therefore as the noise ratio increases, the samples actually used for training will be reduced accordingly”, that is, noisy label correction allows for high classification accuracy even on very noisy datasets. Such a combination would be obvious.
Quader teaches the following further limitations that neither Li nor Liu teaches:
determining that the training data set includes a plurality of noisy data samples … ((Quader Abstract) “The processor(s) applies ensemble machine-learning and a generative model to the training dataset to detect noisy labeled datapoints in the training dataset”)
in response to completing the number of iterations of the cross-label correction, outputting, by the computer system, the training data set with the cross label correction to a machine learning classifier; ((Quader [0058]) “Program code repeats the datapoint label rectification process to gradually create a labeled dataset (DL) 423 (FIG. 4) as the union of the cleaned dataset Dclean and the actively-learned dataset DAL 806. Note that the classifier f 428 can be retrained for each iteration through phase 2 310. The phase outputs the completely cleaned dataset DL 432 as the completely cleaned version of the original noisy and incomplete dataset 402, along with the classifier f 428 (i.e., machine-learning model) trained using completely cleaned dataset DL 432”)
and training, by the computer system, the machine learning classifier of a machine learning system based on the training data set with the cross label correction ((Quader [0058]) “The phase outputs the completely cleaned dataset DL 432 as the completely cleaned version of the original noisy and incomplete dataset 402, along with the classifier f 428 (i.e., machine-learning model) trained using completely cleaned dataset DL 432”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, and Quader to use the method of training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, and the method including noisy label cross-correction, taught jointly by Li and Liu, with detecting the presence of noisy labels in a dataset, outputting the cleaned dataset, and using it to train a machine learning classifier, as taught by Quader, as it is well-known within the art that machine learning models trained on data with more accurate labels will themselves be more accurate when deployed. Such a combination would be obvious.
Bai teaches the following further limitation that neither Li, nor Liu, nor Quader teaches:
training, by the computer system, the [first] machine learning model [based on the first portion and the second machine learning model based on the second portion] utilizing a number of training epochs selected to minimize a likelihood of overfitting the [first and second] machine learning models on the training data set, ((Bai Pg. 3) “to alleviate the impact of noisy labels for latter layers, we reinitialize and progressively train latter DNN layers by using smaller numbers of epochs with preceding DNN layers fixed”, (Bai Pg. 2) “the memorization effect−if clean labels are of majority within the noisy labels for each class, deep networks tend to first memorize and fit majority (clean) patterns and then overfit minority (noisy) patterns”, Li and Liu teach first and second machine learning models with first and second portions of the training data set)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, and Bai to use the method of training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, noisy label cross-correction, and detecting the presence of noisy labels and outputting a corrected training data set, taught jointly by Li, Liu, and Quader, and adding the use of selecting a number of training epochs that likely do not overfit the training data, taught by Bai, as Bai teaches: (Bai Pg. 2) “As deep networks have large model capacities, they can easily memorize and eventually overfit the noisy labels, leading to poor generalization performance [38]. Therefore, it is of great importance to develop a methodology that is robust to noisy annotations”. Such a combination would be obvious.
Zhang-2018 teaches the following further limitation that neither Li, nor Liu, nor Quader, nor Bai teaches:
wherein the performing the number of iterations includes: determining, by the computer system, that a number of corrections to labels in the training data set has been performed, ((Zhang-2018 Pg. 7) “The BC baseline noise correction algorithm adopts the framework in [43], namely, self-training semisupervised noise correction. The potential noises are first filtered out from the original training set as unlabeled instances. The remaining training set is used to build a learning model. All unlabeled instances (noises) are predicted by this learning model, but only a small portion are confirmed as repaired with high confidence. The repaired instances are removed from the unlabeled set and merged into the training set for the next iteration of correction. The remaining unlabeled instances will be further corrected in the next iteration. The correction loop stops when the number of iterations reaches its predefined maximum value or all noises are confirmed”, iterative label correction that stops when all noisy labels are confirmed to be repaired corresponds to determining that a number of corrections to labels have been performed)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, Bai, and Zhang-2018 to use the method of training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, noisy label cross-correction, detecting the presence of noisy labels and outputting a corrected training data set, and selecting a number of training epochs that likely do not overfit the training data, taught jointly by Li, Liu, Quader, and Bai, and including, during performance of a number of iterations, that a number of corrections to labels have been performed, taught by Zhang-2018, as doing so allows for the iterative process to be stopped once a satisfactory amount of labels have relabeled, such as all of the noisy labels, which provides the predictable benefit of improving computational efficiency by not performing unnecessary iterations. Such a combination would be obvious.
Regarding claim 9,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The method of claim 8,
Li further teaches:
wherein the selecting the one or more first noisy data samples from the first portion comprises determining a number of noisy data samples from the first portion ((Li Pg. 3) “At each epoch, we perform co-divide, where one network divides the noisy training dataset into a clean labeled set (X) and a noisy unlabeled set (U), which are then used by the other network”), that have a largest loss value for a loss function that measures a performance of the classifying the first portion using the first machine learning model ((Li Pg. 3) “Following Arazo et al. (2019), we aim to find the probability of a sample being clean by fitting a mixture model to the per-sample loss distribution”, (Li Pg. 3) “For each sample, its clean probability wi is the posterior probability...We divide the training data into a labeled set and an unlabeled set by setting a threshold τ on wi”, Li Pg. 3 Fig. 1 shows that there is a first portion of noisy data samples)
and wherein the selecting the one or more second noisy data samples from the second portion comprises determining a number of noisy data samples from the second portion ((Li Pg. 3) “At each epoch, we perform co-divide, where one network divides the noisy training dataset into a clean labeled set (X) and a noisy unlabeled set (U), which are then used by the other network”) that have a largest loss value for a loss function that measures a performance of the classifying the second portion using the second machine learning model ((Li Pg. 3) “Following Arazo et al. (2019), we aim to find the probability of a sample being clean by fitting a mixture model to the per-sample loss distribution”, (Li Pg. 3) “For each sample, its clean probability wi is the posterior probability...We divide the training data into a labeled set and an unlabeled set by setting a threshold τ on wi”, Li Pg. 3 Fig. 1 shows that there is a second portion of noisy data samples)
At the time of filing, one of ordinary skill in the art would have motivation to combine the method jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 9, claim 8. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 11,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The method of claim 8,
Li further teaches:
wherein the swapping, by the computer system, the data samples is further based on a condition for performing the cross-label correction (Li Pg. 4, Algorithm 1 shows that the swapping of data samples at line 25 is only performed on the condition that the maximum number of iterations at line 9 hasn’t been reached)
At the time of filing, one of ordinary skill in the art would have motivation to combine the method jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 11, claim 8. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 12,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The method of claim 8,
Li further teaches:
further comprising determining, by the computer system, additional portions of the training data set for training the first machine learning model and the second machine learning model after the [swapping] ((Quader [0075]) “To accomplish this task, dataset Dtrain can be split into a labeled training dataset Dlabeled of size w and a data pool Dpool containing the remaining points. Then, Dlabeled can be used to train c and produce a model md that is used to provide predictions to the datapoints in Dtest and estimate the corresponding classifications loss Ld. After that, another datapoint x from the pool Dpool can be randomly selected and added to the Dlabeled to form a new dataset Dx = Dlabeled U {X}. After that, Dx is utilized to train c again, create a new model mx, and test this model using Dtest”, Li teaches swapping)
At the time of filing, one of ordinary skill in the art would have motivation to combine the method jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 12, claim 8. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 13,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The method of claim 8,
Liu further teaches:
wherein the at least one noisy sample of the first noisy data samples is relabeled to have a corresponding corrective label outputted from the classification of the [first] noisy data samples using the second machine learning model ((Liu Pg. 4 Fig. 4 Caption) “We use the label distribution yd to replace the noisy label y-hat”, Liu Pg. 5 Equations 4 and 5 show yd is computed based off the loss of the second machine learning model, Li teaches a set of noisy data samples for the second machine learning model and swapping data samples from a first set into the set used to train the second machine learning model)
and wherein the at least one noisy sample of the second noisy data samples is relabeled to have a corresponding corrective label outputted from the classification of the [second] noisy data samples using the first machine learning model ((Liu Pg. 4 Fig. 4 Caption) “We use the label distribution yd to replace the noisy label y-hat”, Liu Pg. 5 Equations 4 and 5 show yd is computed based off the loss of the first machine learning model, Li teaches a set of noisy data samples for the first machine learning model and swapping data samples from a second set into the set used to train the first machine learning model)
At the time of filing, one of ordinary skill in the art would have motivation to combine the method jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 13, claim 8. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 16,
Li teaches A non-transitory machine-readable medium having instructions stored thereon, wherein the instructions are executable to cause a machine of a system to perform operations comprising: ((Li Abstract) “Code is available at [link]”)
performing a cross-label correction of a training data set using a first machine learning model and a second machine learning model, wherein the performing the cross-label correction comprises: (Li Pg. 3, Fig. 1 shows a cross-label correction being performed using two different neural networks, which are first and second machine learning models)
accessing the first machine learning model trained using a first portion of the training data set, and the second machine learning model trained using a second portion of the training data set, (Li Pg. 3, Fig. 1 shows different portions of the training data are used to train networks A and B after the co-divide step, broadest reasonable interpretation of two portions of the training data set includes two versions of the same dataset split differently into a clean portion and a noisy portion, broadest reasonable interpretation of accessing a trained machine learning model includes continuing to train a machine learning model that has already undergone one or more iterations of training)
determining that [the training data set includes] a plurality of noisy data samples that increase the likelihood of overfitting the first and second machine learning models on the training data set; ((Li Pg. 1) “A recent study (Zhang et al., 2017) shows that DNNs can easily overfit to noisy labels and results in poor generalization performance”, Li does not explicitly teach determining that the training set includes noisy data)
determining one or more first noisy data samples from the first portion of the training data set and one or more second noisy data samples from the second portion of the training data set based on a first set of loss function values associated with the first portion and a second set of loss function values associated with the second portion; ((Li Pg. 3, Fig. 1 Caption) “At each epoch, a network models its per-sample loss distribution with a GMM to divide the dataset into a labeled set (mostly clean) and an unlabeled set (mostly noisy), which is then used as training data for the other network (i.e. co-divide)”, Li Pg. 3, Fig. 1 shows that both portions of the training data set, portion A used to train network A, and portion B used to train network B, contain noisy unlabeled data samples, which is determined by a modeled loss distribution for the respective network)
classifying the first portion of the training data set using the first machine learning model (Li Pg. 3, Fig. 1 shows a portion of the training data is used to train network A after the co-divide step, training a classifier includes classification of the training data)
selecting one or more first noisy data samples from the first portion based on the first set of loss function values ((Li Pg. 3) “At each epoch, we perform co-divide, where one network divides the noisy training dataset into a clean labeled set (X) and a noisy unlabeled set (U), which are then used by the other network”, Li Pg. 3, Fig. 1 shows the portion used by the network A has its own noisy set U and its own per-sample loss distribution)
classifying the second portion of the training data set using the second machine learning model (Li Pg. 3, Fig. 1 shows a portion of the training data is used to train network B after the co-divide step, training a classifier includes classification of the training data)
selecting one or more second noisy data samples from the second portion based on the second set of loss function values ((Li Pg. 3) “At each epoch, we perform co-divide, where one network divides the noisy training dataset into a clean labeled set (X) and a noisy unlabeled set (U), which are then used by the other network”, Li Pg. 3, Fig. 1 shows the portion used by the network B has its own noisy set U and its own per-sample loss distribution)
classifying the one or more second noisy data samples using the first machine learning model (Li Pg. 3, Fig. 1 shows that the second noisy data samples are mixed into the input for the first machine learning model for training, which includes classification)
classifying the one or more first noisy data samples using the second machine learning model (Li Pg. 3, Fig. 1 shows that the first noisy data samples are mixed into the input for the second machine learning model for training, which includes classification)
performing a number of iterations of the cross-label correction of the training data set, (Li Pg. 4, Algorithm 1 shows iterations of cross-label correction being performed at multiple points, such as a number of training epochs (at line 3), a number of iterations on two training sets (at line 9), and a number of iterations on mini-batches (at line 12))
and swapping, based on the determining that the number of corrections has been performed, data samples between the first and second machine learning model in a subsequent iteration of the training the first and second machine learning models, (Li Pg. 4, Algorithm 1 shows that application of MixMatch, in which swapping of data samples between two data sets which are used to train two different machine learning models occurs, takes place only after it is determined that a number of mini-batch correction iterations from lines 12 to 22 (specifically B iterations) have been performed)
Liu teaches the following further limitation more explicitly than what is taught in Li:
relabeling at least one noisy sample of the [first] noisy data samples and at least one noisy sample of the [second] noisy data based on the classifying of the one or more [first and second] noisy data samples ((Liu Pg. 4, Fig. 4 shows how the losses from the classification of the first and second machine learning model (backbone CNN-1 and backbone CNN-2) are used to compute the distribution used to correct noisy labels, (Liu Pg. 4, Fig. 4 Caption) “We use the label distribution yd to replace the noisy label y-hat”, Li teaches two sets of noisy data samples)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li and Liu to use the medium with instructions for training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, taught by Li, with the method including noisy label cross-correction taught by Liu, as Liu teaches (Liu Pg. 7) “the methods that do not update the label have a significant decrease in accuracy as the proportion of noises increases. These methods try to avoid learning noisy samples. Therefore as the noise ratio increases, the samples actually used for training will be reduced accordingly”, that is, noisy label correction allows for high classification accuracy even on very noisy datasets. Such a combination would be obvious.
Quader teaches the following further limitations that neither Li nor Liu teaches:
determining that the training data set includes a plurality of noisy data samples … ((Quader Abstract) “The processor(s) applies ensemble machine-learning and a generative model to the training dataset to detect noisy labeled datapoints in the training dataset”)
in response to completing the number of iterations of the cross-label correction, outputting the training data set with the cross label correction to a machine learning classifier; ((Quader [0058]) “Program code repeats the datapoint label rectification process to gradually create a labeled dataset (DL) 423 (FIG. 4) as the union of the cleaned dataset Dclean and the actively-learned dataset DAL 806. Note that the classifier f 428 can be retrained for each iteration through phase 2 310. The phase outputs the completely cleaned dataset DL 432 as the completely cleaned version of the original noisy and incomplete dataset 402, along with the classifier f 428 (i.e., machine-learning model) trained using completely cleaned dataset DL 432”)
and training the machine learning classifier of a machine learning system based on the training data set with the cross label correction ((Quader [0058]) “The phase outputs the completely cleaned dataset DL 432 as the completely cleaned version of the original noisy and incomplete dataset 402, along with the classifier f 428 (i.e., machine-learning model) trained using completely cleaned dataset DL 432”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, and Quader to use the medium of training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, and the medium including noisy label cross-correction, taught jointly by Li and Liu, with detecting the presence of noisy labels in a dataset, outputting the cleaned dataset, and using it to train a machine learning classifier, as taught by Quader, as it is well-known within the art that machine learning models trained on data with more accurate labels will themselves be more accurate when deployed.
Bai teaches the following further limitation that neither Li, nor Liu, nor Quader teaches:
wherein the [first and second] machine learning models are trained utilizing a number of training epochs selected to minimize a likelihood of overfitting the [first and second] machine learning models on the training data set; ((Bai Pg. 3) “to alleviate the impact of noisy labels for latter layers, we reinitialize and progressively train latter DNN layers by using smaller numbers of epochs with preceding DNN layers fixed”, (Bai Pg. 2) “the memorization effect−if clean labels are of majority within the noisy labels for each class, deep networks tend to first memorize and fit majority (clean) patterns and then overfit minority (noisy) patterns”, Li and Liu teach first and second machine learning models with first and second portions of the training data set)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, and Bai to use the medium with instructions for training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, noisy label cross-correction, and detecting the presence of noisy labels and outputting a corrected training data set, taught jointly by Li, Liu, and Quader, and adding the use of selecting a number of training epochs that likely do not overfit the training data, taught by Bai, as Bai teaches: (Bai Pg. 2) “As deep networks have large model capacities, they can easily memorize and eventually overfit the noisy labels, leading to poor generalization performance [38]. Therefore, it is of great importance to develop a methodology that is robust to noisy annotations”. Such a combination would be obvious.
Zhang-2018 teaches the following further limitation that neither Li, nor Liu, nor Quader, nor Bai teaches:
wherein the performing the number of iterations includes: determining, based on the relabeling, that a number of corrections to labels in the training data set has been performed, ((Zhang-2018 Pg. 7) “The BC baseline noise correction algorithm adopts the framework in [43], namely, self-training semisupervised noise correction. The potential noises are first filtered out from the original training set as unlabeled instances. The remaining training set is used to build a learning model. All unlabeled instances (noises) are predicted by this learning model, but only a small portion are confirmed as repaired with high confidence. The repaired instances are removed from the unlabeled set and merged into the training set for the next iteration of correction. The remaining unlabeled instances will be further corrected in the next iteration. The correction loop stops when the number of iterations reaches its predefined maximum value or all noises are confirmed”, iterative label correction that stops when all noisy labels are confirmed to be repaired corresponds to determining that a number of corrections to labels have been performed)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, Bai, and Zhang-2018 to use the medium with instructions for training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, noisy label cross-correction, detecting the presence of noisy labels and outputting a corrected training data set, and selecting a number of training epochs that likely do not overfit the training data, taught jointly by Li, Liu, Quader, and Bai, and including, during performance of a number of iterations, that a number of corrections to labels have been performed, taught by Zhang-2018, as doing so allows for the iterative process to be stopped once a satisfactory amount of labels have relabeled, such as all of the noisy labels, which provides the predictable benefit of improving computational efficiency by not performing unnecessary iterations. Such a combination would be obvious.
Regarding claim 17,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The non-transitory machine-readable medium of claim 16,
Li further teaches:
wherein the classifying the one or more second noisy data samples using the first machine learning model ((Li Pg. 5) “we use the ensemble of predictions from both networks to “co-guess” the labels for unlabeled samples (algorithm 1, line 20), which can produce more reliable guessed labels”), results in a first confidence score that exceeds a second confidence score, wherein the second confidence score is based on the classifying using the second machine learning model, the one or more second noisy data samples as part of the second portion (Li Pg. 4 Algorithm 1, line 20 shows averaging of the prediction probabilities to compute a label, (Li Pg. 3) “where pcmodel is the model’s output softmax probability for class c”, the first model’s prediction probability could exceed that of the second’s, broadest reasonable interpretation of a confidence score includes a prediction probability)
and wherein the one or more second noisy data samples are [relabeled] using one or more corresponding corrective labels provided by the classifying the one or more second noisy data samples using the first machine learning model (Li Pg. 4 Algorithm 1, line 20 shows averaging of the prediction probabilities to compute a label, (Li Pg. 3) “where pcmodel is the model’s output softmax probability for class c”, the first model’s prediction probability could exceed that of the second’s, in which case the first model’s label is used since the probabilities are averaged, Liu teaches relabeling)
At the time of filing, one of ordinary skill in the art would have motivation to combine the method jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 17, claim 16. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 18,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The non-transitory machine-readable medium of claim 17,
Li further teaches:
wherein the classifying the one or more first noisy data samples using the second machine learning model ((Li Pg. 5) “we use the ensemble of predictions from both networks to “co-guess” the labels for unlabeled samples (algorithm 1, line 20), which can produce more reliable guessed labels”), results in a third confidence score that exceeds a fourth confidence score, wherein the fourth confidence score is based on the classifying, using the first machine learning model, the one or more first noisy data samples as part of the first portion (Li Pg. 4 Algorithm 1, line 20 shows averaging of the prediction probabilities to compute a label, (Li Pg. 3) “where pcmodel is the model’s output softmax probability for class c”, the second model’s prediction probability could exceed that of the first’s, broadest reasonable interpretation of a confidence score includes a prediction probability)
and wherein the one or more first noisy data samples are [relabeled] using one or more corresponding corrective labels provided by the classifying the one or more first noisy data samples using the second machine learning model (Li Pg. 4 Algorithm 1, line 20 shows averaging of the prediction probabilities to compute a label, (Li Pg. 3) “where pcmodel is the model’s output softmax probability for class c”, the second model’s prediction probability could exceed that of the first’s, in which case the second model’s label is used since the probabilities are averaged, Liu teaches relabeling)
At the time of filing, one of ordinary skill in the art would have motivation to combine the method jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 18, claim 17. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 19,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The non-transitory machine-readable medium of claim 16,
Li further teaches:
wherein the swapping the data samples is further based on a condition for performing the cross-label correction (Li Pg. 4, Algorithm 1 shows that the swapping of data samples at line 25 is only performed on the condition that the maximum number of iterations at line 9 hasn’t been reached)
At the time of filing, one of ordinary skill in the art would have motivation to combine the method jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 for the parent claim of claim 19, claim 16. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Claims 6, 10, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Li, further in view of Liu, further in view of Quader, further in view of Bai, further in view of Zhang-2018, further in view of Yu et al. “How does Disagreement Help Generalization against Label Corruption?”, hereinafter Yu.
Regarding claim 6,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The computer system of claim 5,
Yu teaches the following further limitations that neither Li, nor Liu, nor Quader, nor Bai, nor Zhang-2018 explicitly teach:
wherein a percentage for selection of the one or more first noisy data samples relative to all data samples in the [first portion] is reduced in each iteration of the number of iterations, ((Yu Pg. 4) “Thus, at the beginning of training, we keep more small-loss data (with a large λ(e)) in each mini-batch, which is equivalent to dropping less data. Since deep networks will fit clean data first, noisy data do not matter at the initial training epochs. With the increase of epochs, we keep less small-loss data (with a small λ(e)) in each mini-batch. As deep networks will over-fit noisy data gradually, we should drop more data. The gradual decrease of λ(e) prevents deep networks over-fitting noisy data to some degree”, Li teaches a first data portion)
and wherein a percentage for selection of the one or more second noisy data samples relative to all data samples in the [second portion] is reduced in each iteration of the number of iterations ((Yu Pg. 4) “Thus, at the beginning of training, we keep more small-loss data (with a large λ(e)) in each mini-batch, which is equivalent to dropping less data. Since deep networks will fit clean data first, noisy data do not matter at the initial training epochs. With the increase of epochs, we keep less small-loss data (with a small λ(e)) in each mini-batch. As deep networks will over-fit noisy data gradually, we should drop more data. The gradual decrease of λ(e) prevents deep networks over-fitting noisy data to some degree”, Li teaches a second data portion)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, Bai, Zhang-2018, and Yu to use the system of training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, correcting labels, outputting the corrected dataset to train a machine learning model, and choosing a number of training epochs to reduce overfitting, jointly taught by Li, Liu, Quader, Bai, and Zhang-2018, with the method for decreasing the amount of designated noisy labels after every training iteration taught by Yu, as Yu teaches (Yu Pg. 4) “The gradual decrease of λ(e) prevents deep networks over-fitting noisy data to some degree”. Such a combination would be obvious.
Regarding claim 10,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The method of claim 9,
Yu teaches the following further limitation that neither Li, nor Liu, nor Quader, nor Bai, nor Zhang-2018 explicitly teaches:
wherein the number of noisy data samples from the [first] portion and the number of noisy data samples from the [second] portion are derived from a sampling percentage that decreases with each iteration [of the relabeling] ((Yu Pg. 4) “Thus, at the beginning of training, we keep more small-loss data (with a large λ(e)) in each mini-batch, which is equivalent to dropping less data. Since deep networks will fit clean data first, noisy data do not matter at the initial training epochs. With the increase of epochs, we keep less small-loss data (with a small λ(e)) in each mini-batch. As deep networks will over-fit noisy data gradually, we should drop more data. The gradual decrease of λ(e) prevents deep networks over-fitting noisy data to some degree”, broadest reasonable interpretation of a sampling percentage includes an amount of data chosen for each mini-batch, Li teaches a first and second data portion, Liu teaches relabeling)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, Bai, Zhang-2018, and Yu to use the method for training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, correcting labels, outputting the corrected dataset, using it to train a machine learning model, and choosing a number of training epochs to reduce overfitting, jointly taught by Li, Liu, Quader, Bai, and Zhang-2018 with the method for decreasing the amount of designated noisy labels after every training iteration taught by Yu, as Yu teaches (Yu Pg. 4) “The gradual decrease of λ(e) prevents deep networks over-fitting noisy data to some degree”. Such a combination would be obvious.
Regarding claim 15,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The method of claim 8,
Yu teaches the following further limitation that neither Li, nor Liu, nor Quader, nor Bai, nor Zhang-2018 explicitly teaches:
wherein the training the first machine learning model and the second machine learning model using the training data set is performed using a predefined number of epochs as a hyperparameter ((Yu Pg. 5) “Adam optimizer (momentum=0.9) is with an initial learning rate of 0.001, and the batch size is set to 128 and we run 200 epochs”), that prevents an initial overfitting to the training data set ((Yu Pg. 5) “Over the increase of epochs, deep network will over-fit noisy labels gradually, which decreases its test accuracy accordingly. Thus, a robust training method should alleviate or even stop the decreasing trend in test accuracy”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, Bai, Zhang-2018, and Yu to use the method for training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, correcting labels, outputting the corrected dataset to train a machine learning model, and choosing a number of training epochs to reduce overfitting, jointly taught by Li, Liu, Quader, Bai, and Zhang-2018, with the method for limiting overfitting by using a specific predefined number of epochs as a hyperparameter, taught by Yu, as using a limited number of training epochs is a well-known technique within the art for preventing a model from overfitting to training data. Such a combination would be obvious.
Claims 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Li, further in view of Liu, further in view of Quader, further in view of Bai, further in view of Zhang-2018, further in view of Zhang-2021 et al. “Fraud Detection under Multi-Sourced Extremely Noisy Annotations”, hereinafter Zhang-2021.
Regarding claim 14,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The method of claim 8,
Zhang-2021 teaches the following further limitation that neither Li, nor Liu, nor Quader, nor Bai, nor Zhang-2018 explicitly teaches:
wherein the training data set comprises training examples corresponding to electronic service transactions ((Zhang-2021 Pg. 2) “We collect more than a million transaction cases in two different real-world fraud scenarios from Alipay, which are related to credit card theft (CCT) and promotion abuse fraud (PAF), respectively. With multi-sourced noisy labels automatically assigned by the aforementioned annotation process, we construct two corresponding fraud detection datasets”), that are labeled as either fraudulent or legitimate ((Zhang-2021 Pg. 3) “where yki = -1 for a genuine transaction and where yki = +1 for a fraudulent transaction”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, Bai, Zhang-2018, and Zhang-2021 to use the method for training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, correcting labels, outputting a corrected dataset to train a machine learning model, and choosing a number of training epochs to reduce overfitting, jointly taught by Li, Liu, Quader, Bai, and Zhang-2018, with the dataset consisting of electronic service transactions that are labeled as fraudulent or legitimate taught by Zhang-2021, as Zhang-2021 teaches (Zhang-2021 Abstract) “the true labels for training a supervised fraud detection model are difficult to collect in many real-world cases. To circumvent this issue, a series of automatic annotation techniques are employed instead in generating multiple noisy annotations for each unknown activity”, and (Zhang-2021 Pg. 1) “Therefore, it is highly demanded to devise a reliable learning approach with massive multi-sourced noisy annotations for accurate fraud detection in e-commerce”. Such a combination would be obvious.
Regarding claim 20,
Li, Liu, Quader, Bai, and Zhang-2018 jointly teach The non-transitory machine-readable medium of claim 16,
Zhang-2021 teaches the following further limitations that neither Li, nor Liu, nor Quader, nor Bai, nor Zhang-2018 explicitly teach:
wherein the training data set comprises electronic service transactional data wherein the training data set comprises training examples corresponding to electronic service transactions ((Zhang-2021 Pg. 2) “We collect more than a million transaction cases in two different real-world fraud scenarios from Alipay, which are related to credit card theft (CCT) and promotion abuse fraud (PAF), respectively. With multi-sourced noisy labels automatically assigned by the aforementioned annotation process, we construct two corresponding fraud detection datasets”)
and wherein the relabeling comprises relabeling a label corresponding to a fraudulent transaction for at least one noisy sample to a corrective label corresponding to a legitimate transaction ((Zhang-2021 Pg. 3) “where yki = -1 for a genuine transaction and where yki = +1 for a fraudulent transaction”, (Zhang-2021 Pg. 5) “Finally, we obtain the output of label correction stage, namely, the corrected labels”, correcting the labels for a set of fraudulent and legitimate transactions will include correcting a label from indicating a fraudulent transaction to indicating a legitimate transaction)
At the time of filing, one of ordinary skill in the art would have motivation to combine Li, Liu, Quader, Bai, Zhang-2018, and Zhang-2021 to use the medium for training a pair of machine learning models with noisy labels, including splitting the training data into portions for each model, correcting labels, outputting a corrected dataset to train a machine learning model, and choosing a number of training epochs to reduce overfitting, jointly taught by Li, Liu, Quader, Bai, and Zhang-2018, with the dataset consisting of electronic service transactions that are labeled as fraudulent or legitimate, and correcting incorrect noisy labels indicating fraud, taught by Zhang-2021, as Zhang-2021 teaches (Zhang-2021 Abstract) “the true labels for training a supervised fraud detection model are difficult to collect in many real-world cases. To circumvent this issue, a series of automatic annotation techniques are employed instead in generating multiple noisy annotations for each unknown activity”, and (Zhang-2021 Pg. 1) “Therefore, it is highly demanded to devise a reliable learning approach with massive multi-sourced noisy annotations for accurate fraud detection in e-commerce”. Such a combination would be obvious.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Li et al. (U.S. Patent No. 11,599,792) teaches learning two machine learning models from data with noisy labels.
Han et al. “Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels” also teaches learning two machine learning models from data with noisy labels.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VICTOR A NAULT whose telephone number is (703) 756-5745. The examiner can normally be reached M - F, 12 - 8.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/V.A.N./Examiner, Art Unit 2124
/Kevin W Figueroa/Primary Examiner, Art Unit 2124
Prosecution Insights