DETAILED ACTION
Claims 1-20 are presented for examination.
Claims 1-6, 8-10, 12, 14-17, 19 and 20 were amended.
This is a Final Action.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
101 abstract idea has been obviated due to current amendment to the claims, specifically, the claim integrates abstract idea concepts into a practical application by reciting a specific technique for generating intentionally mislabeled training data, evaluating model efficacy using classification scores associated with the intentionally mislabeled data a conditionally retraining the machine learning model based on the evaluation.
In view of compact prosecution, examiner respectfully requests that the applicant modify claim 8 to incorporate limitations from claim 1 so claim 8 recites all the limitations contained in claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 8, 9 10, 11 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (US 12,482,242) further in view of Forman et al. (US 7,792,353) further in view of Basak et al. (US 2022/0335335)
1. Zhou teaches, A system, comprising:
a non-transitory memory; and one or more hardware processors coupled with the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations (Claim 19 – teaches computer system with one or more processors and a non-transitory computer-readable medium having instructions encoded thereon, Zhou) comprising:
obtaining first training data for training a machine learning model configured to classify data sets into a first classification or a second classification (Claim 1 - teaches obtaining a plurality of labeled samples… each labeled with a ground-truth label – thus teaching labeled training data for a classifier having multiple classifications, Zhou), wherein the training data comprises a first group of data sets and a second group of data sets (Claim 1 – dividing the plurality of labeled samples into plurality of training subsets and hold-out test subsets, Zhou);
training the machine learning model using the training data (Claim 1 – teaches training a machine learning model using a corresponding training subset, Zhou);
obtaining, from the trained machine learning model, a first plurality of classification scores based on the first group of data sets (Claim 1 – teaches each prediction label has a confidence score indicating a likelihood that the prediction label is correct – thus teaching obtaining confidence scores from the trained model, Zhou) and a second plurality of classification scores based on the second group of data sets (Claim 1 – teaches prediction label has a confidence score – thus teaching confidence scores are obtained for all labeled samples, Zhou); and
evaluating an efficacy of the machine learning model in detecting the mislabeled data sets (Abstract, claim 1- teaches each prediction label has a confidence score indicating a likelihood that the prediction label is correct… the system determines whether the candidate mislabel is a mislabel based in part on a confidence score of the prediction label. Under BRI this constitutes evaluating the effectiveness (i.e. efficacy) of the ML model in detecting mislabeled data sets, Zhou);
Zhou does not explicitly teach,
wherein each data set in the first group of data sets is labeled with the first classification, and wherein each data set in the second group of data sets is labeled with the second classification;
generating second training data comprising mislabeled data sets based on modifying a classification label associated with a first subset of data sets from the first group of data sets;
training the machine learning model using first training data and second training data;
… and a third plurality of classification scores based on the mislabeled data sets;
…the mislabeled data sets in the second training data based on the third plurality of classification scores;
determining that the efficacy of the machine learning is detecting the mislabeled data sets is below a threshold;
and
modifying the first group of data sets and the second group of data sets based on the first plurality of classification scores and the second plurality of classification scores, wherein the modifying comprises relabeling at least one data set in the second group of data sets from the second classification to the first classification; and re-training the machine learning model based on the modified first group of data sets and the modified second group of data sets.
However, Forman teaches,
wherein each data set in the first group of data sets is labeled with the first classification and wherein each data set in the second group of data sets is labeled with the second classification (Col 10: lines 25-33 – teaches positive samples with be mostly gathered together, and group-select them and label them positive (first group / first classification) and same for the negative samples thus teaching group selection and labeling negative (second group / second classification, Forman); and
modifying the first group of data sets and the second group of data sets based on the first plurality of classification scores and the second plurality of classification scores (Fig 8, Col 10: lines 25-33 - teaches samples are selected for labeling, sorted by their prediction strength (e.g., probability of belonging to the positive class according to the current classifiers)… the positive samples will be mostly gathered together… making it easier for the user to group-select them and label them positive (same for the negative samples) – thus disclosing modifying groups of data samples, where the grouping and modification are based on classification scores produced by the classifier, Forman), wherein the modifying comprises relabeling at least one data set in the second group of data sets from the second classification to the first classification (Col 10: lines 2-34 and Fig 8:138 - teaches relabeling samples by assigning a positive label to group samples and assigning negative labels to other grouped samples, thereby changing the classification of at least some samples accordingly – thus teaching relabeling samples from one classification to another, Forman); and
re-training the machine learning model based on the modified first group of data sets and the modified second group of data sets (Fig 8:138 & Col 10: lines 16-21 – teaches retraining the classified based on modified training set and lopping back to reprocess samples as explicitly cross-referenced in Fig 2, Forman).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to modify Zhou’s system to relabel identified mislabeled samples and retrain the classifier as taught by Forman because both prior arts are in the same field of endeavor of supervised ML classification with label quality issues, thereby would improve classification accuracy and robustness to mislabeled data.
Basak teaches,
generating second training data comprising mislabeled data sets based on modifying a classification label associated with a first subset of data sets from the first group of data sets (Paragraph 29 - teaches the system 100 introduces fake data samples in the training data, and F is a set of fake samples which have been intentionally mislabeled and the fake samples are added to the unmodified training data, Basak);
training the machine learning model using first training data and the second training data (Paragraph 29 – teaches the fake samples are added to the unmodified training data. For explanation purposes, the training data containing the fake samples is termed as ‘modified training data.’ The system 100 then trains the data driven model using the modified training data, Basak);
… and a third plurality of classification scores based on the mislabeled data sets (Paragraph 29 - teaches the system 100 then chooses a threshold of absolute difference of probability scores/logits corresponding to the tagged labels and the predicted labels of the fake samples, and further defining logit value for jth fake sample… for class k, Basak);
…the mislabeled data sets in the second training data based on the third plurality of classification scores; determining that the efficacy of the machine learning is detecting the mislabeled data sets is below a threshold (Claim 1 and Paragraph 29 – teaches providing intentionally fake samples included in the training data; obtaining probability scores or logits corresponding specifically to the intentionally mislabeled fake samples and choosing a threshold of absolute difference of probability scores/logits corresponding to the tagged labels and the predicted labels of the fake samples and selecting samples having values below the threshold as candidate mislabels , Basak).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to modify the combination of Zhou and Forman’s systems to include Basak’s technique of introducing known intentionally mislabeled samples in the training data, training the model outputs associated with the intentionally mislabeled samples and evaluating the model outputs associated with the intentionally mislabeled samples. A POSITA would have been motivated to make this modification to provide a known benchmark for assessing the reliability of the classifier’s mislabel determinations before relying on those determinations to modify the original training data and retrain the model, thereby reducing erroneous relabeling and predictably improving the accuracy and robustness of the subsequently trained classifier.
3. The combination of Zhou, Forman and Basak teaches, The system of claim 1, evaluating the efficacy of the machine learning model further comprises:
determining that the trained machine learning model incorrectly classifies one data sets (Claim 1 and Col 8: lines 43-46 – teaches determining whether the candidate mislabel is a mislabel & Col 11: lines 38-40, Zhou) based on the third plurality of classification scores (Claim 1, Col 6: lines 15-18– teaches each prediction label has a confidence score indicating a likelihood that the prediction label is correct, and determining whether the candidate mislabel is a mislabel based in part on the confidence score, Zhou; Abstract – teaches the one or more mislabeled data sets, Basak).
Claim 8 is similar to claim 1 hence rejected similarly.
Claim 9 is similar to the combination of claim 1 and 3 hence rejected similarly.
10. The combination of Zhou, Forman and Basak teaches, The method of claim 9, wherein the plurality of classification scores comprises a first set of classification scores obtained from the machine learning model based on the first set of training data (Claim 1 - teaches each prediction label has a confidence score indicating a likelihood that the prediction label is correct – discloses classification scores (confidence scores) output by a trained machine learning model for labeled training samples, Zhou), and wherein the method further comprises:
determining a threshold based on the first set of classification scores (Claim 1 - teaches determining whether the candidate mislabel is a mislabel based in part on the confidence score – discloses using confidence scores to establish decision criteria for identifying candidate mislabels; such criteria inherently operate as a threshold on the confidence scores, Zhou), wherein the determining that the portion of the training data has been mislabeled is based on the threshold (Claim 1 – identifying candidate mislabeled samples and determining whether the candidate mislabel is a mislabel based on confidence scores – discloses that mislabel determination is made by comparing confidence scores against a decision criterion (threshold), Zhou).
11. The combination of Zhou, Forman and Basak teaches, The method of claim 10, wherein the threshold corresponds to at least one of a highest classification score (Col 10: lines 25-29 - teaches samples are sorted by their prediction strength (e.g. probability of belonging to the positive class according to the current classifier – discloses sorting by prediction strength explicitly identifies samples with the highest classification scores, which operates as an upper-end threshold, Forman) or a lowest classification score (Col 10: lines 29-33 – teaches the positive samples will be mostly gathered together … (same for the negative samples), with a few individual clicks to treat the exceptions – discloses sorting necessarily places lowest-confidence / lowest-score samples at the opposite end, which operate as a lower-end threshold, Forman) in the first set of classification scores (Claim 1 – each prediction label has a confidence score indicating a likelihood that the prediction label is correct – discloses that classification scores are generated for the training data, Zhou and Forman operates on such scores).
The combination of claim 15 is similar to claim 1 hence rejected similarly.
Claims 2, 13 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (US 12,482,242) further in view of Forman et al. (US 7,792,353) and Basak et al. (US 2022/0335335) further in view of Tickoo et al. (US 2015/0117766)
All the limitations of claim 1 are taught above.
2. The combination of Zhou, Forman and Basak do not explicitly teach, wherein the training the machine learning model is based on an objective function that minimizes a within-group output variance and/or maximizes a between-group output variance of the machine learning model.
However, Tickoo teaches, wherein the training the machine learning model is based on an objective function that minimizes a within-group output variance and/or maximizes a between-group output variance of the machine learning model (Abstract - teaches “using the direction optimization set to calculate an optimum transformation vector that maximizes inter-class separability and minimizes intra-class variance of the feature samples with respect to corresponding class labels” thus disclosing a training based on an optimization criterion that maximizes inter-class separability and minimizes intra-class variance, as the direction optimization set is used to calculate an optimum transformation vector satisfying those criteria, which constitutes an objective function, Tickoo).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to modify the combination of the combination of Zhou, Forman and Basak system because they are in the same field of endeavor, as directed to training supervised machine learning classifiers using labeled data and improving classification accuracy through training methodologies. Thus, the motivation to combine would be to train the machine learning model of Zhou using the objective function taught by Tickoo in order to improve class separation and classification accuracy.
Claim 13 is similar to claim 2 hence rejected similarly.
Claim 18 is similar to claim 2 hence rejected similarly.
Claims 4, 12 and 17 rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (US 12,482,242) further in view of Forman et al. (US 7,792,353) and Basak et al. (US 2022/0335335) further in view of Anwar et al. (US 2022/0156574)
4. The combination of Zhou, Forman and Basak teach, The system of claim 1, wherein the operations further comprise:
calculating a first value based on the third plurality of classification scores (Paragraph 29 – teaches using logits/probability scores associated with deliberately mislabeled samples and can support calculating a value from those scores, Basak; and Claim 1 - each prediction label has a confidence score indicating a likelihood that the prediction label is correct – discloses plural confidence scores output by the classifier, Zhou), wherein the first value indicates the efficacy of the machine learning model (Paragraph 29 – teaches the mislabeled samples and their scores, Basak) in detecting the mislabeled data sets (Claim 1 and training-set correction logic – teaches identifying mislabeled samples based on classifier outputs and retraining impacts thus supplies the mislabeled-data context, Zhou).
The combination of Zhou and Forman do not explicitly teach or suggest,
determining whether the first value is larger than a second value calculated during a previous training iteration of the machine learning model, and wherein the re-training the machine learning model is responsive to determining that the first value is larger than the second value by a threshold.
However, Anwar teaches,
determining whether the first value is larger than a second value (Paragraph 56 - defines a loss difference Δθ=θ.sub.i−θ.sub.i-1 – discloses comparing a current value to a prior value, Anwar) calculated during a previous training iteration of the machine learning model (Paragraph 56 – teaches θ.sub.i is the loss … for the current training iteration and θ.sub.i-1 is the loss… for the preceding training iteration – thus disclosing values calculated during previous training iterations, Anwar), and wherein the re-training the machine learning model is responsive to determining that the first value is larger than the second value by a threshold (Paragraph 59 – teaches the decision … can be decided through a statically defined threshold between loss differences and Paragraph 57 – teaches if the loss difference is Δθ is large … continue to train the model – thus disclosing threshold-based training control, Anwar).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to incorporate Anwar’s iteration-based, threshold-controlled training metric into the mislabel-detection and retraining system of Zhou, as implemented using Forman’s iterative retraining framework, and Basak’s utilization of Mislabeled training data sets, in order to quantitatively determine when retraining is warranted and to avoid unnecessary retraining cycles.
Claim 12 is similar to claim 4 hence rejected similarly.
Claim 17 is similar to claim 4 hence rejected similarly
Claims 5, 6, 14, 19, and 20 rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (US 12,482,242) in view of Forman et al. (US 7,792,353) and Basak et al. (US 2022/0335335) further in view of Vahdat (US 11,531,852)
All the limitations of claim 1 are taught above.
5. The combination of Zhou, Forman and Basak do not explicitly teach or suggest, prior to the training the machine learning model, relabeling a first subset of data sets in the first group from the first classification to the second classification and moving the first subset of data sets from the first group to the second group.
However, Vahdat teaches,
prior to the training the machine learning model ( Fig 530 (521, 524, 526)- teaches 521 (label noise estimation), 524 (label correction) and 526 (dataset preparation) which produces a trained joint distribution later used for inference at block 540 – thus disclosing placing label correction before completion of training of the machine learning system presented by block 530, Vahdat), relabeling a first subset of data sets in the first group (Abstract - teaches incorrect labels may be corrected by flipping the labels to their correct state – thus disclosing relabeling a subset of training samples, Vahdat) from the first classification to the second classification (Claim 1 - teaches label noise is the form of stochastic label flips on binary labels – thus disclosing that label flips explicitly teach changing a data set from one classification to another, Vahdat) and moving the first subset of data sets from the first group to the second group (Col5: lines 20-30 – teaches reassigning a data set from one class group to another, Vahdat).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to combine Vahdat with the combination of Zhou, Forman and Basak because all the references are in the same field of supervised machine learning model training and address the common problem of incorrect labels in training data, a person of ordinary skill in the art would have been motivated to incorporate Vahdat’s pre-training label correction techniques into the systems of Zhou, Forman and Basak in order to further improve classifier performance.
6. The combination of Zhou, Forman, Basak and Vahdat teach, The system of claim 5, wherein the operations further comprise randomly selecting (Claim 1 – teaches selecting candidate data sets for further processing based on classifier outputs and confidence scores – this discloses randomly selecting because randomly is not defined on how the data sets are random, what probability distribution is used or how the selection differs from other non-deterministic or unspecifying a deterministic selection rule, under BRI, Zhou)the first subset of data sets (Abstract - teaches incorrect labels may be corrected by flipping the labels to their correct state – discloses operating on a subset of the data sets, satisfying the ‘first subset” requirement, Vahdat) from the first group of data sets (Claim 1 – teaches training data sets are grouped according to classification labels and candidate data sets are selected from a labeled group – explicitly discloses classification-based grouping of data sets and selecting subsets from a group, Zhou).
Claim 14 is similar to claim 5, hence rejected similarly.
Claim 19 is similar to claim 5, hence rejected similarly.
Claim 20 is similar to claim 6, hence rejected similarly.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (US 12,482,242) further in view of Forman et al. (US 7,792,353) and Basak et al. (US 2022/0335335) further in view of Zoldi et al. (US 11,367,074)
All the limitations of claim 1 are taught above.
7. The combination of Zhou, Forman and Basak does not explicitly teach, wherein each data set in the first group of data sets and the second group of data sets corresponds to a transaction, wherein the first classification corresponds to a fraudulent classification, and wherein the second classification corresponds to a non-fraudulent classification.
However, Zoldi teaches,
wherein each data set in the first group of data sets and the second group of data sets corresponds to a transaction (Col 1: lines 7-10 - teaches the subject matter described herein relates to fraud detection, and more particularly to high resolution transaction-level fraud detection and Col 4: lines 28-29 – teaches the pinpoint model is trained on transactions within the fraud window – discloses that the data processed and modeled are transactions, satisfying the requirement that each data set corresponds to a transaction, Zoldi), wherein the first classification corresponds to a fraudulent classification (Abstract - teaches distinguish fraudulent transactions from a legitimate transaction, and Claim 1 – teaches trained … to distinguish between fraudulent and legitimate transactions – thus disclosing a fraudulent classification applied to transactions, Zoldi), and wherein the second classification corresponds to a non-fraudulent classification (Abstract - distinguish fraudulent transactions from a legitimate transaction – fraud / non-fraud transactions – thus disclosing a non-fraudulent (legitimate) classification distinct from fraudulent transactions, Zoldi).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to combine Zoldi with the combination of Zhou, Forman and Basak because they are in the same field of endeavor, namely training supervised machine learning classifiers on labeled datasets, differing only in the application domain (fraud). Because Zoldi employs supervised learning on labeled transaction data and addresses classification accuracy, a person of ordinary skill in the art would have been motivated to apply the mislabel-detection and retraining techniques of Zhou, Forman and Basak to the transaction-level fraud detection system for Zoldi is order to improve fraud classification performance.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al. (US 12,482,242) further in view of Forman et al. (US 7,792,353) and Basak et al. (US 2022/0335335) further in view of Sandepudi et al. (US 2020/0387835)
All the limitations of claim 15 are taught above.
16. The combination of Zhou, Forman and Basak does not explicitly teach, wherein each data set in the first group of data sets and the second group of data sets corresponds to a transaction, wherein the first classification corresponds to a fraudulent classification, and wherein the second classification corresponds to a non-fraudulent classification.
Sandepudi teaches,
wherein each data set in the first group of data sets and the second group of data sets corresponds to a transaction, wherein the first classification corresponds to a fraudulent classification, and wherein the second classification corresponds to a non-fraudulent classification (Fig 2 – teaches a transaction record classified as fraud or not fraud status, Sandepudi).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to apply mislabeled data detection and retaining techniques of Zhou, Forman and Basak to the fraud-detection system of Sandepudi because Sandepudi teaches a ML classifier trained using transaction data labeled as fraudulent and non-fraudulent. Applying the known technique for identifying and correcting mislabeled training data would have predictably improved the accuracy and reliability of Sandepudi’s fraud classifier by reducing errors caused by incorrect transaction labels.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Nguyen et al. (US 2022/0383321) – teaches fraud rule including fraud criteria used to analyze financial transaction data to detect the fraud scenario and fraud response (Abstract, Fig 3)).
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMRESH SINGH whose telephone number is (571)270-3560. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann J. Lo can be reached at (571) 272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMRESH SINGH/Primary Examiner, Art Unit 2159
Prosecution Insights