Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are presented in the case.
Priority
Applicant's claim for the benefit of a prior-filed Provisional application 62/986,380 filed on 03/06/2020 is acknowledged.
Information Disclosure Statement
The information disclosure statement submitted on 07/05/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 2 and 4 are objected to because of the following informalities:
Claim 2, line 1 recites the phrase “wherein the subset of objects are” which should be “wherein each object in the subset”
Claim 4, line 2 recites the phrase “at least one of the prelabel or the relabel” which should be “at least one of the corresponding prelabel or a corresponding relabel”
For the informalities above and wherever else they may occur appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 8, 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Forman et al. (US 20080103996 A1) hereinafter Forman in view of Munro et al. (US 20160162464 A1) hereinafter Munro.
As to independent claim 1, Forman teaches a machine learning method implemented using one or more computing devices of a machine learning platform, the method comprising: [machine learning system/classifier ¶18-24]
generating, for objects in a dataset comprising a set of prelabeled objects, machine-predicted labels by [training set Fig. 3 45 with samples (objects) and classification labels ¶20-21 " training set 45, which includes the set of samples and their assigned labels"]
(i) training, via supervised learning, a machine learning model on a majority of the dataset and [trained via supervised techniques, such as SVM, using training set ¶4, ¶26 "training sample(s) are selected from training set 45 and designated as selected training samples 77"]
(ii) applying the trained machine learning model to the remainder of the dataset to obtain machine-predicted labels and corresponding confidence scores, [10-fold cross validation includes 9 folds for majority and remaining fold is held out when applying training data ¶27 "5 ten times in a known-manner 10-fold cross-validation"], [classification score (confidence score) ¶29 "generates a soft classification score and then makes a hard classification decision based on that score"]
wherein (i) and (ii) are iteratively repeated until every object in the set of objects has a prelabel, a machine-predicted label, and a confidence score corresponding to the machine-predicted label; [10 fold (ten time iteration) so each item is evaluated ¶27 "10-fold cross-validation so that each item evaluated is not directly included in the training set of the classifier)"]
receiving, via a graphical user interface, one or more relabels for one or more objects in a subset of objects that have machine-predicted labels in disagreement from corresponding prelabels, [interface Fig. 3 55 that receives selected samples for labelling based on disparity (disagreement) ¶30-36 "selected training samples 77 are presented to the user 57 by labeling interface module 55"…" module 53 selects one or more of the samples from training set 45 that have the largest disparity"]
the graphical user interface comprising the subset of objects and, for each object in the subset, a user field configured to trigger, in response to a detected input therein, relabeling of the object; [Fig. 6 illustrates sample 90 (object) and radio button choices (fields) for re-assigning labels ¶35-38 " a label (in this case, label 93) has been pre-selected, based either on the previously assigned classification label or the predicted classification label for e-mail message 90. Accordingly, the user 57 only needs to click on the appropriate radio button 91, 92, 94 or 95 if he or she disagrees with the pre-selection"]
Forman does not specifically teach determining that a first object of the one or more objects in the subset of objects has been relabeled greater than a threshold number of times; and excluding the first object from the dataset responsive to determining that the first object has been relabeled greater than the threshold number of times.
However, Munro teaches determining that a first object of the one or more objects in the subset of objects has been relabeled greater than a threshold number of times; and [determines a criterion based on a number of annotations (labels Fig. 2 220) ¶12-13 "third criterion is satisfied when the number of annotations received is at least a maximum number "]
excluding the first object from the dataset responsive to determining that the first object has been relabeled greater than the threshold number of times. [not used in model training (exclude training object) ¶138, ¶122 "the label is ignored. In other words, the annotations for the label are discarded and are not used for model training."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the classification disclosed by Forman by incorporating the determining that a first object of the one or more objects in the subset of objects has been relabeled greater than a threshold number of times; and excluding the first object from the dataset responsive to determining that the first object has been relabeled greater than the threshold number of times disclosed by Munro because both techniques address the same field of machine learning and by incorporating Munro into Forman provides a more efficient development of models for more accurate processing of information [Munro ¶4-5, ¶43].
As to dependent claim 8, the rejection of claim 1 is incorporated, Forman and Munro further teach generating a second dataset comprising the set of objects and, for each object, either the corresponding prelabel or the corresponding relabel; and [Forman modified training set ¶47]
generating, for objects in the second dataset, second machine-predicted labels by (i) training, via supervised learning, a second machine learning model on a majority of the second dataset and [Forman trained via supervised techniques, such as SVM, using training set ¶4, ¶26 "training sample(s) are selected from training set 45 and designated as selected training samples 77"] (ii) applying the second trained machine learning model to the remainder of the second dataset to obtain second machine-predicted labels and corresponding second confidence scores, [Forman 10-fold cross validation includes 9 folds for majority and remaining fold is held out when applying training data ¶27 "5 ten times in a known-manner 10-fold cross-validation"], [classification score (confidence score) ¶29 "generates a soft classification score and then makes a hard classification decision based on that score"] wherein (i) and (ii) are repeated until every object in the set of objects has (1) a second machine-predicted label and (2) and a second confidence score corresponding to the second machine-predicted label. [Forman 10 fold (ten time iteration) so each item is evaluated ¶27 "10-fold cross-validation so that each item evaluated is not directly included in the training set of the classifier)"]
As to dependent claim 10, the rejection of claim 1 is incorporated, Forman and Munro further teach before generating machine-predicted labels, manually prelabeling each object in the set of objects by presenting a second graphical user interface with selectable fields, and detecting, via the selectable fields, prelabel selections for each object in the set of objects. [Forman user label ¶21 "allows a user 44 to designate an appropriate label for each presented training sample 7"]
As to independent claim 20, Forman teaches a machine learning platform comprising a computing device having a processor and a memory with instructions executable by the processor to cause the computing device to: [processor, memory ¶69-70, machine learning system/classifier ¶18-24]
generate, for objects in a dataset comprising a set of prelabeled objects, machine-predicted labels by [training set Fig. 3 45 with samples (objects) and classification labels ¶20-21 " training set 45, which includes the set of samples and their assigned labels"](i) training, via supervised learning, a machine learning model on a majority of the dataset and [trained via supervised techniques, such as SVM, using training set ¶4, ¶26 "training sample(s) are selected from training set 45 and designated as selected training samples 77"] (ii) applying the trained machine learning model to the remainder of the dataset to obtain machine-predicted labels and corresponding confidence scores, [10-fold cross validation includes 9 folds for majority and remaining fold is held out when applying training data ¶27 "5 ten times in a known-manner 10-fold cross-validation"], [classification score (confidence score) ¶29 "generates a soft classification score and then makes a hard classification decision based on that score"] wherein (i) and (ii) are repeated until every object in the set of objects has a prelabel, a machine-predicted label, and a confidence score corresponding to the machine-predicted label; [10 fold (ten time iteration) so each item is evaluated ¶27 "10-fold cross-validation so that each item evaluated is not directly included in the training set of the classifier)"]
receive, via a graphical user interface comprising the subset of objects and, for each object in the subset, a user field configured to trigger, in response to a detected input therein, relabeling of the object; [Fig. 6 illustrates sample 90 (object) and radio button choices (fields) for re-assigning labels ¶35-38 in an interface ¶30-36 " a label (in this case, label 93) has been pre-selected, based either on the previously assigned classification label or the predicted classification label for e-mail message 90. Accordingly, the user 57 only needs to click on the appropriate radio button 91, 92, 94 or 95 if he or she disagrees with the pre-selection"]
Forman does not specifically teach determine that a first object of the one or more objects in the subset of objects has been relabeled greater than a threshold number of times; and exclude the first object from the dataset responsive to determining that the first object has been relabeled greater than the threshold number of times.
However, Munro teaches determine that a first object of the one or more objects in the subset of objects has been relabeled greater than a threshold number of times; and [determines a criterion based on a number of annotations (labels Fig. 2 220) ¶12-13 "third criterion is satisfied when the number of annotations received is at least a maximum number "]
exclude the first object from the dataset responsive to determining that the first object has been relabeled greater than the threshold number of times. [not used in model training (exclude training object) ¶138, ¶122 "the label is ignored. In other words, the annotations for the label are discarded and are not used for model training."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the classification disclosed by Forman by incorporating the determine that a first object of the one or more objects in the subset of objects has been relabeled greater than a threshold number of times; and exclude the first object from the dataset responsive to determining that the first object has been relabeled greater than the threshold number of times disclosed by Munro because both techniques address the same field of machine learning and by incorporating Munro into Forman provides a more efficient development of models for more accurate processing of information [Munro ¶4-5, ¶43].
Claims 2-4 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Forman in view of Munro, as applied to the rejection of claim 1 and 8 above, and further in view of Carmeli et al. (US 20140214732 A1) hereinafter Carmeli.
As to dependent claim 2, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach wherein the subset of objects are objects for which machine-predicted labels have a corresponding confidence score greater than a threshold confidence score.
However, Carmeli teaches wherein the subset of objects are objects for which machine-predicted labels have a corresponding confidence score greater than a threshold confidence score. [Carmeli removes those with high confidence above threshold ¶42 " remove all instances for which all classifiers predict with high confidence"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman and Munro by incorporating the wherein the subset of objects are objects for which machine-predicted labels have a corresponding confidence score greater than a threshold confidence score disclosed by Carmeli because all techniques address the same field of machine learning and by incorporating Carmeli into Forman and Munro improves the communication rapid and convenient data entry while reducing inconsistences [Carmeli ¶3-4].
As to dependent claim 3, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach for each machine-predicted label with a corresponding confidence score that is greater than a confidence threshold, comparing the machine-predicted label with the corresponding prelabel to generate the subset of objects that have machine-predicted labels in disagreement from corresponding prelabels.
However, Carmeli teaches for each machine-predicted label with a corresponding confidence score that is greater than a confidence threshold, comparing the machine-predicted label with the corresponding prelabel to generate the subset of objects that have machine-predicted labels in disagreement from corresponding prelabels. [Carmeli different labels and mislabeled instances based on high confidence that find inconsistencies ¶42, ¶45 "potential inconsistencies in the ER may be determined The predicted label and the actual label of the ER may be compared and in case they are different it may be determined as a potential inconsistency"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman and Munro by incorporating the for each machine-predicted label with a corresponding confidence score that is greater than a confidence threshold, comparing the machine-predicted label with the corresponding prelabel to generate the subset of objects that have machine-predicted labels in disagreement from corresponding prelabels disclosed by Carmeli because all techniques address the same field of machine learning and by incorporating Carmeli into Forman and Munro improves the communication rapid and convenient data entry while reducing inconsistences [Carmeli ¶3-4].
As to dependent claim 4, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach wherein the subset of objects are objects for which machine-predicted labels have a corresponding confidence score greater than a threshold confidence score.
However, Carmeli teaches wherein the subset of objects are objects for which machine-predicted labels have a corresponding confidence score greater than a threshold confidence score. [Carmeli removes those with high confidence above threshold ¶42 " remove all instances for which all classifiers predict with high confidence"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman and Munro by incorporating the wherein the subset of objects are objects for which machine-predicted labels have a corresponding confidence score greater than a threshold confidence score disclosed by Carmeli because all techniques address the same field of machine learning and by incorporating Carmeli into Forman and Munro improves the communication rapid and convenient data entry while reducing inconsistences [Carmeli ¶3-4].
As to dependent claim 9, the combination of Forman and Munro teach all the limitations of claim 8 that is incorporated.
Forman and Munro further teach presenting a second graphical user interface comprising the second subset of objects and, for each object in the second subset, a second user field configured to trigger, in response to a detected input therein, relabeling of the object; and [Forman Fig. 6 illustrates sample 90 (object) and radio button choices (fields) for re-assigning labels ¶35-38 " a label (in this case, label 93) has been pre-selected, based either on the previously assigned classification label or the predicted classification label for e-mail message 90. Accordingly, the user 57 only needs to click on the appropriate radio button 91, 92, 94 or 95 if he or she disagrees with the pre-selection"]
detecting, via the second graphical user interface, one or more relabels for one or more objects in the subset. [Forman user relabels ¶35-38]
Forman and Munro do not specifically teach for each second machine-predicted label with a corresponding second confidence score that is greater than a second threshold, comparing the second machine-predicted label with the corresponding prior label to generate a second subset of objects that have machine-predicted labels in disagreement from corresponding prior labels.
However, Carmeli teaches for each second machine-predicted label with a corresponding second confidence score that is greater than a second threshold, comparing the second machine-predicted label with the corresponding prior label to generate a second subset of objects that have machine-predicted labels in disagreement from corresponding prior labels; [Carmeli different labels and mislabeled instances based on high confidence that find inconsistencies ¶42, ¶45 "potential inconsistencies in the ER may be determined The predicted label and the actual label of the ER may be compared and in case they are different it may be determined as a potential inconsistency"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman and Munro by incorporating for each second machine-predicted label with a corresponding second confidence score that is greater than a second threshold, comparing the second machine-predicted label with the corresponding prior label to generate a second subset of objects that have machine-predicted labels in disagreement from corresponding prior labels disclosed by Carmeli because all techniques address the same field of machine learning and by incorporating Carmeli into Forman and Munro improves the communication rapid and convenient data entry while reducing inconsistences [Carmeli ¶3-4].
Claims 5-7 and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Forman in view of Munro, as applied to the rejection of claim 1 and 8 above, and further in view of Nguyen et al. (US 20190042887 A1) hereinafter Nguyen.
As to dependent claim 5, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach wherein model-training iterations are independent such that the machine learning model trained in a second iteration is not further training the machine learning model of a first iteration.
However, Nguyen teaches wherein model-training iterations are independent such that the machine learning model trained in a second iteration is not further training the machine learning model of a first iteration. [multiple (separate iterations) models on portions of training data ¶25 " train models on every possible combination of segmented portions (e.g., folds) of training data sets."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the model by Forman and Munro by incorporating the wherein model-training iterations are independent such that the machine learning model trained in a second iteration is not further training the machine learning model of a first iteration disclosed by Nguyen because all techniques address the same field of machine learning and by incorporating Nguyen into Forman and Munro saves time before deploying models with less reprogramming required [Nguyen ¶5-6].
As to dependent claim 6, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach wherein the method comprises ten model-training iterations, with each iteration training the machine learning model on about 90% of the dataset and using the trained machine learning model to test the remaining about 10% of the dataset.
However, Nguyen teaches wherein the method comprises ten model-training iterations, with each iteration training the machine learning model on about 90% of the dataset and using the trained machine learning model to test the remaining about 10% of the dataset. [Nguyen K=10 iterations with configurable test splits ¶25, ¶66 " split the training data into k folds and provide k-1 folds to the training systems as training sets while retaining the kth fold as a test set. “k” may be specified in the model training specification. For example, if k is specified to have a value of 10, and the test set of data has 1 million items, then the training data may be randomly split into 10 folds, each fold including 100,000 items"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the model by Forman and Munro by incorporating the wherein the method comprises ten model-training iterations, with each iteration training the machine learning model on about 90% of the dataset and using the trained machine learning model to test the remaining about 10% of the dataset disclosed by Nguyen because all techniques address the same field of machine learning and by incorporating Nguyen into Forman and Munro saves time before deploying models with less reprogramming required [Nguyen ¶5-6].
As to dependent claim 7, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach wherein the majority of the dataset comprises at least 80% of the dataset as training data and the remainder as test data, wherein the test data of the iterations are substantially non-overlapping.
However, Nguyen teaches wherein the majority of the dataset comprises at least 80% of the dataset as training data and the remainder as test data, wherein the test data of the iterations are substantially non-overlapping. [Nguyen different folds with different samples as test ¶25, ¶66 " each fold having different data from each of the other folds"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the model by Forman and Munro by incorporating the wherein the majority of the dataset comprises at least 80% of the dataset as training data and the remainder as test data, wherein the test data of the iterations are substantially non-overlapping disclosed by Nguyen because all techniques address the same field of machine learning and by incorporating Nguyen into Forman and Munro saves time before deploying models with less reprogramming required [Nguyen ¶5-6].
As to dependent claim 12, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach wherein training the machine learning model comprises using random forest modeling.
However, Nguyen teaches wherein training the machine learning model comprises using random forest modeling. [Nguyen random forest ¶24]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the model by Forman and Munro by incorporating the wherein training the machine learning model comprises using random forest modeling disclosed by Nguyen because all techniques address the same field of machine learning and by incorporating Nguyen into Forman and Munro saves time before deploying models with less reprogramming required [Nguyen ¶5-6].
As to dependent claim 13, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach wherein training the machine learning model comprises using gradient boosting.
However, Nguyen teaches wherein training the machine learning model comprises using gradient boosting. [Nguyen gradient boosted ¶24]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the model by Forman and Munro by incorporating the wherein training the machine learning model comprises using gradient boosting disclosed by Nguyen because all techniques address the same field of machine learning and by incorporating Nguyen into Forman and Munro saves time before deploying models with less reprogramming required [Nguyen ¶5-6].
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Forman in view of Munro, as applied to the rejection of claim 1 above, and further in view of Yankov et al. (US 20120095943 A1) hereinafter Yankov.
As to dependent claim 11, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach wherein a number of potential labels is no greater than 1% of a number of objects in the set of objects.
However, Yankov teaches wherein a number of potential labels is no greater than 1% of a number of objects in the set of objects. [103 categories (labels) of 804K dataset is less than a percent ¶80 " data set consisting of 804,414 news articles from the Reuters news agency classified into 103 categories"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman and Munro by incorporating the wherein a number of potential labels is no greater than 1% of a number of objects in the set of objects disclosed by Yankov because all techniques address the same field of machine learning and by incorporating Yankov into Forman and Munro maximize the efficiency of labeling for improved classifiers [Yankov ¶2-5].
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Forman in view of Munro, as applied to the rejection of claim 1 above, and further in view of Li et al (US 20190370394 A1) hereinafter Li.
As to dependent claim 14, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach lemmatizing text in manual categorizations of the objects to generate the prelabels.
However, Li teaches lemmatizing text in manual categorizations of the objects to generate the prelabels. [lemmatizing text ¶7 " lemmatizing the first corpus of unstructured computer text and the second corpus of unstructured text"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman and Munro by incorporating the lemmatizing text in manual categorizations of the objects to generate the prelabels disclosed by Li because all techniques address the same field of machine learning and by incorporating Li into Forman and Munro alleviate issues with the amount of training data and better automate classification [Li ¶3].
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Forman in view of Munro and Yankov, as applied to the rejection of claim 11 above, and further in view of GOYAL et al. (US 20190025800 A1) hereinafter Goyal.
As to dependent claim 15, the combination of Forman and Munro teach all the limitations of claim 11 that is incorporated.
Forman, Munro and Yankov do not specifically teach removing stop words and white space from the manual categorizations to generate the prelabels.
However, Goyal teaches removing stop words and white space from the manual categorizations to generate the prelabels. [removes stop words ¶71]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman, Munro and Yankov by incorporating the removing stop words and white space from the manual categorizations to generate the prelabels disclosed by Goyal because all techniques address the same field of machine learning and by incorporating Goyal into Forman, Munro and Yankov improves the quality and accuracy of automating tasks [Goyal ¶1-2].
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Forman in view of Munro, as applied to the rejection of claim 1 above, and further in view of Lee et al. (US 20140288970 A1) hereinafter Lee.
As to dependent claim 16, the combination of Forman and Munro teach all the limitations of claim 1 that is incorporated.
Forman and Munro do not specifically teach generating n-grams for text in manual categorizations by parsing the text to generate the prelabels, and one-hot encoding the n-grams to obtain a 1 when the n-gram is present in a text field and a 0 when the n-gram is not present in the text field, and feeding a resulting set of 1s and 0s to the machine learning models.
However, Lee teaches generating n-grams for text in manual categorizations by parsing the text to generate the prelabels, and one-hot encoding the n-grams to obtain a 1 when the n-gram is present in a text field and a 0 when the n-gram is not present in the text field, and feeding a resulting set of 1s and 0s to the machine learning models. [n-grames from text ¶25 and vector with 0 and 1 ¶26 "real-valued number between 0 and 1, where 0 indicates that the processed sentence or set of words does not contain a recommendation, 1 indicates that it does,"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman and Munro by incorporating the generating n-grams for text in manual categorizations by parsing the text to generate the prelabels, and one-hot encoding the n-grams to obtain a 1 when the n-gram is present in a text field and a 0 when the n-gram is not present in the text field, and feeding a resulting set of 1s and 0s to the machine learning models by Lee because all techniques address the same field of machine learning and by incorporating Lee into Forman and Munro reduce the amount of time and effort in performing tasks [Lee ¶13-14].
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Forman in view of Carmeli.
As to independent claim 17, Forman teaches a machine learning method implemented using one or more computing devices of a machine learning platform, the method comprising: [machine learning system/classifier ¶18-24]
generating, for objects in a dataset comprising a set of prelabeled objects, machine-predicted labels by [training set Fig. 3 45 with samples (objects) and classification labels ¶20-21 " training set 45, which includes the set of samples and their assigned labels"]
(i) training, via supervised learning, a machine learning model on a majority segment of the dataset and [trained via supervised techniques, such as SVM, using training set ¶4, ¶26 "training sample(s) are selected from training set 45 and designated as selected training samples 77"]
(ii) applying the trained machine learning model to a minority segment of the dataset to obtain machine-predicted labels, [10-fold cross validation includes 9 folds for majority and remaining fold is held out when applying training data (90% train 10 remaining) ¶27 "5 ten times in a known-manner 10-fold cross-validation"],
wherein (i) and (ii) are repeated iteratively until every object in the set of objects has a prelabel and a machine-predicted label, and [10 fold (ten time iteration) so each item is evaluated ¶27 "10-fold cross-validation so that each item evaluated is not directly included in the training set of the classifier)"]
wherein in each iteration, the majority segment comprises from about 80% of the dataset to about 95% of the segment, and the minority segment comprises the remaining about 5% to about 20% of the dataset; [10 fold is 90% training ¶27]
modifying prelabels in the dataset in response to finding differences between the prelabels and corresponding machine-predicted labels for a subset of objects in the dataset; [comparison module compares labels to predicted labels and does changes, selections and scoring to them (modifies) based on disparity ¶29-34 "changing the assigned classification"]
determining that the machine-predicted label of a first object of the set of prelabeled objects does not match the prelabel of the first object; and [compares for largest disparity (not match) ¶28-34 "measure of disparity can comprise, e.g., a simple difference, a ratio, or any other selected function"]
wherein applying the trained machine learning models yields a confidence score for each machine-predicted label, [classification score (confidence score) ¶29 "generates a soft classification score and then makes a hard classification decision based on that score"]
Forman does not specifically teach excluding the first object from the dataset responsive to determining that the machine-predicted label of the first object does not match the prelabel of the first object; and wherein modifying prelabels in the dataset comprises modifying prelabels for objects in the subset of objects only when an object's machine-predicted label has a corresponding confidence score exceeding a threshold.
However, Carmeli teaches excluding the first object from the dataset responsive to determining that the machine-predicted label of the first object does not match the prelabel of the first object; [Carmeli removes those with high confidence above threshold Fig. 2 220, ¶42 " remove all instances for which all classifiers predict with high confidence"]
and wherein modifying prelabels in the dataset comprises modifying prelabels for objects in the subset of objects only when an object's machine-predicted label has a corresponding confidence score exceeding a threshold. [Carmeli different labels and mislabeled instances based on high confidence that find inconsistencies ¶42, ¶45 "potential inconsistencies in the ER may be determined The predicted label and the actual label of the ER may be compared and in case they are different it may be determined as a potential inconsistency"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman by incorporating the excluding the first object from the dataset responsive to determining that the machine-predicted label of the first object does not match the prelabel of the first object; and wherein modifying prelabels in the dataset comprises modifying prelabels for objects in the subset of objects only when an object's machine-predicted label has a corresponding confidence score exceeding a threshold disclosed by Carmeli because both techniques address the same field of machine learning and by incorporating Carmeli into Forman improves the communication rapid and convenient data entry while reducing inconsistences [Carmeli ¶3-4].
As to dependent claim 18, the rejection of claim 1 is incorporated, Forman and Carmeli further teach generating a second dataset comprising the set of objects and, for each object, either the corresponding prelabel or the corresponding relabel; and [Forman modified training set ¶47]
generating, for objects in the second dataset, second machine-predicted labels by (i) training, via supervised learning, a second machine learning model on a majority of the second dataset and [Forman trained via supervised techniques, such as SVM, using training set ¶4, ¶26 "training sample(s) are selected from training set 45 and designated as selected training samples 77"] (ii) applying the second trained machine learning model to the remainder of the second dataset to obtain second machine-predicted labels and corresponding second confidence scores, [Forman 10-fold cross validation includes 9 folds for majority and remaining fold is held out when applying training data ¶27 "5 ten times in a known-manner 10-fold cross-validation"], [classification score (confidence score) ¶29 "generates a soft classification score and then makes a hard classification decision based on that score"] wherein (i) and (ii) are repeated until every object in the set of objects has (1) a second machine-predicted label and (2) and a second confidence score corresponding to the second machine-predicted label. [Forman 10 fold (ten time iteration) so each item is evaluated ¶27 "10-fold cross-validation so that each item evaluated is not directly included in the training set of the classifier)"]
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Forman in view of Carmeli, as applied to the rejection of claim 17 above, and further in view of Li and Lee.
As to dependent claim 19, the combination of Forman and Carmeli teach all the limitations of claim 17 that is incorporated.
Forman and Munro do not specifically teach lemmatizing text in manual categorizations of the objects to generate the prelabels.
However, Li teaches lemmatizing text in manual categorizations of the objects to generate the prelabels. [lemmatizing text ¶7 " lemmatizing the first corpus of unstructured computer text and the second corpus of unstructured text"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman and Munro by incorporating the lemmatizing text in manual categorizations of the objects to generate the prelabels disclosed by Li because all techniques address the same field of machine learning and by incorporating Li into Forman and Munro alleviate issues with the amount of training data and better automate classification [Li ¶3].
Forman, Carmeli and Li do not specifically teach generating n-grams for text in manual categorizations by parsing the text to generate the prelabels, and one-hot encoding the n-grams to obtain a 1 when the n-gram is present in a text field and a 0 when the n-gram is not present in the text field, and feeding a resulting set of 1s and 0s to the machine learning models.
However, Lee teaches generating n-grams for text in manual categorizations by parsing the text to generate the prelabels, and one-hot encoding the n-grams to obtain a 1 when the n-gram is present in a text field and a 0 when the n-gram is not present in the text field, and feeding a resulting set of 1s and 0s to the machine learning models. [n-grames from text ¶25 and vector with 0 and 1 ¶26 "real-valued number between 0 and 1, where 0 indicates that the processed sentence or set of words does not contain a recommendation, 1 indicates that it does,"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the training data by Forman and Munro by incorporating the generating n-grams for text in manual categorizations by parsing the text to generate the prelabels, and one-hot encoding the n-grams to obtain a 1 when the n-gram is present in a text field and a 0 when the n-gram is not present in the text field, and feeding a resulting set of 1s and 0s to the machine learning models by Lee because all techniques address the same field of machine learning and by incorporating Lee into Forman and Munro reduce the amount of time and effort in performing tasks [Lee ¶13-14].
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
SHIRAI et al. (US 20210019656 A1) teaches confidence data of prelabels in machine learning (see ¶92).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Beau Spratt whose telephone number is 571 272 9919. The examiner can normally be reached 8:30am to 5:00pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at 571 272 7212. The fax phone number for the organization where this application or proceeding is assigned is 571 483 7388.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866 217 9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800 786 9199 (IN USA OR CANADA) or 571 272 1000.
/BEAU D SPRATT/ Primary Examiner, Art Unit 2143
Prosecution Insights