DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/29/2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 8, and 15 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 12-13 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 12 recites “a toggle”. It is unclear if “a toggle” is referring to the “pre-selection toggle interface element” recited in claim 1 or a different toggle. For examination purposes, Examiner interprets “a toggle” is referring to the “pre-selection toggle interface element” in claim 1. Claim 13 is a dependent claim that does not cure the deficiencies and is rejected for the same reason.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-7 are rejected under 35 U.S.C. 103 as being unpatentable over Forman et al. (US-20080103996-A1) in view of Dutta et al. (US-10783167-B1), Riley et al. (US-20150012288-A1), and Arthur et al. (US-7640305-B1).
Regarding Claim 1,
Forman teaches a method of training an artificial intelligence model, the method comprising:
ingesting a plurality of training data items (para [0020] “Referring to FIG. 2, initially (in step 10) an initial training set 45 (shown in FIG. 3) is obtained. Initial training set 45 could have been generated in the conventional manner. That is, referring to FIG. 3, various samples 7 are selected and designated for labeling. Ordinarily, the samples 7 are chosen so as to be representative of the types of samples which one desires to classify using the resulting machine-learning classifier (e.g., unlabeled samples 8).”);
determining a plurality of applicable labels within the plurality of training data items (para [0021] “The selected samples 7 are labeled via interface module 43 in order to generate the training set 45, which includes the set of samples and their assigned labels.”);
automatically pre-selecting, using the artificial intelligence model, an automatic preliminary classification to be assigned to one or more items of the plurality of training data items according to a current state of training of the artificial intelligence model (para [0036]-[0038] discloses a preselection of label based on a previous classification (i.e. preliminary classification). para [0023]-[0025] Previous classification is based on a current training state of the ML model.);
generating for display, to the user within the user interface, a plurality of indicia corresponding to a plurality of labels, wherein each label of the plurality of labels is applicable to one or more items of the plurality of training data items (para [0021] “More preferably, module 43 provides a user interface which allows a user 44 to designate an appropriate label for each presented training sample 7. In one representative embodiment, module 43 displays to user 44 (here, a human domain expert) each sample 7 together with a set of radio buttons. The user 44 then clicks on one of the radio buttons, thereby designating the appropriate label for the sample 7.”);
receiving, from the user through the user interface, a selection of one or more selected indicia from the plurality of indicia (para [0021] “More preferably, module 43 provides a user interface which allows a user 44 to designate an appropriate label for each presented training sample 7. In one representative embodiment, module 43 displays to user 44 (here, a human domain expert) each sample 7 together with a set of radio buttons. The user 44 then clicks on one of the radio buttons, thereby designating the appropriate label for the sample 7.”);
responsive to the selection of the one or more selected indicia, generating an updated classification for the one or more items of the plurality of training data items based on the one or more selected indicia (para [0025]-[0027] And para [0044] The classifier is retrained (i.e., updated) based on user label selection.),
wherein the updated classification replaces the automatic preliminary classification (para [0036]-[0038] discloses a preselection of label based on a previous classification (i.e. preliminary classification). para [0045]-[0046] The updated classification for the training sample replaces the previous classification (i.e. preliminary classification).),
training the artificial intelligence model based on the one or more respective items, the one or more selected indicia, and the updated classification for the one or more items (para [0044] And para [0047] “In any event, once the training set 45 has been updated, the classifier 3 is retrained using the modified training set 45 (e.g., by training module 5), and the retrained classifier 3 is used to re-process at least some of the labeled training samples 45, thereby obtaining new predictions 4. And para [0064] Finally, in step 138 the training set 45 is modified based on the labels received in step 137, the classifier 3 is retrained based on the modified training set 45, and at least some of the samples 2 and 7 are reprocessed using classifier 3.”); and
classifying, using the artificial intelligence model, one or more real-time data items (para [0023] “Referring back to FIG. 2, in step 12 the classifier 3 is trained, e.g., using training module 5 (shown also in FIG. 3). Generally speaking, the training involves attempting to find an optimal (according to some underlying criteria) mapping from the supplied feature set values for the samples 7 to the corresponding classification labels 8, so that the resulting classifier 3 can receive new unlabeled samples 2 and provide classification labels 4 for them based on its best guess in view of the feature set values for such unlabeled samples 2.”).
Forman does not explicitly disclose
causing for display of a pre-selection toggle interface element to a user within a user interface, the pre-selection toggle interface element allowing the user to activate automatic classifications of the plurality of training data items;
responsive to an activation of the pre-selection toggle interface element,
storing, with one or more respective items from the plurality of training data items, the one or more selected indicia within a data store;
responsive to training the artificial intelligence model, automatically sorting, using the artificial intelligence model, the plurality of labels for at least one subsequent data item of the plurality of training data items based on the updated classification for the one or more items of the plurality of training data items;
However, Dutta (US 10783167 B1) teaches
storing, with one or more respective items from the plurality of training data items (col. 5 lines 61-64; “The classification data 102(1) and item data 106 may be stored in association with one or more classification servers 108 or other types of computing devices.”), the one or more selected indicia within a data store (col. 8 lines 13-16; “As additional user interactions 116 associated with the classification labels 104 are determined and stored as user interaction data 120, the classification module 122 may continue to modify the classification data 102(2).” And col. 6 lines 45-50; “For example, the user interaction data 120 may include an average length of time (e.g., “Avg. Time”) spent by users that selected the “Running” label, during which the users viewed or otherwise interacted with item data 106 or other classification labels 104 presented in the user interface 112.”);
Forman and Dutta are analogous because they are both directed towards the same field of endeavor of machine-learning classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the labeling system of Dutta.
Doing so would allow for excluding less useful, redundant, or erroneous labels from the classification data, the accuracy and efficiency of an automated classification process may be improved, enabling items to be classified using less processing time and other computing resources, improving the overall speed of the system (Dutta col. 5 lines 20-25;).
Riley (US 20150012288 A1) teaches
responsive to training the artificial intelligence model, automatically sorting, using the artificial intelligence model, the plurality of labels for at least one subsequent data item of the plurality of training data items based on the updated classification for the one or more items of the plurality of training data items (para [0049] “Classification algorithms may also learn by example, using training data that has been organized into classes manually or through some automated process. By observing the relationship between features and classes, the algorithm may learn which features are important in determining the proper label and which keywords provide little or misleading information about the appropriate label for the symptom in question. The result of the training process is a model that may be used later to classify previously unlabeled objects. The classifier processes the features of the objects to classify and uses its model to determine the best label for each object. Depending upon the classification algorithm used, the classifier may emit a single label or it may emit multiple labels, each accompanied by a score or probability that ranks the label against other possible labels for the object in question.” The classifier (i.e., AI model) is trained on training data (i.e., one or more items of the plurality of items) and ranks (i.e., sorts) the labels for an unclassified object (i.e., subsequent data item).);
Forman and Riley are analogous because they are both directed towards the same field of endeavor of machine-learning classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the label ranking of Riley.
Doing so would allow for ranking labels based on the probability of the label being assigned to a training sample to determine the best label for the sample (Riley para [0049]).
Arthur (US 7640305 B1) teaches
causing for display of a pre-selection toggle interface element to a user within a user interface, the pre-selection toggle interface element allowing the user to activate automatic classifications of the plurality of training data items (col. 3 lines 52-61; “The mail system provides the menu item automatic 115 to allow the user to request that the mail system be put into automatic mode. While in automatic mode, the mail system automatically categorizes mail as junk or not junk and takes appropriate actions based on those categorizations. The automatic mode is further described below with reference to FIGS. 4 and 5. In another embodiment the menu item training 110 and menu item automatic 115 may be implemented via a single toggle button or any other appropriate user interface element.” The toggle allows for automatic categorization (i.e., classification) of training items.);
responsive to an activation of the pre-selection toggle interface element (col. 4 lines 37-49; “FIG. 3 depicts a pictorial representation of an example user interface 300 for an inbox, according to an embodiment of the invention. As shown in the user interface 300 for the inbox, during training mode, the mail system has detected that mail 302 may be junk and has displayed message 305 "The mail system thinks this is junk. What do you think?" in order to receive training data or feedback via the buttons junk 310 and not junk 320, which the user may select in response to the message 305. In another embodiment, the functions of the buttons 310 and 320 may be requested via a single toggle button or any other appropriate user interface element. The user may also specify that any mail is junk or not junk without being asked via buttons 310 and 320.”),
Forman and Arthur are analogous because they are both directed towards a machine learning training interface.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the user interface of Foreman with the toggle of Arthur.
Doing so would allow the user to manually select labels for the training sample or automatically having the system label the training sample (Arthur col. 4 lines 37-49;).
Regarding Claim 3,
Forman, Dutta, Riley, and Arthur teach the method of claim 1. Forman further teaches further comprising: retraining the artificial intelligence model based at least in part on the one or more real-time data items (para [0047] “In any event, once the training set 45 has been updated, the classifier 3 is retrained using the modified training set 45 (e.g., by training module 5), and the retrained classifier 3 is used to re-process at least some of the labeled training samples 45, thereby obtaining new predictions 4.”).
Regarding Claim 4,
Forman, Dutta, Riley, and Arthur teach the method of claim 1. Forman further teaches further comprising:
receiving, from the user through the user interface, a subsequent selection corresponding to a misclassified data item of the plurality of training data items (para [0025] “As used herein, "confirmation/re-labeling" refers to the process of submitting an existing training sample for labeling so that its previously assigned label is either confirmed or contradicted, e.g., by a domain expert or other person. A request for confirmation/re-labeling can include, e.g.: (i) a mere request to indicate whether the previously assigned classification label is correct; and/or (ii) a request to designate a different label if the previously assigned classification label is incorrect.”);
responsive to the subsequent selection, relabeling the misclassified data item (para [0025] and para [0046]); and
retraining the artificial intelligence model based at least in part on the misclassified data item (para [0044] “In any event, the confirmation/re-labeling information is received for the submitted training examples and, in step 18, that information is used to modify the training set 45 and then retrain the classifier 3, e.g., using training module 5 and the revised training set 45. In the present embodiment, there are two possibilities for each resubmitted training sample 77.”).
Regarding Claim 5,
Forman, Dutta, Riley, and Arthur teach the method of claim 1. Forman further teaches further comprising:
automatically preselecting one or more labels of the plurality of labels using the artificial intelligence model (para [0036]-[0038] “Once again, a label (in this case, label 93) has been pre-selected, based either on the previously assigned classification label or the predicted classification label for e-mail message 90.”).
Regarding Claim 6,
Forman, Dutta, Riley, and Arthur teach the method of claim 5. Forman further teaches further comprising: responsive to the selection from the user, automatically deselecting the one or more labels that were preselected by the artificial intelligence model (para [0036] “In the present example, the label 82 indicating a valid e-mail message has been pre-selected (e.g., by interface module 55), and the user 57 only needs to click on the radio button for spam label 81 if the user 57 disagrees with this pre-selection.”).
Regarding Claim 7,
Forman, Dutta, Riley, and Arthur teach the method of claim 1. Forman teaches further comprising:
responsive to the selection of the one or more selected indicia, updating the display of the user interface with the one or more selected indicia (para [0037] “FIG. 6 illustrates a non-binary example in which an incoming e-mail message 90 is being labeled according to the category of subject matter (in this case, particular types of hardware) to which it most closely pertains. In the present example, the choices are printer 91, PC 92, display 93, server 94, or keyboard/mouse 95. Once again, a label (in this case, label 93) has been pre-selected, based either on the previously assigned classification label or the predicted classification label for e-mail message 90. Accordingly, the user 57 only needs to click on the appropriate radio button 91, 92, 94 or 95 if he or she disagrees with the pre-selection.”).
Claims 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Forman et al. (US-20080103996-A1) in view of Dutta et al. (US-10783167-B1), Riley et al. (US-20150012288-A1), and Arthur et al. (US-7640305-B1).
Regarding Claim 8,
Forman teaches one or more non-transitory computer-readable media that store computer-executable instructions that, when executed by at least one processor (para [0069]), perform a method of training an artificial intelligence model, the method comprising:
receiving a plurality of training data items… (para [0020] “Referring to FIG. 2, initially (in step 10) an initial training set 45 (shown in FIG. 3) is obtained.”);
ingesting the plurality of training data items (para [0020] “Referring to FIG. 2, initially (in step 10) an initial training set 45 (shown in FIG. 3) is obtained. Initial training set 45 could have been generated in the conventional manner. That is, referring to FIG. 3, various samples 7 are selected and designated for labeling. Ordinarily, the samples 7 are chosen so as to be representative of the types of samples which one desires to classify using the resulting machine-learning classifier (e.g., unlabeled samples 8).”);
automatically pre-selecting, using the artificial intelligence model, an automatic preliminary classification to be assigned to one or more items of the plurality of training data items according to a current state of training of the artificial intelligence model (para [0036]-[0038] discloses a preselection of label based on a previous classification (i.e. preliminary classification). para [0023]-[0025] Previous classification is based on a current training state of the ML model.);
determining a plurality of applicable labels within the plurality of training data items (para [0021] “The selected samples 7 are labeled via interface module 43 in order to generate the training set 45, which includes the set of samples and their assigned labels.”);
generating for display, to the user within the user interface, a plurality of indicia corresponding to a plurality of labels, wherein each label of the plurality of labels is applicable to one or more items of the plurality of training data items (para [0021] “More preferably, module 43 provides a user interface which allows a user 44 to designate an appropriate label for each presented training sample 7. In one representative embodiment, module 43 displays to user 44 (here, a human domain expert) each sample 7 together with a set of radio buttons. The user 44 then clicks on one of the radio buttons, thereby designating the appropriate label for the sample 7.”);
receiving, from the user through the user interface, a selection of one or more selected indicia from the plurality of indicia (para [0021] “More preferably, module 43 provides a user interface which allows a user 44 to designate an appropriate label for each presented training sample 7. In one representative embodiment, module 43 displays to user 44 (here, a human domain expert) each sample 7 together with a set of radio buttons. The user 44 then clicks on one of the radio buttons, thereby designating the appropriate label for the sample 7.”);
responsive to the selection of the one or more selected indicia, generating an updated classification for the one or more items of the plurality of training data items based on the one or more selected indicia (para [0025]-[0027] And para [0044] The classifier is retrained (i.e., updated) based on user label selection.), wherein the updated classification replaces the automatic preliminary classification (para [0036]-[0038] discloses a preselection of label based on a previous classification (i.e. preliminary classification). para [0045]-[0046] The updated classification for the training sample replaces the previous classification (i.e. preliminary classification).),
training the artificial intelligence model based on the one or more respective items, the one or more selected indicia, and the updated classification for the one or more items (para [0044] And para [0047] “In any event, once the training set 45 has been updated, the classifier 3 is retrained using the modified training set 45 (e.g., by training module 5), and the retrained classifier 3 is used to re-process at least some of the labeled training samples 45, thereby obtaining new predictions 4. And para [0064] Finally, in step 138 the training set 45 is modified based on the labels received in step 137, the classifier 3 is retrained based on the modified training set 45, and at least some of the samples 2 and 7 are reprocessed using classifier 3.”);
receiving, from the user through the user interface, a subsequent selection corresponding to a misclassified data item of the plurality of training data items (para [0036]-[0038]);
responsive to the subsequent selection, relabeling the misclassified data item (para [0044]-[0046]); and
retraining the artificial intelligence model based at least in part on the misclassified data item (para] [0047] “In any event, once the training set 45 has been updated, the classifier 3 is retrained using the modified training set 45 (e.g., by training module 5), and the retrained classifier 3 is used to re-process at least some of the labeled training samples 45, thereby obtaining new predictions 4. Upon completion of step 18, processing returns to step 14 to select additional samples from training set 45 and repeat the foregoing process.”).
Forman does not explicitly disclose
…from a training data store;
storing, with one or more respective items from the plurality of training data items, the one or more selected indicia within a data store;
responsive to retraining the artificial intelligence model, automatically sorting, using the artificial intelligence model, the plurality of labels for at least one subsequent data item of the plurality of training data items based on the updated classification for the one or more items of the plurality of training data items.
the input associated with a back interface element of the user interface and responsive to the input, moving from the current training data item to a previous one of the plurality of training data items.
However, Dutta (US 10783167 B1) teaches
…training data items from a training data store (col. 5 lines 61-64; “The classification data 102(1) and item data 106 may be stored in association with one or more classification servers 108 or other types of computing devices.”);
storing, with one or more respective items from the plurality of training data items, the one or more selected indicia within a data store (col. 8 lines 13-16; “As additional user interactions 116 associated with the classification labels 104 are determined and stored as user interaction data 120, the classification module 122 may continue to modify the classification data 102(2).” And col. 6 lines 45-50; “For example, the user interaction data 120 may include an average length of time (e.g., “Avg. Time”) spent by users that selected the “Running” label, during which the users viewed or otherwise interacted with item data 106 or other classification labels 104 presented in the user interface 112.”);
Forman and Dutta are analogous because they are both directed towards the same field of endeavor of machine-learning classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the labeling system of Dutta.
Doing so would allow for excluding less useful, redundant, or erroneous labels from the classification data, the accuracy and efficiency of an automated classification process may be improved, enabling items to be classified using less processing time and other computing resources, improving the overall speed of the system (Dutta col. 5 lines 20-25;).
Riley (US 20150012288 A1) teaches
responsive to retraining the artificial intelligence model, automatically sorting, using the artificial intelligence model, the plurality of labels for at least one subsequent data item of the plurality of training data items based on the updated classification for the one or more items of the plurality of training data items (para [0049] “Classification algorithms may also learn by example, using training data that has been organized into classes manually or through some automated process. By observing the relationship between features and classes, the algorithm may learn which features are important in determining the proper label and which keywords provide little or misleading information about the appropriate label for the symptom in question. The result of the training process is a model that may be used later to classify previously unlabeled objects. The classifier processes the features of the objects to classify and uses its model to determine the best label for each object. Depending upon the classification algorithm used, the classifier may emit a single label or it may emit multiple labels, each accompanied by a score or probability that ranks the label against other possible labels for the object in question.” The classifier (i.e., AI model) is trained on training data (i.e., one or more items of the plurality of items) and ranks (i.e., sorts) the labels for an unclassified object (i.e., subsequent data item). Training can include several iterations including re-training as evidenced by Forman.);
Forman and Riley are analogous because they are both directed towards the same field of endeavor of machine-learning classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the label ranking of Riley.
Doing so would allow for ranking labels based on the probability of the label being assigned to a training sample to determine the best label for the sample (Riley para [0049]).
Arthur (US 7640305 B1) teaches
causing for display of a pre-selection toggle interface element to a user within a user interface, the pre-selection toggle interface element allowing the user to activate automatic classifications of the plurality of training data items (col. 3 lines 52-61; “The mail system provides the menu item automatic 115 to allow the user to request that the mail system be put into automatic mode. While in automatic mode, the mail system automatically categorizes mail as junk or not junk and takes appropriate actions based on those categorizations. The automatic mode is further described below with reference to FIGS. 4 and 5. In another embodiment the menu item training 110 and menu item automatic 115 may be implemented via a single toggle button or any other appropriate user interface element.” The toggle allows for automatic categorization (i.e., classification) of training items.);
responsive to an activation of the pre-selection toggle interface element (col. 4 lines 37-49; “FIG. 3 depicts a pictorial representation of an example user interface 300 for an inbox, according to an embodiment of the invention. As shown in the user interface 300 for the inbox, during training mode, the mail system has detected that mail 302 may be junk and has displayed message 305 "The mail system thinks this is junk. What do you think?" in order to receive training data or feedback via the buttons junk 310 and not junk 320, which the user may select in response to the message 305. In another embodiment, the functions of the buttons 310 and 320 may be requested via a single toggle button or any other appropriate user interface element. The user may also specify that any mail is junk or not junk without being asked via buttons 310 and 320.”),
Forman and Arthur are analogous because they are both directed towards a machine learning training interface.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the user interface of Foreman with the toggle of Arthur.
Doing so would allow the user to manually select labels for the training sample or automatically having the system label the training sample (Arthur col. 4 lines 37-49;).
Regarding Claim 9,
Forman, Dutta, Riley, and Arthur teach the one or more non-transitory computer-readable media of claim 8. Forman further teaches wherein the user interface comprises a first pane displaying:
at least a subset of the plurality of training data items (para [0036] “Specifically, FIG. 5 illustrates an example in which an incoming e-mail message 80 is being labeled as spam 81 or a valid e-mail message 82 (i.e., a binary classification problem).” Figure 5 shows a first pane and an e-mail message which is a subset of the training items.); and
an indication of a current training data item of the plurality of training data items (para [0036] “Specifically, FIG. 5 illustrates an example in which an incoming e-mail message 80 is being labeled as spam 81 or a valid e-mail message 82 (i.e., a binary classification problem).” The email is indicated as the current training item.).
Regarding Claim 14,
Forman, Dutta, Riley, and Arthur teach the one or more non-transitory computer-readable media of claim 8. Forman further teaches wherein the plurality of indicia comprises at least one icon corresponding to a respective label of the plurality of labels (para [0037] “FIG. 6 illustrates a non-binary example in which an incoming e-mail message 90 is being labeled according to the category of subject matter (in this case, particular types of hardware) to which it most closely pertains. In the present example, the choices are printer 91, PC 92, display 93, server 94, or keyboard/mouse 95. Once again, a label (in this case, label 93) has been pre-selected, based either on the previously assigned classification label or the predicted classification label for e-mail message 90. Accordingly, the user 57 only needs to click on the appropriate radio button 91, 92, 94 or 95 if he or she disagrees with the pre-selection.” Icons are available for the user to select/click the label).
Claims 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Forman/Dutta/Riley/Arthur, as applied above, and further in view of Munro et al. (US-20160162456-A1).
Regarding Claim 10,
Forman, Dutta, Riley, and Arthur teach the one or more non-transitory computer-readable media of claim 9.
Forman, Dutta, Riley, and Arthur do not explicitly disclose
wherein the user interface further comprises a second pane displaying: the plurality of indicia corresponding to the plurality of labels.
However, Munro (US-20160162456-A1) teaches
wherein the user interface further comprises a second pane displaying: the plurality of indicia corresponding to the plurality of labels (Fig. 10B; para [0117] “the panel on the right includes an annotation prompt and a series of labels”).
Forman, Dutta, Riley, and Arthur are analogous because they are both directed towards the same field of endeavor of machine-learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the interface for labeling training data of Forman with the interface for labeling training data of Munro.
Doing so would a human user to provide annotations (Munro para [0126] “In some embodiments and as previously alluded to, the display interface for allowing a human annotator to supply an annotation is called a work unit.”).
Regarding Claim 11,
Forman, Dutta, Riley, Arthur, and Munro teach the one or more non-transitory computer-readable media of claim 10. Forman further teaches
wherein the user interface is configured such that the user can select and unselect one or more labels of the plurality of labels (para [0036] “Specifically, FIG. 5 illustrates an example in which an incoming e-mail message 80 is being labeled as spam 81 or a valid e-mail message 82 (i.e., a binary classification problem). In the present example, the label 82 indicating a valid e-mail message has been pre-selected (e.g., by interface module 55), and the user 57 only needs to click on the radio button for spam label 81 if the user 57 disagrees with this pre-selection.”),
wherein, by selecting the one or more labels of the plurality of labels, the user indicates that the artificial intelligence model should apply the one or more labels when classifying the current training data item (para [0037] “Once again, a label (in this case, label 93) has been pre-selected, based either on the previously assigned classification label or the predicted classification label for e-mail message 90. Accordingly, the user 57 only needs to click on the appropriate radio button 91, 92, 94 or 95 if he or she disagrees with the pre-selection. And para [0044] In any event, the confirmation/re-labeling information is received for the submitted training examples and, in step 18, that information is used to modify the training set 45 and then retrain the classifier 3, e.g., using training module 5 and the revised training set 45. In the present embodiment, there are two possibilities for each resubmitted training sample 77.”),
wherein the automatic preliminary classification of the plurality of training data items occurs in real time (para [0036] “Depending upon the particular sub-embodiment, the pre-selection is based either on the previously assigned classification label or the predicted class for the subject training sample 80.”),
Dutta further teaches
wherein the artificial intelligence model filters out one or more filtered labels of the plurality of labels that are not applicable to the current training data item based on the automatic preliminary classification (Col. 14 lines 61-67; “At 412, the classification label 104 may be excluded from use for subsequent classifying of items. For example, initial classification data 102(1) may include the "Cycling" label as a child label associated with the "Athletic" label, which is in turn a child label associated with the "Shoes" parent label. However, based on the accuracy determination 404, the "Cycling" label may be inaccurate or non-useful for other reasons. Because interaction with the "Cycling" label is not currently useful to users, the classification server(s) 108 may generate modified classification data 102(2) that excludes the "Cycling" label.”).
Forman, Dutta, Riley, Arthur, and Munro are analogous because they are both directed towards the same field of endeavor of machine-learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the label filtering of Dutta.
Doing so would allow for excluding less useful, redundant, or erroneous labels from the classification data, the accuracy and efficiency of an automated classification process may be improved, enabling items to be classified using less processing time and other computing resources, improving the overall speed of the system (Dutta col. 5 lines 20-25;).
Claims 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Forman/Dutta/Riley/Arthur/Munro, as applied above, and further in view of Amershi et al. (US-20150242761-A1).
Regarding Claim 12,
Forman, Dutta, Riley, Arthur and Munro teach the one or more non-transitory computer-readable media of claim 11. Forman further teaches wherein the second pane further displays:
preselects one or more preselected labels of the plurality of labels as applicable to the current training data item based on the automatic preliminary classification (para [0036] “In the present example, the label 82 indicating a valid e-mail message has been pre-selected (e.g., by interface module 55), and the user 57 only needs to click on the radio button for spam label 81 if the user 57 disagrees with this pre-selection. Depending upon the particular sub-embodiment, the pre-selection is based either on the previously assigned classification label or the predicted class for the subject training sample 80. In either event, the task of assigning classification labels generally will be easier for the user 57.”),
wherein the automatic preliminary classification of subsequent data items is updated based on the user selecting labels and unselecting preselected labels corresponding to the current training data item (para [0037] “Once again, a label (in this case, label 93) has been pre-selected, based either on the previously assigned classification label or the predicted classification label for e-mail message 90. Accordingly, the user 57 only needs to click on the appropriate radio button 91, 92, 94 or 95 if he or she disagrees with the pre-selection. And para [0044] In any event, the confirmation/re-labeling information is received for the submitted training examples and, in step 18, that information is used to modify the training set 45 and then retrain the classifier 3, e.g., using training module 5 and the revised training set 45. In the present embodiment, there are two possibilities for each resubmitted training sample 77.”);
Riley further teaches
…wherein the artificial intelligence model sorts the one or more labels based on an estimated likelihood of the one or more labels being applicable to the current training data item (para [0049] “Classification algorithms may also learn by example, using training data that has been organized into classes manually or through some automated process. By observing the relationship between features and classes, the algorithm may learn which features are important in determining the proper label and which keywords provide little or misleading information about the appropriate label for the symptom in question. The result of the training process is a model that may be used later to classify previously unlabeled objects. The classifier processes the features of the objects to classify and uses its model to determine the best label for each object. Depending upon the classification algorithm used, the classifier may emit a single label or it may emit multiple labels, each accompanied by a score or probability that ranks the label against other possible labels for the object in question.” Probability/score (i.e., likelihood).) and
Forman and Riley are analogous because they are both directed towards the same field of endeavor of machine-learning classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the label ranking of Riley.
Doing so would allow for ranking labels based on the probability of the label being assigned to a training sample to determine the best label for the sample (Riley para [0049]).
Arthur further teaches
a toggle (col. 3 lines 52-61; “In another embodiment the menu item training 110 and menu item automatic 115 may be implemented via a single toggle button or any other appropriate user interface element.”),
Munro further teaches
a not-applicable indicium to indicate that at least one of the plurality of training data items is not relevant to any of the one or more labels (Figure 11A; The figure shows a “none are good labels” button.);
Forman, Dutta, Riley, Arthur, and Munro are analogous because they are both directed towards the same field of endeavor of machine-learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the interface for labeling training data of Forman with the interface for labeling training data of Munro.
Doing so would a human user to provide annotations (Munro para [0126] “In some embodiments and as previously alluded to, the display interface for allowing a human annotator to supply an annotation is called a work unit.”).
Forman, Dutta, Riley, and Munro do not explicitly disclose
a toggle,
a function for displaying indicia corresponding to all available labels, when the user determines an applicable label has been incorrectly filtered out based on the automatic preliminary classification.
Dutta teaches filtering out labels based on an automatic preliminary classification (Dutta Col. 14 lines 61-67;). Dutta does not explicitly disclose a function that displays indicia for all available labels.
However, Amershi (US 20150242761 A1) teaches
a function for displaying indicia corresponding to all available labels, when the user determines an applicable label has been incorrectly filtered out based on the automatic preliminary classification (para [0017] “In an exemplary embodiment, the present invention provides a GUI that displays all test and training items, along with their associated labels, and arranges the items according to scores assigned by the machine-learned model.” And para [0082] “In further embodiments, a filtering feature is provided, which enables as user to filter according to data that is desired to be visualized. For example, the filtering feature may enable a user to filter the items such that only item representations that represent items having updated scores that are different from previous scores are displayed.”).
Forman, Dutta, Riley, Munro, Arthur, and Amershi are analogous because they are both directed towards the same field of endeavor of machine-learning with filtering functions.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman, Dutta, and Munro with the GUI of Amershi.
Doing so would allow for visualizing item-level performance, including whether a prediction made by the machine-learned model regarding a particular item agrees with the predetermined label assigned to the item. In this way, the present invention enables a user to quickly identify, prioritize, and inspect item-specific errors (Amershi para [0017]).
Regarding Claim 13,
Forman, Dutta, Riley, Munro, and Amershi the one or more non-transitory computer-readable media of claim 12.
Munro further teaches
and the second pane further displays:
a forward interface element to allow the user to move from the current training data item to a next one of the plurality of training data items (Fig. 10B; para [0117] Referring to FIG. 10B, illustration 1027 shows an example annotation interface to collect input in the form of document annotations/classifications from human analysts and other experts, according to some embodiments. Examiner note: The interface has a “Next Document” button to navigate to the next document (training example).); and
responsive to the input, moving from the current training data item to a next one of the plurality of training data items (Fig. 10B; “para [0154] To determine which documents should be used in this next round of annotations.”).
Forman and Munro are analogous because they are both directed towards the same field of endeavor of machine-learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the interface for labeling training data of Forman with the interface for labeling training data of Munro.
Doing so would a human user to provide annotations (Munro para [0126] “In some embodiments and as previously alluded to, the display interface for allowing a human annotator to supply an annotation is called a work unit.”).
Claim(s) 15-17 and 19-22 are rejected under 35 U.S.C. 103 as being unpatentable over Forman et al. (US-20080103996-A1) in view of Dutta et al. (US-10783167-B1), Fitzgerald (US-20170270526-A1), Riley et al. (US-20150012288-A1), and Arthur et al. (US-7640305-B1).
Regarding Claim 15,
Forman teaches a system comprising:
at least one processor (para [0069]); and
one or more non-transitory computer-readable media that store computer-executable instructions that, when executed by the at least one processor, perform a method of training an artificial intelligence model (para [0069]), the method comprising:
receiving a plurality of training data items… (para [0020] “Referring to FIG. 2, initially (in step 10) an initial training set 45 (shown in FIG. 3) is obtained.”);
ingesting the plurality of training data items (para [0020] “Referring to FIG. 2, initially (in step 10) an initial training set 45 (shown in FIG. 3) is obtained. Initial training set 45 could have been generated in the conventional manner. That is, referring to FIG. 3, various samples 7 are selected and designated for labeling. Ordinarily, the samples 7 are chosen so as to be representative of the types of samples which one desires to classify using the resulting machine-learning classifier (e.g., unlabeled samples 8).”);
determining a plurality of applicable labels within the plurality of training data items (para [0021] “The selected samples 7 are labeled via interface module 43 in order to generate the training set 45, which includes the set of samples and their assigned labels.”);
automatically pre-selecting, using the artificial intelligence model, an automatic preliminary classification to be assigned to one or more items of the plurality of training data items according to a current state of training of the artificial intelligence model (para [0036]-[0038] discloses a preselection of label based on a previous classification (i.e. preliminary classification). para [0023]-[0025] Previous classification is based on a current training state of the ML model.);
generating for display, to the user within the user interface, a plurality of indicia corresponding to a plurality of labels, wherein each label of the plurality of labels is applicable to one or more items of the plurality of training data items (para [0021] “More preferably, module 43 provides a user interface which allows a user 44 to designate an appropriate label for each presented training sample 7. In one representative embodiment, module 43 displays to user 44 (here, a human domain expert) each sample 7 together with a set of radio buttons. The user 44 then clicks on one of the radio buttons, thereby designating the appropriate label for the sample 7.”);
receiving, from the user through the user interface, a selection of one or more selected indicia from the plurality of indicia (para [0021] “More preferably, module 43 provides a user interface which allows a user 44 to designate an appropriate label for each presented training sample 7. In one representative embodiment, module 43 displays to user 44 (here, a human domain expert) each sample 7 together with a set of radio buttons. The user 44 then clicks on one of the radio buttons, thereby designating the appropriate label for the sample 7.”);
responsive to the selection of the one or more selected indicia, generating an updated classification for the one or more items of the plurality of training data items based on the one or more selected indicia (para [0025]-[0027] And para [0044] The classifier is retrained (i.e., updated) based on user label selection.), wherein the updated classification replaces the automatic preliminary classification (para [0036]-[0038] discloses a preselection of label based on a previous classification (i.e. preliminary classification). para [0045]-[0046] The updated classification for the training sample replaces the previous classification (i.e. preliminary classification).),
Forman does not explicitly disclose
…from a training data store;
causing for display of a pre-selection toggle interface element to a user within a user interface, the pre-selection toggle interface element allowing the user to activate automatic classifications of the plurality of training data items;
responsive to an activation of the pre-selection toggle interface element,
storing, with one or more respective items from the plurality of training data items, the one or more selected indicia within a data store, wherein each training data item of the plurality of training data items comprises a statement about a taxpayer's financial situation.
automatically sorting, using the artificial intelligence model, the plurality of labels for at least one subsequent data item of the plurality of training data items based on the updated classification for the one or more items of the plurality of training data items.
However, Dutta (US 10783167 B1) teaches
…training data items from a training data store (col. 5 lines 61-64; “The classification data 102(1) and item data 106 may be stored in association with one or more classification servers 108 or other types of computing devices.”);
storing, with one or more respective items from the plurality of training data items, the one or more selected indicia within a data store (col. 8 lines 13-16; “As additional user interactions 116 associated with the classification labels 104 are determined and stored as user interaction data 120, the classification module 122 may continue to modify the classification data 102(2).” And col. 6 lines 45-50; “For example, the user interaction data 120 may include an average length of time (e.g., “Avg. Time”) spent by users that selected the “Running” label, during which the users viewed or otherwise interacted with item data 106 or other classification labels 104 presented in the user interface 112.”);
Forman and Dutta are analogous because they are both directed towards the same field of endeavor of machine-learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the labeling system of Dutta.
Doing so would allow for excluding less useful, redundant, or erroneous labels from the classification data, the accuracy and efficiency of an automated classification process may be improved, enabling items to be classified using less processing time and other computing resources, improving the overall speed of the system (Dutta col. 5 lines 20-25;).
Fitzgerald (US 20170270526 A1) teaches
wherein each training data item of the plurality of training data items comprises a statement about a taxpayer's financial situation (para [0026] “Tax return submission data store 216 stores, for each submission of tax return data, tax data for that tax return submission, submission data for that tax return submission, and a fraud score or fraud classification for that tax return.”).
Forman and Fitzgerald are analogous because they are both directed towards the same field of endeavor of machine-learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the training of Fitzgerald.
Doing so would allow for refining a set of rules for analyzing documents to improve accuracy (Fitzgerald para [0033]).
Riley (US 20150012288 A1) teaches
automatically sorting, using the artificial intelligence model, the plurality of labels for at least one subsequent data item of the plurality of training data items based on the updated classification for the one or more items of the plurality of training data items (para [0049] “Classification algorithms may also learn by example, using training data that has been organized into classes manually or through some automated process. By observing the relationship between features and classes, the algorithm may learn which features are important in determining the proper label and which keywords provide little or misleading information about the appropriate label for the symptom in question. The result of the training process is a model that may be used later to classify previously unlabeled objects. The classifier processes the features of the objects to classify and uses its model to determine the best label for each object. Depending upon the classification algorithm used, the classifier may emit a single label or it may emit multiple labels, each accompanied by a score or probability that ranks the label against other possible labels for the object in question.” The classifier (i.e., AI model) is trained on training data (i.e., one or more items of the plurality of items) and ranks (i.e., sorts) the labels for an unclassified object (i.e., subsequent data item).);
Forman and Riley are analogous because they are both directed towards the same field of endeavor of machine-learning classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the label ranking of Riley.
Doing so would allow for ranking labels based on the probability of the label being assigned to a training sample to determine the best label for the sample (Riley para [0049]).
Arthur (US 7640305 B1) teaches
causing for display of a pre-selection toggle interface element to a user within a user interface, the pre-selection toggle interface element allowing the user to activate automatic classifications of the plurality of training data items (col. 3 lines 52-61; “The mail system provides the menu item automatic 115 to allow the user to request that the mail system be put into automatic mode. While in automatic mode, the mail system automatically categorizes mail as junk or not junk and takes appropriate actions based on those categorizations. The automatic mode is further described below with reference to FIGS. 4 and 5. In another embodiment the menu item training 110 and menu item automatic 115 may be implemented via a single toggle button or any other appropriate user interface element.” The toggle allows for automatic categorization (i.e., classification) of training items.);
responsive to an activation of the pre-selection toggle interface element (col. 4 lines 37-49; “FIG. 3 depicts a pictorial representation of an example user interface 300 for an inbox, according to an embodiment of the invention. As shown in the user interface 300 for the inbox, during training mode, the mail system has detected that mail 302 may be junk and has displayed message 305 "The mail system thinks this is junk. What do you think?" in order to receive training data or feedback via the buttons junk 310 and not junk 320, which the user may select in response to the message 305. In another embodiment, the functions of the buttons 310 and 320 may be requested via a single toggle button or any other appropriate user interface element. The user may also specify that any mail is junk or not junk without being asked via buttons 310 and 320.”),
Forman and Arthur are analogous because they are both directed towards a machine learning training interface.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the user interface of Foreman with the toggle of Arthur.
Doing so would allow the user to manually select labels for the training sample or automatically having the system label the training sample (Arthur col. 4 lines 37-49;).
Regarding Claim 16,
Forman, Dutta, Riley, Fitzgerald, and Arthur teach the system of claim 15. Forman further teaches wherein the method further comprises:
training the artificial intelligence model based on the one or more respective items, the one or more selected indicia, and the updated classification for the one or more items (para [0044] And para [0047] “In any event, once the training set 45 has been updated, the classifier 3 is retrained using the modified training set 45 (e.g., by training module 5), and the retrained classifier 3 is used to re-process at least some of the labeled training samples 45, thereby obtaining new predictions 4. And para [0064] Finally, in step 138 the training set 45 is modified based on the labels received in step 137, the classifier 3 is retrained based on the modified training set 45, and at least some of the samples 2 and 7 are reprocessed using classifier 3.”).
Regarding Claim 17,
Forman, Dutta, Riley, Fitzgerald, and Arthur teach the system of claim 16. Forman further teaches wherein the automatic preliminary classification of the plurality of training data items occurs in real time (para [0036] “Depending upon the particular sub-embodiment, the pre-selection is based either on the previously assigned classification label or the predicted class for the subject training sample 80.”).
Regarding Claim 19,
Forman, Dutta, Riley, Fitzgerald, and Arthur teach the system of claim 16. Fitzgerald further teaches wherein each label of the plurality of labels corresponds to a tax topic (para [0029] “Each submission of tax return data may further be associated with a fraud score or fraud classification. In some embodiments, the fraud score is a simple binary value (i.e., the return is either determined to be genuine or fraudulent). In other embodiments, the fraud score can take on a range of values, and returns are classified as genuine or fraudulent based on whether their associated fraud score exceeds a predetermined threshold.” Fraudulent or genuine (i.e. tax topic labels).).
Forman, Dutta, Riley, Arthur, and Fitzgerald are analogous because they are both directed towards the same field of endeavor of machine-learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the training of Fitzgerald.
Doing so would allow for refining a set of rules for analyzing documents to improve accuracy (Fitzgerald para [0033]).
Regarding Claim 20,
Forman, Dutta, Riley, Fitzgerald, and Arthur teach the system of claim 19. Fitzgerald further teaches wherein the user is a tax professional and the user interface is generated for display on a user device of the tax professional (para [0021] “User interface engine 206 provides a front end into system 204 for user 202 to enter the tax-related information (referred to herein as tax data) needed to prepare the return. Information input by the user includes information identifying the taxpayer for the return. It should be appreciated that the tax information discussed herein relates to a particular taxpayer, although a user of the invention may be the taxpayer or an authorized third party operating on behalf of the taxpayer, such as a professional tax preparer (“tax professional”) or an authorized agent of the taxpayer.”).
Forman, Dutta, Riley, Arthur, and Fitzgerald are analogous because they are both directed towards the same field of endeavor of machine-learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the training of Fitzgerald.
Doing so would allow for refining a set of rules for analyzing documents to improve accuracy (Fitzgerald para [0033]).
Regarding Claim 21,
Forman, Dutta, Riley, Fitzgerald, and Arthur teach the system of claim 16. Forman further teaches further comprising: classifying, using the artificial intelligence model, one or more real-time data items (para [0023] “Referring back to FIG. 2, in step 12 the classifier 3 is trained, e.g., using training module 5 (shown also in FIG. 3). Generally speaking, the training involves attempting to find an optimal (according to some underlying criteria) mapping from the supplied feature set values for the samples 7 to the corresponding classification labels 8, so that the resulting classifier 3 can receive new unlabeled samples 2 and provide classification labels 4 for them based on its best guess in view of the feature set values for such unlabeled samples 2.”).
Regarding Claim 22,
Forman, Dutta, Riley, and Arthur teach the method of claim 1.
Forman, Dutta, Riley, and Arthur do not explicitly disclose
wherein each training data item of the plurality of training data items comprises a statement about a taxpayer's financial situation.
However, Fitzgerald (US 20170270526 A1) teaches
wherein each training data item of the plurality of training data items comprises a statement about a taxpayer's financial situation (para [0026] “Tax return submission data store 216 stores, for each submission of tax return data, tax data for that tax return submission, submission data for that tax return submission, and a fraud score or fraud classification for that tax return.”).
Forman, Dutta, Riley, Arthur, and Fitzgerald are analogous because they are both directed towards the same field of endeavor of machine-learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the machine learning model of Forman with the training of Fitzgerald.
Doing so would allow for refining a set of rules for analyzing documents to improve accuracy (Fitzgerald para [0033]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HENRY K NGUYEN whose telephone number is (571)272-0217. The examiner can normally be reached Mon - Fri 7:00am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at 5712723768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HENRY NGUYEN/Examiner, Art Unit 2121
Prosecution Insights