Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Regarding rejection of claims under 35 U.S.C. 103, Applicant submits that the amended claims are not taught by the previous cited combination of references. In particular, Applicant submits that the references do not teach the generation of training data as described the claims.
Examiner respectfully disagrees. Claim 1 describes a step of “generating training data according to the learning scope,” the learning scope having been determined:
“determining a learning scope of training the first classifier and the second classifier of the plurality of classifiers, wherein the learning scope comprises one or more consecutive utterance data in the dialogue data of the input data corresponding to the automatically updated at least a part of the first set of the classification results, and the learning scope limits an extent of subsequent training of the first classifier and the second classifier according to the interactively received modification of the second set of the classification results and the automatically updated at least a part of the first set of the classification results by traversing the ascending direction of the hierarchical relationship, and the learning scope specifies a part of a combination of the first classifier and the second classifier in the hierarchical relationship for training”
Neither the claim nor the specification limit the meaning of “generating”; its broadest reasonable interpretation includes merely selecting or designating data to be used in training. The phrase “according to” includes the meaning “consistent with.” The phrase “generating training data according to the learning scope” therefore means the designation of training data can be reasonably described as consistent with the learning scope. Conversely, the “learning scope” can be interpreted as including both the data used in further training, and other data or mechanisms used to determine which part of the classification hierarchy are to be trained.
Singaraju describes a method of identifying training data from a learning scope that teaches these claim elements are recited in the claim:
Singaraju, paragraph 0008, “In some embodiments, the input may correspond to a user utterance to a chatbot, and the plurality of classes may correspond to user intents associated with user utterances [wherein the learning scope comprises one or more consecutive utterance data in the dialogue data of the input data].”
Singaraju, paragraph 0109, “The hierarchical classification model may be updated by independently retraining or otherwise updating a classification model associated with any node of the tree structure, without the need to retrain or update other classification models associated with other nodes of the tree structure [hence including the first classifier and the second classifier of the plurality of classifiers]. For example, in some embodiments, a classification model may be retrained using new training samples, without the need to retrain other classification models. In some embodiments, a classification model may be replaced with a new classification model that is a different type of classification model, and the new classification model may be trained using training samples used to train the original classification model. In some embodiments, some classification models may be retrained or otherwise updated concurrently and independently. For example, when additional training data associated with a particular class is available, such as through feedback from end users regarding previous classification results [described elsewhere as received through a GUI; previous classification results including modifications to multiple levels of the hierarchy, hence, according to the interactively received modification of the second set of the classification results and the automatically updated at least a part of the first set of the classification results] using the hierarchical classification model, classification models for all or some nodes associated with the particular class [hence, training is performed on relevant classification models in the hierarchy, hence, the learning scope specifies a part of a combination of the first classifier and the second classifier in the hierarchical relationship for training] may be retrained using the additional training data [designating this as training data means the learning scope limits an extent of subsequent training of the first classifier and the second classifier][this retraining meaning the method has automatically updated at least a part of the first set of the classification results]. In some embodiments, a new class may be added to the plurality of classes and associated with a set of nodes ( e.g., one node on each layer), and only the set of nodes associated with the new class may be retrained and updated.”
In short, Singaraju teaches selecting particular data for use in further training, which is the broadest reasonable interpretation of “generating training data according to the learning scope,” and further teaches all of the elements of how the learning scope is determined in this claim paragraph, except:
That (bold only) “the learning scope comprises one or more consecutive utterance data in the dialogue data.” This further element is taught by Ariyoshi in paragraph 0078: “That is, the acquisition unit 32 includes training data for prediction, which is the consecutive time-series data of 3 days of the prediction use period highly correlated with the prediction data, and training data for correct answer acquisition of the prediction target period of 2 days, which is subsequent to the training data for prediction, in the training data.” It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the consecutive data of Ariyoshi to the teachings of Singaraju to arrive at the present invention, in order to improve accuracy through correlation of consecutive data in a time sequence, as described in Ariyoshi, paragraph 0078, “However, if data close to the latest usage condition is included in the training data, it is possible to generate a prediction model with high prediction accuracy.”
That the updates are performed “by traversing the ascending direction of the hierarchical relationship.” This further element is taught by Kirshenbaum in paragraph 0029: “In some embodiments, retraining of classifiers (using other classifiers as features) is performed in a ‘bottom-up’ approach, starting with the leaf classes (those that do not dominate any other class, e.g., classes associated with leaf nodes 120 in the partial order data structure 118 of FIG. 1) and moving upwardly in the hierarchy. Classifiers associated with leaf classes are referred to as leaf classifiers.” It would have been obvious for a person having ordinary skill in the art as of the effective filing date of the present invention to have added this bottom-up updating of the hierarchy to the teachings of Singaraju to have created the device of claim 1.
Examiner further notes that “the learning scope limits an extent of subsequent training … according to the interactively received modification,” in its broadest reasonable interpretation, indicates the learning scope limits are based on the interactively received modification, but does not limit the further training only to the classifiers that would be necessarily affected by the modifications, or require the further training to include all the classifiers that would be necessarily affected by the modifications.
The arguments are therefore found unpersuasive.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-3, 5-11, and 13 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites “the training data enables the first classifier and the second classifier in the hierarchical relationship in the plurality of classifiers.” The meaning of the term “enables” is indefinite in this context. The specification provides no definition of the term, and it is not a term of art. The general meaning of the term is “to make possible,” but that sense of the word is not in agreement with its use in this claim, as both “the first classifier and the second classifier” are used in the generation of “the training data,” and therefore must already exist prior to the training data. In computing, “enable” sometimes more narrowly means “to switch on” or “to activate,” as in a phrase such as “enable automatic file backups,” but it is unclear how the training data could be used to activate a classifier in that sense. A person having ordinary skill in the art would therefore be unable to attach a definite meaning to term and would not be able to determine the metes and bounds of the claim.
Claim 8 recites the same language and is rejected by the same argument.
Claims 2-3, 5-11, and 13 are ultimately dependent on claim 1 and are rejected by the same argument.
In further examination below, the claim element will be interpreted as though it read “the training data enables further training of the first classifier and the second classifier in the hierarchical relationship in the plurality of classifiers”; i.e., the data is intended to be used in retraining the classifiers.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, and 8-9 rejected under 35 U.S.C. 103 over Singaraju et al., US Pre-Grant Publication No. 2019/0102701 (hereafter Singaraju) in view of Kirshenbaum et al., US Pre-Grant Publication No. 2008/0154820 (hereafter Kirshenbaum) and Ariyoshi et al., US Pre-Grant Publication No. 2015/0112900 (hereafter Ariyoshi).
Regarding claim 1 and analogous claims 8-9:
Singaraju teaches:
“A learning data generation device comprising a Central Processing Unit (CPU) of a computer configured to execute operations comprising”: Singaraju, paragraph 0004, “In certain embodiments, a computer-implemented method of generating a hierarchical classification model for classifying an input into a class in a plurality of classes is disclosed”; Singaraju, paragraph 0009, “In certain embodiments, a non-transitory computer readable medium may store a plurality of instructions executable by one or more processors”; Singaraju, paragraph 0044, “Where components are described as being ‘configured to’ perform certain operations, such configuration may be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming or controlling electronic circuits ( e.g., microprocessors, or other suitable electronic circuits) to perform the operation [comprising a Central Processing Unit (CPU) of a computer configured to execute operations], or any combination thereof.”
“determining classification results of dialogue data as input data using a plurality of classifiers, wherein the plurality of classifiers comprises a first classifier and a second classifier in a hierarchical relationship”: Singaraju, paragraph 0004, “In certain embodiments, a computer-implemented method of generating a hierarchical classification model for classifying an input into a class in a plurality of classes is disclosed [determining classification results of … input data using a plurality of classifiers]. The method may include associating, by a computer system, the plurality of classes with a root node on a first layer of a tree structure representing the hierarchical classification model, where the hierarchical classification model may include a plurality of classification models. The method may also include generating, by the computer system, the tree structure for the hierarchical classification model, and training, independently by one or more computer systems, the classification model associated with each respective node in the tree structure [wherein the plurality of classifiers comprises a first classifier and a second classifier in a hierarchical relationship]. The tree structure may include leaf nodes and non-leaf nodes. Each non-leaf node may have two child nodes associated with two respective sets of classes in the plurality of classes, where a difference between numbers of classes in the two sets of classes may be zero or one”; Singaraju, paragraph 0008, “In some embodiments, the input may correspond to a user utterance to a chatbot [dialogue data as input data], and the plurality of classes may correspond to user intents associated with user utterances.”
“the classification results represent a multi-class classification of scenes and utterances in a dialogue”: Singaraju, paragraph 0008, “In some embodiments, the input may correspond to a user utterance to a chatbot, and the plurality of classes may correspond to user intents associated with user utterances [the classification results represent a multi-class classification of scenes and utterances in a dialogue].”
“the first classifier is higher in the hierarchical relationship than the second classifier, the classification results comprise a first set of the classification results as output of the first classifier and a second set of the classification results as output of the second classifier, and the second set of the classification results is based on the first set of the classification results according to at least a predictive relationship”: Singaraju, paragraph Fig. 14,
PNG
media_image1.png
561
983
media_image1.png
Greyscale
[showing, for example, classifier 1420 as a first classifier, higher in the hierarchical relationship than classifier 1440, a second classifier, with both classifiers having results, hence, a first set of the classification results as output of the first classifier and a second set of the classification results as output of the second classifier, and where classifier 1440 identifies a portion of the classes that classifier 1420 identifies, hence, classifier 1420’s results could be used to predict classifier 1440’s results or vice-versa, and therefore, the second set of the classification results is based on the first set of the classification results according to at least a predictive relationship].
“interactively receiving, through a graphical user interface, a modification to the second set of the classification results in response to displaying the second set of the classification results”: Singaraju, paragraph 0172, “For example, user interface output devices may include, without limitation, a variety of display devices that visually convey text, graphics [hence, interface can be a graphical user interface] and audio/video information such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, voice output devices, and modems”; Singaraju, paragraph 0109, “The hierarchical classification model may be updated by independently retraining or otherwise updating a classification model associated with any node of the tree structure, without the need to retrain or update other classification models associated with other nodes of the tree structure. For example, in some embodiments, a classification model may be retrained using new training samples, without the need to retrain other classification models. In some embodiments, a classification model may be replaced with a new classification model that is a different type of classification model, and the new classification model may be trained using training samples used to train the original classification model. In some embodiments, some classification models may be retrained or otherwise updated concurrently and independently. For example, when additional training data associated with a particular class is available, such as through feedback from end users regarding previous classification results using the hierarchical classification model [interactively receiving, through a graphical user interface, a modification to the second set of the classification results in response to displaying the second set of the classification results], classification models for all or some nodes associated with the particular class may be retrained using the additional training data.”
(bold only) “automatically updating, based on the received modification to the second set of the classification results, at least a part of the first set of the classification results by traversing in an ascending direction of the hierarchical relationship from the second classifier to the first classifier according to the at least a predictive relationship of the first set and the second set”: Singaraju, paragraph 0109, “The hierarchical classification model may be updated by independently retraining or otherwise updating a classification model associated with any node of the tree structure, without the need to retrain or update other classification models associated with other nodes of the tree structure. For example, in some embodiments, a classification model may be retrained using new training samples, without the need to retrain other classification models. In some embodiments, a classification model may be replaced with a new classification model that is a different type of classification model, and the new classification model may be trained using training samples used to train the original classification model. In some embodiments, some classification models may be retrained or otherwise updated concurrently and independently. For example, when additional training data associated with a particular class is available [based on the received modification to the second set of the classification results], such as through feedback from end users regarding previous classification results using the hierarchical classification model, classification models for all or some nodes associated with the particular class may be retrained using the additional training data [automatically updating … at least a part of the first set of the classification results … according to the at least a predictive relationship of the first set and the second set].”
(bold only) “determining a learning scope of training the first classifier and the second classifier of the plurality of classifiers, wherein the learning scope comprises one or more consecutive utterance data in the dialogue data the input data corresponding to the automatically updated at least a part of the first set of the classification results, and the learning scope limits an extent of subsequent training of the first classifier and the second classifier according to the interactively received modification of the second set of the classification results and the automatically updated at least a part of the first set of the classification results by traversing the ascending direction of the hierarchical relationship, and the learning scope specifies a part of a combination of the first classifier and the second classifier in the hierarchical relationship for training; and”: Singaraju, paragraph 0109, “The hierarchical classification model may be updated by independently retraining or otherwise updating a classification model associated with any node of the tree structure, without the need to retrain or update other classification models associated with other nodes of the tree structure [hence including the first classifier and the second classifier of the plurality of classifiers]. For example, in some embodiments, a classification model may be retrained using new training samples, without the need to retrain other classification models. In some embodiments, a classification model may be replaced with a new classification model that is a different type of classification model, and the new classification model may be trained using training samples used to train the original classification model. In some embodiments, some classification models may be retrained or otherwise updated concurrently and independently. For example, when additional training data associated with a particular class is available, such as through feedback from end users regarding previous classification results [described elsewhere as received through a GUI; previous classification results including modifications to multiple levels of the hierarchy, hence, according to the interactively received modification of the second set of the classification results and the automatically updated at least a part of the first set of the classification results] using the hierarchical classification model, classification models for all or some nodes associated with the particular class [hence, training is performed on relevant classification models in the hierarchy, hence, the learning scope specifies a part of a combination of the first classifier and the second classifier in the hierarchical relationship for training] may be retrained using the additional training data [designating this as training data means the learning scope limits an extent of subsequent training of the first classifier and the second classifier][this retraining meaning the method has automatically updated at least a part of the first set of the classification results]. In some embodiments, a new class may be added to the plurality of classes and associated with a set of nodes ( e.g., one node on each layer), and only the set of nodes associated with the new class may be retrained and updated.”
“generating training data according to the learning scope, wherein the training data comprises a pair of a piece of the input data and a label, and the label represents a classification of the piece of the input data according to the automatically updated at least a part of the first set of the classification results and the second set of the classification results, and the training data enables the first classifier and the second classifier in the hierarchical relationship in the plurality of classifiers”: Singaraju, paragraph 0109, “The hierarchical classification model may be updated by independently retraining or otherwise updating a classification model associated with any node of the tree structure, without the need to retrain or update other classification models associated with other nodes of the tree structure. For example, in some embodiments, a classification model may be retrained using new training samples, without the need to retrain other classification models [the training data enables the first classifier and the second classifier in the hierarchical relationship in the plurality of classifiers]. In some embodiments, a classification model may be replaced with a new classification model that is a different type of classification model, and the new classification model may be trained using training samples used to train the original classification model. In some embodiments, some classification models may be retrained or otherwise updated concurrently and independently. For example, when additional training data associated with a particular class is available, such as through feedback from end users regarding previous classification results using the hierarchical classification model [according to the automatically updated at least a part of the first set of the classification results and the second set of the classification results], classification models for all or some nodes associated with the particular class may be retrained using the additional training data [generating training data according to the learning scope]”; Singaraju, paragraph 1015, “In some embodiments, a LibSVM library may be used for building SVM classification models and multiclass classification models. […] The LibSVM may include a format of: <label>[ wherein the training data comprises … a label]<index 1>:<value 1 >[wherein the training data comprises a pair of a piece of the input data]<index2>: <value2> . . . . Each line in LibSVM may include an instance and is ended by '\n' character. For classification, <label> may include an integer indicating the class label.”
Singaraju does not explicitly teach:
(bold only) “automatically updating, based on the received modification to the second set of the classification results, at least a part of the first set of the classification results by traversing in an ascending direction of the hierarchical relationship from the second classifier to the first classifier according to the at least a predictive relationship of the first set and the second set”
(bold only) “determining a learning scope of training the first classifier and the second classifier of the plurality of classifiers, wherein the learning scope comprises one or more consecutive utterance data in the dialogue data of the input data corresponding to the automatically updated at least a part of the first set of the classification results, and the learning scope limits an extent of subsequent training of the first classifier and the second classifier according to the interactively received modification of the second set of the classification results and the automatically updated at least a part of the first set of the classification results by traversing the ascending direction of the hierarchical relationship, and the learning scope specifies a part of a combination of the first classifier and the second classifier in the hierarchical relationship for training”
Kirshenbaum teaches (bold only) “automatically updating, based on the received modification to the second set of the classification results, at least a part of the first set of the classification results by traversing in an ascending direction of the hierarchical relationship from the second classifier to the first classifier according to the at least a predictive relationship of the first set and the second set” and (bold only) “determining a learning scope of training the first classifier and the second classifier of the plurality of classifiers, wherein the learning scope comprises one or more consecutive utterance data in the dialogue data of the input data corresponding to the automatically updated at least a part of the first set of the classification results, and the learning scope limits an extent of subsequent training of the first classifier and the second classifier according to the interactively received modification of the second set of the classification results and the automatically updated at least a part of the first set of the classification results by traversing the ascending direction of the hierarchical relationship, and the learning scope specifies a part of a combination of the first classifier and the second classifier in the hierarchical relationship for training”: Kirshenbaum, paragraph 0029, “In some embodiments, retraining of classifiers [automatically updating … at least a part of the first set of the classification results] (using other classifiers as features) is performed in a ‘bottom-up’ approach, starting with the leaf classes (those that do not dominate any other class, e.g., classes associated with leaf nodes 120 in the partial order data structure 118 of FIG. 1) and moving upwardly in the hierarchy. Classifiers associated with leaf classes are referred to as leaf classifiers [traversing in an ascending direction of the hierarchical relationship from the second classifier to the first classifier according to the at least a predictive relationship of the first set and the second set][ traversing the ascending direction of the hierarchical relationship]” ; Kirshenbaum, paragraph 0032, “The process (210) is iterated (at 216) to choose the next classifier that does not dominate another class that has not yet been processed. In proceeding up the hierarchy, the modified classifiers are used to add features to training cases of classes that dominate the modified classifiers. Thus, in the context of the example partial order represented with the data structure 118 of FIG. 1, the leaf classifiers (associated with nodes 120) are used to add features to the training cases of classes represented by intermediate nodes 122. Next, the modified classifiers associated with intermediate nodes 122 are used to add features to the training cases for corresponding classes associated with higher-level classes (represented by nodes 124, 126, for example). The process continues up the hierarchy of the partial order (although just a few nodes are depicted in the example partial order data structure 118 of FIG. 1, it is noted that the partial order can include many other nodes). In this manner, the outputs of classifiers are used as features for other classifiers according to the partial order.”
Kirshenbaum and Singaraju are both related to the same field of endeavor (hierarchical classification using machine learning). It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the bottom-up hierarchical classification updates of Kirshenbaum to the hierarchical classification system of Singaraju to arrive at the present invention, in order to propagate improvements from one classifier to another, as stated in Kirshenbaum, paragraph 0005, “A conventional technique of building a multiclass classifier is to train multiple binary classifiers on training sets for each of the classes. The multiple individually trained binary classifiers are then combined to form the multiclass classifier. However, conventional techniques of building multiclass classifiers usually ignore many sources of information (possible features) that may be helpful to build more accurate classifiers.”
Ariyoshi teaches (bold only) “determining a learning scope of training the first classifier and the second classifier of the plurality of classifiers, wherein the learning scope comprises one or more consecutive utterance data in the dialogue data of the input data corresponding to the automatically updated at least a part of the first set of the classification results, and the learning scope limits an extent of subsequent training of the first classifier and the second classifier according to the interactively received modification of the second set of the classification results and the automatically updated at least a part of the first set of the classification results by traversing the ascending direction of the hierarchical relationship, and the learning scope specifies a part of a combination of the first classifier and the second classifier in the hierarchical relationship for training”: Ariyoshi, paragraph 0078, “In the above description, the last time-series data of the prediction data is set as test data, and is not used for the learning of the prediction model. However, if data close to the latest usage condition is included in the training data, it is possible to generate a prediction model with high prediction accuracy. Therefore, the acquisition unit 32 may calculate a correlation between the prediction data and the consecutive time-series data of 3 days of the prediction use period included in the learning data, and select only data, which is determined to have a correlation higher than a predetermined value, as training data for prediction. That is, the acquisition unit 32 includes training data for prediction, which is the consecutive time-series data of 3 days of the prediction use period highly correlated with the prediction data, and training data for correct answer acquisition of the prediction target period of 2 days, which is subsequent to the training data for prediction, in the training data [determining … a learning scope of training the plurality of classifiers, wherein the learning scope comprises one or more consecutive … data, determining a learning scope interpreted as selecting data to be used as, or in constructing, training data]. In this case, the time-series data prediction device 3 may reduce the number of clusters compared with a case where training data is acquired without calculating a correlation, or may not generate the cluster-specific prediction model.”
Ariyoshi and Singaraju are both related to the same field of endeavor (classification using machine learning). It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the consecutive data of Ariyoshi to the teachings of Singaraju to arrive at the present invention, in order to improve accuracy through correlation of consecutive data in a time sequence, as described in Ariyoshi, paragraph 0078, “However, if data close to the latest usage condition is included in the training data, it is possible to generate a prediction model with high prediction accuracy.”
Regarding claim 4:
Singaraju teaches:
“A learning data generation device for generating learning data in a system that performs classification of input data groups using classifiers“: Singaraju, paragraph 0004, “In certain embodiments, a computer-implemented [device] method of generating a hierarchical classification model for classifying an input into a class in a plurality of classes is disclosed [performs classification of input data groups using classifiers]”; Singaraju, paragraph 0009, “In certain embodiments, a non-transitory computer readable medium may store a plurality of instructions executable by one or more processors”; Singaraju, paragraph 0044, “Where components are described as being ‘configured to’ perform certain operations, such configuration may be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming or controlling electronic circuits ( e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.”
(bold only) “the learning data generation device comprising: a computer that determines input data to be a learning scope, on the basis of classification results from a classification of the input data group using the classifiers, generates training data that is the input data determined to be the learning scope to which the classification results of the input data are appended as labels, and inputs a corrected point and a corrected classification result when a correction is made by a user, wherein the computer determines the learning scope to be one or more consecutive input data that include input data corresponding to the corrected classification result”: Singaraju, paragraph 0004, “In certain embodiments, a computer-implemented [a computer] method of generating a hierarchical classification model for classifying an input into a class in a plurality of classes is disclosed”; Singaraju, paragraph 0109, “The hierarchical classification model may be updated by independently retraining or otherwise updating a classification model associated with any node of the tree structure, without the need to retrain or update other classification models associated with other nodes of the tree structure. For example, in some embodiments, a classification model may be retrained using new training samples, without the need to retrain other classification models. In some embodiments, a classification model may be replaced with a new classification model that is a different type of classification model, and the new classification model may be trained using training samples used to train the original classification model. In some embodiments, some classification models may be retrained or otherwise updated concurrently and independently. For example, when additional training data associated with a particular class is available, such as through feedback from end users [inputs a corrected point and a corrected classification result when a correction is made by a user] regarding previous classification results using the hierarchical classification model [determines input data to be a learning scope, on the basis of classification results from a classification of the input data group using the classifiers][wherein the computer determines the learning scope to be one or more consecutive input data that include input data corresponding to the corrected classification result], classification models for all or some nodes associated with the particular class may be retrained using the additional training data [generates training data that is the input data determined to be the learning scope to which the classification results of the input data are appended as labels].”
“the classifiers comprises a first classifier and a second classifier in a hierarchical relationship, the learning scope specifies a part of a combination of a first classifier and a second classifier in the hierarchical relationship for training, and the training data enables learning of the part of a combination of the first classifier and the second classifier in the hierarchical relationship in the plurality of classifiers”: Singaraju, paragraph 0109, “The hierarchical classification model [the classifiers comprises a first classifier and a second classifier in a hierarchical relationship] may be updated by independently retraining or otherwise updating a classification model associated with any node of the tree structure, without the need to retrain or update other classification models associated with other nodes of the tree structure. For example, in some embodiments, a classification model may be retrained using new training samples, without the need to retrain other classification models [the training data enables learning of the part of a combination of the first classifier and the second classifier in the hierarchical relationship in the plurality of classifiers]. In some embodiments, a classification model may be replaced with a new classification model that is a different type of classification model, and the new classification model may be trained using training samples used to train the original classification model. In some embodiments, some classification models may be retrained or otherwise updated concurrently and independently. For example, when additional training data associated with a particular class is available, such as through feedback from end users regarding previous classification results using the hierarchical classification model, classification models for all or some nodes associated with the particular class may be retrained using the additional training data [the learning scope specifies a part of a combination of a first classifier and a second classifier in the hierarchical relationship for training].”
“the learning scope limits an extent of subsequent training of the first classifier and the second classifier according to an automatically updated at least a part of a first set of the classification results by traversing an ascending direction of the hierarchical relationship and an interactively received modification of a second set of the classification results”: Singaraju, paragraph 0109, “The hierarchical classification model may be updated by independently retraining or otherwise updating a classification model associated with any node of the tree structure, without the need to retrain or update other classification models associated with other nodes of the tree structure [the first classifier and the second classifier]. For example, in some embodiments, a classification model may be retrained using new training samples, without the need to retrain other classification models. In some embodiments, a classification model may be replaced with a new classification model that is a different type of classification model, and the new classification model may be trained using training samples used to train the original classification model. In some embodiments, some classification models may be retrained or otherwise updated concurrently and independently. For example, when additional training data associated with a particular class is available, such as through feedback from end users regarding previous classification results [described elsewhere as received through a GUI; previous classification results including modifications to multiple levels of the hierarchy, hence, according to … an interactively received modification of a second set of the classification results] using the hierarchical classification model, classification models for all or some nodes associated with the particular class may be retrained using the additional training data [designating this as training data means the learning scope limits an extent of subsequent training of the first classifier and the second classifier][this retraining meaning the method has an automatically updated at least a part of a first set of the classification results]. In some embodiments, a new class may be added to the plurality of classes and associated with a set of nodes ( e.g., one node on each layer), and only the set of nodes associated with the new class may be retrained and updated.”
Singaraju does not explicitly teach
(bold only) “the learning data generation device comprising: a computer that determines input data to be a learning scope, on the basis of classification results from a classification of the input data group using the classifiers, generates training data that is the input data determined to be the learning scope to which the classification results of the input data are appended as labels, and inputs a corrected point and a corrected classification result when a correction is made by a user, wherein the computer determines the learning scope to be one or more consecutive input data that include input data corresponding to the corrected classification result.”
(bold only) “the learning scope limits an extent of subsequent training of the first classifier and the second classifier according to an automatically updated at least a part of a first set of the classification results by traversing an ascending direction of the hierarchical relationship and an interactively received modification of a second set of the classification results”
Ariyoshi teaches (bold only) “the learning data generation device comprising: a computer that determines input data to be a learning scope, on the basis of classification results from a classification of the input data group using the classifiers, generates training data that is the input data determined to be the learning scope to which the classification results of the input data are appended as labels, and inputs a corrected point and a corrected classification result when a correction is made by a user, wherein the computer determines the learning scope to be one or more consecutive input data that include input data corresponding to the corrected classification result”: Ariyoshi, paragraph 0078, “In the above description, the last time-series data of the prediction data is set as test data, and is not used for the learning of the prediction model. However, if data close to the latest usage condition is included in the training data, it is possible to generate a prediction model with high prediction accuracy. Therefore, the acquisition unit 32 may calculate a correlation between the prediction data and the consecutive time-series data of 3 days of the prediction use period included in the learning data, and select only data, which is determined to have a correlation higher than a predetermined value, as training data for prediction. That is, the acquisition unit 32 includes training data for prediction, which is the consecutive time-series data of 3 days of the prediction use period highly correlated with the prediction data, and training data for correct answer acquisition of the prediction target period of 2 days, which is subsequent to the training data for prediction, in the training data [the computer determines the learning scope to be one or more consecutive … data, determining a learning scope interpreted as selecting data to be used as, or in constructing, training data]. In this case, the time-series data prediction device 3 may reduce the number of clusters compared with a case where training data is acquired without calculating a correlation, or may not generate the cluster-specific prediction model.”
Ariyoshi and Singaraju are both related to the same field of endeavor (classification using machine learning). It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the consecutive data of Ariyoshi to the teachings of Singaraju to arrive at the present invention, in order to improve accuracy through correlation of consecutive data in a time sequence, as described in Ariyoshi, paragraph 0078, “However, if data close to the latest usage condition is included in the training data, it is possible to generate a prediction model with high prediction accuracy.”
Kirshenbaum teaches (bold only) “the learning scope limits an extent of subsequent training of the first classifier and the second classifier according to an automatically updated at least a part of a first set of the classification results by traversing an ascending direction of the hierarchical relationship and an interactively received modification of a second set of the classification results”: Kirshenbaum, paragraph 0029, “In some embodiments, retraining of classifiers [automatically updated at least a part of a first set of the classification results] (using other classifiers as features) is performed in a ‘bottom-up’ approach, starting with the leaf classes (those that do not dominate any other class, e.g., classes associated with leaf nodes 120 in the partial order data structure 118 of FIG. 1) and moving upwardly in the hierarchy. Classifiers associated with leaf classes are referred to as leaf classifiers [traversing an ascending direction of the hierarchical relationship]” ; Kirshenbaum, paragraph 0032, “The process (210) is iterated (at 216) to choose the next classifier that does not dominate another class that has not yet been processed. In proceeding up the hierarchy, the modified classifiers are used to add features to training cases of classes that dominate the modified classifiers. Thus, in the context of the example partial order represented with the data structure 118 of FIG. 1, the leaf classifiers (associated with nodes 120) are used to add features to the training cases of classes represented by intermediate nodes 122. Next, the modified classifiers associated with intermediate nodes 122 are used to add features to the training cases for corresponding classes associated with higher-level classes (represented by nodes 124, 126, for example). The process continues up the hierarchy of the partial order (although just a few nodes are depicted in the example partial order data structure 118 of FIG. 1, it is noted that the partial order can include many other nodes). In this manner, the outputs of classifiers are used as features for other classifiers according to the partial order.”
Kirshenbaum and Singaraju are both related to the same field of endeavor (hierarchical classification using machine learning). It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the bottom-up hierarchical classification updates of Kirshenbaum to the hierarchical classification system of Singaraju to arrive at the present invention, in order to propagate improvements from one classifier to another, as stated in Kirshenbaum, paragraph 0005, “A conventional technique of building a multiclass classifier is to train multiple binary classifiers on training sets for each of the classes. The multiple individually trained binary classifiers are then combined to form the multiclass classifier. However, conventional techniques of building multiclass classifiers usually ignore many sources of information (possible features) that may be helpful to build more accurate classifiers.”
Claims 2-3 rejected under 35 U.S.C. 103 over Singaraju as modified by Kirshenbaum and Ariyoshi in view of Xue et al., “Deep Classification in Large-Scale Text Hierarchies,” 2008, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, 619-626 (hereafter Xue).
Regarding claim 2:
Singaraju as modified by Kirshenbaum and Ariyoshi teaches the learning data generation device according to claim 1.
Singaraju further teaches (bold only) “wherein the computer inputs a corrected point and a corrected classification result when a correction is made by a user, and determines the learning scope to be one or more consecutive input data that include input data corresponding to the corrected classification result and have the same classification result of the top level classifier”: Singaraju, paragraph 0109, “The hierarchical classification model may be updated by independently retraining or otherwise updating a classification model associated with any node of the tree structure, without the need to retrain or update other classification models associated with other nodes of the tree structure. For example, in some embodiments, a classification model may be retrained using new training samples, without the need to retrain other classification models. In some embodiments, a classification model may be replaced with a new classification model that is a different type of classification model, and the new classification model may be trained using training samples used to train the original classification model. In some embodiments, some classification models may be retrained or otherwise updated concurrently and independently. For example, when additional training data associated with a particular class is available, such as through feedback from end users regarding previous classification results [inputs a corrected point and a corrected classification result when a correction is made by a user] using the hierarchical classification model, classification models for all or some nodes associated with the particular class may be retrained using the additional training data [determines the learning scope to be one or more … input data that include input data corresponding to the corrected classification result].”
Ariyoshi further teaches (bold only) “wherein the computer inputs a corrected point and a corrected classification result when a correction is made by a user, and determines the learning scope to be one or more consecutive input data that include input data corresponding to the corrected classification result and have the same classification result of the top level classifier”: Ariyoshi, paragraph 0078, “In the above description, the last time-series data of the prediction data is set as test data, and is not used for the learning of the prediction model. However, if data close to the latest usage condition is included in the training data, it is possible to generate a prediction model with high prediction accuracy. Therefore, the acquisition unit 32 may calculate a correlation between the prediction data and the consecutive time-series data of 3 days of the prediction use period included in the learning data, and select only data, which is determined to have a correlation higher than a predetermined value, as training data for prediction. That is, the acquisition unit 32 includes training data for prediction, which is the consecutive time-series data of 3 days of the prediction use period highly correlated with the prediction data, and training data for correct answer acquisition of the prediction target period of 2 days, which is subsequent to the training data for prediction, in the training data [determines the learning scope to be one or more consecutive input data, determining a learning scope interpreted as selecting data to be used as, or in constructing, training data]. In this case, the time-series data prediction device 3 may reduce the number of clusters compared with a case where training data is acquired without calculating a correlation, or may not generate the cluster-specific prediction model.”
Ariyoshi and Singaraju are combinable for the rationale given under claim 1.
Xue teaches (bold only) “wherein the computer inputs a corrected point and a corrected classification result when a correction is made by a user, and determines the learning scope to be one or more consecutive input data that include input data corresponding to the corrected classification result and have the same classification result of the top level classifier”: Xue, section 5.1.3. paragraph 1, “However, as discussed in Section 1, an ideal strategy for training data selection should take this structural information into account. Thus, we propose the ancestor-assistant strategy to utilize this information. This strategy is guided by the following two observations. First, the training data from the category candidate itself may be insufficient in size, especially for a deep category. Thus, we need to obtain more data elsewhere. Second, although the training data from its higher up ancestors may be too general to reflect the characteristics of the deep category candidate, we can borrow data from the ancestors. We should not do this for ancestors that are too high up. Hence, we propose a trade-off between the hierarchical strategy and flat strategy by combining the training data from the category candidate itself and the training data [determines the learning scope, interpreted as selecting data to be used as, or in constructing, training data] from its ancestors [have the same classification result of the top level classifier], as long as they do not share the common ancestors of other category candidates. By considering the structure of the hierarchy, the scarcity of training data on deep categories can be alleviated. In addition, we include the training data from a node itself to reserve the characteristics of the categories and the training data will not be largely affected by the training data from its ancestors.”
Xue and Singaraju are both related to the same field of endeavor (classification using machine learning). It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the hierarchical training of Xue to the teachings of Singaraju to arrive at the present invention, in order to improve model training, as described in Xue, section 5.1.3, paragraph 1, “By considering the structure of the hierarchy, the scarcity of training data on deep categories can be alleviated. In addition, we include the training data from a node itself to reserve the characteristics of the categories and the training data will not be largely affected by the training data from its ancestors.”
Regarding claim 3:
Singaraju as modified by Kirshenbaum, Ariyoshi, and Xue teaches the learning data generation device according to claim 2.
Kirshenbaum further teaches “wherein the computer, when a classification result of a classifier of a particular level is corrected, corrects classification results of classifiers of levels higher than said particular level to conform to the correction of the classification result, or excludes classification results of classifiers of levels lower than said particular level from the training data”: Kirshenbaum, paragraph 0029, “In some embodiments, retraining of (using other classifiers as features) is performed in a ‘bottom-up’ approach, starting with the leaf classes (those that do not dominate any other class, e.g., classes associated with leaf nodes 120 in the partial order data structure 118 of FIG. 1) and moving upwardly in the hierarchy. Classifiers associated with leaf classes are referred to as leaf classifiers” ; Kirshenbaum, paragraph 0032, “The process (210) is iterated (at 216) to choose the next classifier that does not dominate another class that has not yet been processed. In proceeding up the hierarchy, the modified classifiers are used to add features to training cases of classes that dominate the modified classifiers [when a classification result of a classifier of a particular level is corrected, corrects classification results of classifiers of levels higher than said particular level to conform to the correction of the classification result]. Thus, in the context of the example partial order represented with the data structure 118 of FIG. 1, the leaf classifiers (associated with nodes 120) are used to add features to the training cases of classes represented by intermediate nodes 122. Next, the modified classifiers associated with intermediate nodes 122 are used to add features to the training cases for corresponding classes associated with higher-level classes (represented by nodes 124, 126, for example). The process continues up the hierarchy of the partial order (although just a few nodes are depicted in the example partial order data structure 118 of FIG. 1, it is noted that the partial order can include many other nodes). In this manner, the outputs of classifiers are used as features for other classifiers according to the partial order.”
Kirshenbaum and Singaraju are combinable for the rationale given in claim 1.
Claims 5 and 7 rejected under 35 U.S.C. 103 over Singaraju as modified by Kirshenbaum and Ariyoshi in view of over Gokalp, US Pre-Grant Publication No. 2023/0376857 (hereafter Gokalp).
Regarding claim 5:
Singaraju as modified by Kirshenbaum and Ariyoshi teaches the learning generation device of claim 1.
Singaraju further teaches “wherein the computer performs a multi-class classification of the input data group using a plurality of classifiers and generates classification results”: Singaraju, paragraph 0004, “In certain embodiments, a computer-implemented method of generating a hierarchical classification model for classifying an input into a class in a plurality of classes is disclosed [a multi-class classification of the input data group using a plurality of classifiers and generates classification results].”
Singaraju as modified by Kirshenbaum and Ariyoshi does not explicitly teach:
“and generates a learning form having the classification results and a correction interface for rectifying the classification results and causes the learning form to be displayed on a display”
“and the classification results are corrected using the correction interface displayed on the display”
Gokalp teaches:
“and generates a learning form having the classification results and a correction interface for rectifying the classification results and causes the learning form to be displayed on a display”: Gokalp, Fig. 19,
PNG
media_image2.png
801
1226
media_image2.png
Greyscale
; Gokalp, paragraph 0108, “In at least some embodiments, a set of interactive programmatic interfaces which is organized using tabs into portions corresponding (at least approximately) to different stages of classifier development and/or to different user roles may be implemented ”; Gokalp, paragraph 0113, “The labeling candidate data items themselves may each be represented by a respective panel 1914, such as 1914A and 1914B in the depicted embodiment. For a given data item 1914, several pieces of information similar to those shown in FIG. 7 may be provided---e.g., an item image 1916 may be shown, an image title 1919 may be presented, description details 1917 may be provided, and additional summarized attributes 1918 may be included in the display in various embodiments [generates a learning form having the classification results and a correction interface for rectifying the classification results and causes the learning form to be displayed on a display].”
“and the classification results are corrected using the correction interface displayed on the display”: Gokalp, paragraph 0138, “One or more training iterations may be initiated for the classification problem being addressed (element 2607), in which the resources identified in operations corresponding to element 2604 may be utilized. At a high level, a given training iteration may comprise at least two categories of operations in various embodiments: back-end operations at the classification service, in which one or more classifiers may be trained using the available labeled data, and front-end or client-side operations, in which new labels (or, in some cases, corrected labels) may be requested and obtained from a set of label providers via a set of guided labeling feedback sessions [the classification results are corrected using the correction interface displayed on the display].”
Gokalp and Singaraju are both related to the same field of endeavor (classification using machine learning). It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the user interface of Gokalp to the teachings of Singaraju to arrive at the present invention, in order to provide an intuitive method for users to update or correct classifications, as stated in Gokalp, paragraph 0059, “In interactive labeling sessions 183, in various embodiments one or more label providers may be presented at client devices 180C with candidate data items for which labeling feedback is requested, and such label providers may submit labels for the candidate items, submit filtering requests to view and/or label additional data items, and so on. Intuitive, easy-to-use feature-rich customizable interactive programmatic interfaces 177 may be provided for each of the three categories of user sessions indicated in FIG. 1 in various embodiments; details of various aspects of the interfaces are provided below.”
Regarding claim 7:
Singaraju as modified by Kirshenbaum, Ariyoshi, and Gokalp teaches the learning data generation device according to claim 5.
Gokalp further teaches “wherein the computer generates a correction interface including a button for adding a classification result, a button for deleting a classification result, and a region for inputting a corrected classification result”: Gokalp, Fig. 19,
PNG
media_image2.png
801
1226
media_image2.png
Greyscale
[showing example of graphical user interface for interacting with classes and data items, including buttons, regions, etc.; button interpreted as a selectable element in a graphical user interface, region interpreted as a display area in a graphical user interface]; Gokalp, paragraph 0087, “In at least one embodiment, the classification service may generate class predictions for at least some data items for which labels have already been provided, e.g., in order to determine the extent to which the classifier differs in its conclusions from the label providers. FIG. 9 illustrates an example scenario in which a label provider may be requested, via an interactive interface [a correction interface], to reconsider whether a previously-provided label is appropriate for a labeling feedback candidate, according to at least some embodiments [inputting a corrected classification result]”; Gokalp, paragraph 0091, “In some embodiments, if and when a user eventually decides that all the data items that were assigned a particular user-defined label should be designated as members of a particular target class, a label filter 1004 may be used to retrieve all the items to which the user-defined label was assigned, and an interface element similar to the ‘label all option’ shown in FIG. 7 may be used to assign the data items in bulk to the particular target class [adding a classification result]. Such an approach may enable the label providers to avoid labeling such data items one at a time, thereby further enhancing the user experience of the label providers. In at least some embodiments, if a label provider decides that all the items with a user-defined label are to be assigned to a target class, the metadata stored at the classification service regarding the user-defined label may optionally be deleted-that is, information about user-defined labels may only be retained/stored for periods during which a decision about the target class of the items assigned the user-defined label has not yet been made [deleting a classification result].”
Gokalp and Singaraju are combinable for the rationale given under claim 5.
Claims 6 and 13 rejected as unpatentable under Singaraju as modified by Kirshenbaum, Ariyoshi, and Gokalp in view of Cahoon et al., US Patent No. 11,880,746 (hereafter Cahoon).
Regarding claim 6:
Singaraju as modified by Kirshenbaum, Ariyoshi, and Gokalp teaches the learning data generation device according to claim 5.
Singaraju as modified by Kirshenbaum, Ariyoshi, and Gokalp does not explicitly teach “wherein the computer generates a learning form which shows, in a categorized manner for the respective classification results, the classification results from the top level classifier, and shows, within a region for displaying the classification results of the top level classifier, classification results for the classifiers of the respective lower levels.“
Cahoon teaches “the computer generates a learning form which shows, in a categorized manner for the respective classification results, the classification results from the top level classifier, and shows, within a region for displaying the classification results of the top level classifier, classification results for the classifiers of the respective lower levels”: Cahoon, col. 6, lines 25-35, “In some embodiments, a hierarchy of labels may be used instead, such that selecting a label 308 causes pane 306 to display an additional set of sublabels corresponding to the selected label. For example, in the image-recognition example described above, the top-level labels might correspond to ‘person,’ ‘animal,’ ‘plant,’ ‘man-made object’ and so forth. When the subject-matter expert selects ‘animal,’ a new set of sublabels including ‘cat’ ‘dog,’ ‘horse,’ and so on might be displayed [shows, in a categorized manner for the respective classification results, the classification results from the top level classifier, and shows, within a region for displaying the classification results of the top level classifier, classification results for the classifiers of the respective lower levels]. In this way, a larger set of labels can be employed than would be practical without a label hierarchy.”
Cahoon and Singaraju are both related to the same field of endeavor (classification using machine learning). It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the hierarchical classification display of Cahoon to the teachings of Singaraju to arrive at the present invention, in order to provide a greater amount of useful information on the display at the same time, as described in Cahoon, paragraph 0078, “In this way, a larger set of labels can be employed than would be practical without a label hierarchy.”
Regarding claim 13:
Singaraju as modified by Kirshenbaum, Ariyoshi, Gokalp, and Cahoon teaches the learning data generation device according to claim 6.
Gokalp further teaches “wherein the computer generates a rectification interface including a button for adding a classification result, a button for deleting a classification result, and a region for inputting a rectified classification result”: Gokalp, Fig. 19,
PNG
media_image2.png
801
1226
media_image2.png
Greyscale
[showing example of graphical user interface for interacting with classes and data items, including buttons, regions, etc.; button interpreted as a selectable element in a graphical user interface, region interpreted as a display area in a graphical user interface]; Gokalp, paragraph 0087, “In at least one embodiment, the classification service may generate class predictions for at least some data items for which labels have already been provided, e.g., in order to determine the extent to which the classifier differs in its conclusions from the label providers. FIG. 9 illustrates an example scenario in which a label provider may be requested, via an interactive interface [a rectification interface], to reconsider whether a previously-provided label is appropriate for a labeling feedback candidate, according to at least some embodiments [inputting a corrected classification result]”; Gokalp, paragraph 0091, “In some embodiments, if and when a user eventually decides that all the data items that were assigned a particular user-defined label should be designated as members of a particular target class, a label filter 1004 may be used to retrieve all the items to which the user-defined label was assigned, and an interface element similar to the ‘label all option’ shown in FIG. 7 may be used to assign the data items in bulk to the particular target class [adding a classification result]. Such an approach may enable the label providers to avoid labeling such data items one at a time, thereby further enhancing the user experience of the label providers. In at least some embodiments, if a label provider decides that all the items with a user-defined label are to be assigned to a target class, the metadata stored at the classification service regarding the user-defined label may optionally be deleted-that is, information about user-defined labels may only be retained/stored for periods during which a decision about the target class of the items assigned the user-defined label has not yet been made [deleting a classification result].”
Gokalp and Singaraju are combinable for the rationale given under claim 5.
Claims 10-11 rejected as unpatentable under Singaraju as modified by Kirshenbaum, Ariyoshi, and Xue in view of Gokalp.
Regarding claim 10:
Singaraju as modified by Kirshenbaum, Ariyoshi, and Xue teaches the learning data generation device of claim 2.
Singaraju further teaches (bold only) “wherein the computer performs a multilevel classification of the input data group using a plurality of classifiers and generates classification results”: Singaraju, paragraph 0004, “In certain embodiments, a computer-implemented method of generating a hierarchical classification model for classifying an input into a class in a plurality of classes is disclosed [wherein the computer performs a multilevel classification of the input data group using a plurality of classifiers and generates classification results].”
Singaraju as modified by Kirshenbaum, Ariyoshi, and Xue does not explicitly teach:
“generates a learning screen having the classification results and a rectification interface for rectifying the classification results and causes the learning screen to be displayed on a display”
“and the classification results are rectified using the rectification interface displayed on the display”
Gokalp teaches:
“generates a learning screen having the classification results and a rectification interface for rectifying the classification results and causes the learning screen to be displayed on a display”: Gokalp, Fig. 19,
PNG
media_image2.png
801
1226
media_image2.png
Greyscale
; Gokalp, paragraph 0108, “In at least some embodiments, a set of interactive programmatic interfaces which is organized using tabs into portions corresponding (at least approximately) to different stages of classifier development and/or to different user roles may be implemented ”; Gokalp, paragraph 0113, “The labeling candidate data items themselves may each be represented by a respective panel 1914, such as 1914A and 1914B in the depicted embodiment. For a given data item 1914, several pieces of information similar to those shown in FIG. 7 may be provided---e.g., an item image 1916 may be shown, an image title 1919 may be presented, description details 1917 may be provided, and additional summarized attributes 1918 may be included in the display in various embodiments [generates a learning form having the classification results and a correction interface for rectifying the classification results and causes the learning form to be displayed on a display].”
“and the classification results are rectified using the rectification interface displayed on the display”: Gokalp, paragraph 0138, “One or more training iterations may be initiated for the classification problem being addressed (element 2607), in which the resources identified in operations corresponding to element 2604 may be utilized. At a high level, a given training iteration may comprise at least two categories of operations in various embodiments: back-end operations at the classification service, in which one or more classifiers may be trained using the available labeled data, and front-end or client-side operations, in which new labels (or, in some cases, corrected labels) may be requested and obtained from a set of label providers via a set of guided labeling feedback sessions [the classification results are corrected using the correction interface displayed on the display].”
Gokalp and Singaraju are both related to the same field of endeavor (classification using machine learning). It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the user interface of Gokalp to the teachings of Singaraju to arrive at the present invention, in order to provide an intuitive method for users to update or correct classifications, as stated in Gokalp, paragraph 0059, “In interactive labeling sessions 183, in various embodiments one or more label providers may be presented at client devices 180C with candidate data items for which labeling feedback is requested, and such label providers may submit labels for the candidate items, submit filtering requests to view and/or label additional data items, and so on. Intuitive, easy-to-use feature-rich customizable interactive programmatic interfaces 177 may be provided for each of the three categories of user sessions indicated in FIG. 1 in various embodiments; details of various aspects of the interfaces are provided below.”
Regarding claim 11:
Singaraju as modified by Kirshenbaum, Ariyoshi, and Xue teaches the learning data generation device of claim 2.
Singaraju further teaches (bold only) “wherein the computer performs a multilevel classification of the input data group using a plurality of classifiers and generates classification results”: Singaraju, paragraph 0004, “In certain embodiments, a computer-implemented method of generating a hierarchical classification model for classifying an input into a class in a plurality of classes is disclosed [wherein the computer performs a multilevel classification of the input data group using a plurality of classifiers and generates classification results].”
Singaraju as modified by Kirshenbaum, Ariyoshi, and Xue does not explicitly teach:
“generates a learning screen having the classification results and a rectification interface for rectifying the classification results and causes the learning screen to be displayed on a display”
“and the classification results are rectified using the rectification interface displayed on the display”
Gokalp teaches:
“generates a learning screen having the classification results and a rectification interface for rectifying the classification results and causes the learning screen to be displayed on a display”: Gokalp, Fig. 19,
PNG
media_image2.png
801
1226
media_image2.png
Greyscale
; Gokalp, paragraph 0108, “In at least some embodiments, a set of interactive programmatic interfaces which is organized using tabs into portions corresponding (at least approximately) to different stages of classifier development and/or to different user roles may be implemented ”; Gokalp, paragraph 0113, “The labeling candidate data items themselves may each be represented by a respective panel 1914, such as 1914A and 1914B in the depicted embodiment. For a given data item 1914, several pieces of information similar to those shown in FIG. 7 may be provided---e.g., an item image 1916 may be shown, an image title 1919 may be presented, description details 1917 may be provided, and additional summarized attributes 1918 may be included in the display in various embodiments [generates a learning form having the classification results and a correction interface for rectifying the classification results and causes the learning form to be displayed on a display].”
“and the classification results are rectified using the rectification interface displayed on the display”: Gokalp, paragraph 0138, “One or more training iterations may be initiated for the classification problem being addressed (element 2607), in which the resources identified in operations corresponding to element 2604 may be utilized. At a high level, a given training iteration may comprise at least two categories of operations in various embodiments: back-end operations at the classification service, in which one or more classifiers may be trained using the available labeled data, and front-end or client-side operations, in which new labels (or, in some cases, corrected labels) may be requested and obtained from a set of label providers via a set of guided labeling feedback sessions [the classification results are corrected using the correction interface displayed on the display].”
Gokalp and Singaraju are both related to the same field of endeavor (classification using machine learning). It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the user interface of Gokalp to the teachings of Singaraju to arrive at the present invention, in order to provide an intuitive method for users to update or correct classifications, as stated in Gokalp, paragraph 0059, “In interactive labeling sessions 183, in various embodiments one or more label providers may be presented at client devices 180C with candidate data items for which labeling feedback is requested, and such label providers may submit labels for the candidate items, submit filtering requests to view and/or label additional data items, and so on. Intuitive, easy-to-use feature-rich customizable interactive programmatic interfaces 177 may be provided for each of the three categories of user sessions indicated in FIG. 1 in various embodiments; details of various aspects of the interfaces are provided below.”
Claim 12 rejected under 35 U.S.C. 103 over Singaraju as modified by Ariyoshi in view of Gokalp.
Singaraju as modified by Ariyoshi teaches the learning data generation device of claim 2.
Singaraju further teaches (bold only) “wherein the computer performs a multilevel classification of the input data group using a plurality of classifiers and generates classification results”: Singaraju, paragraph 0004, “In certain embodiments, a computer-implemented method of generating a hierarchical classification model for classifying an input into a class in a plurality of classes is disclosed [wherein the computer performs a multilevel classification of the input data group using a plurality of classifiers and generates classification results].”
Singaraju as modified by Ariyoshi does not explicitly teach:
“generates a learning screen having the classification results and a rectification interface for rectifying the classification results and causes the learning screen to be displayed on a display”
“and the classification results are rectified using the rectification interface displayed on the display”
Gokalp teaches:
“generates a learning screen having the classification results and a rectification interface for rectifying the classification results and causes the learning screen to be displayed on a display”: Gokalp, Fig. 19,
PNG
media_image2.png
801
1226
media_image2.png
Greyscale
; Gokalp, paragraph 0108, “In at least some embodiments, a set of interactive programmatic interfaces which is organized using tabs into portions corresponding (at least approximately) to different stages of classifier development and/or to different user roles may be implemented ”; Gokalp, paragraph 0113, “The labeling candidate data items themselves may each be represented by a respective panel 1914, such as 1914A and 1914B in the depicted embodiment. For a given data item 1914, several pieces of information similar to those shown in FIG. 7 may be provided---e.g., an item image 1916 may be shown, an image title 1919 may be presented, description details 1917 may be provided, and additional summarized attributes 1918 may be included in the display in various embodiments [generates a learning form having the classification results and a correction interface for rectifying the classification results and causes the learning form to be displayed on a display].”
“and the classification results are rectified using the rectification interface displayed on the display”: Gokalp, paragraph 0138, “One or more training iterations may be initiated for the classification problem being addressed (element 2607), in which the resources identified in operations corresponding to element 2604 may be utilized. At a high level, a given training iteration may comprise at least two categories of operations in various embodiments: back-end operations at the classification service, in which one or more classifiers may be trained using the available labeled data, and front-end or client-side operations, in which new labels (or, in some cases, corrected labels) may be requested and obtained from a set of label providers via a set of guided labeling feedback sessions [the classification results are corrected using the correction interface displayed on the display].”
Gokalp and Singaraju are both related to the same field of endeavor (classification using machine learning). It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the user interface of Gokalp to the teachings of Singaraju to arrive at the present invention, in order to provide an intuitive method for users to update or correct classifications, as stated in Gokalp, paragraph 0059, “In interactive labeling sessions 183, in various embodiments one or more label providers may be presented at client devices 180C with candidate data items for which labeling feedback is requested, and such label providers may submit labels for the candidate items, submit filtering requests to view and/or label additional data items, and so on. Intuitive, easy-to-use feature-rich customizable interactive programmatic interfaces 177 may be provided for each of the three categories of user sessions indicated in FIG. 1 in various embodiments; details of various aspects of the interfaces are provided below.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Stokes et al., “Asking for a Second Opinion: Re-Querying of Noisy Multi-Class Labels,” 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE Press, 2329–2333. https://doi.org/10.1109/ICASSP.2016.7472093, discloses methods for automatically selecting data from a set of mislabeled classifier output, for use in updating or retraining the classifier, in which the classifier may be an ensemble.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT SPRAUL whose telephone number is (703) 756-1511. The examiner can normally be reached M-F 9:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MICHAEL HUNTLEY can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VAS/Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights