Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
3. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office Action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
4. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
5. Claims 1-6, 8-11, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over CN 115034300 A (“Bao”) in view of U.S. Patent No. 11416904 (“He”, as cited in Applicants’ IDS submission) and further in view of Non-Patent Literature “Partial Label Learning via Feature-Aware Disambiguation” (“Zhang”) and Non-Patent Literature “Learning from Ambiguously Labeled Examples” (“Hüllermeier”).
Regarding claim 1, BAO teaches A system for selecting labels based on dataset similarity for machine learning model training ... text data (see the problem/need statement found at the top of page 2, discussing challenges in model training that effect efficiency, cost, etc. resulting from the labeling of the data used to the train the model, and further that a “partial label” approach to model training, as mentioned there, and as detailed just below on the same page (under the heading Contents of the Invention), see the discussion of the classification model training method with steps for obtaining, inputting, and conditioned determining and outputting – which includes a classification result that is akin to label selection that addressed the problem/need as mentioned just prior; and further, where the selection of the label is performed in part by comparing one candidate label to a target label in terms of accuracy, and making the label selection in accordance with that accuracy-based comparison (i.e., a machine learning basis for “dataset similarity” as recited); and further, where the text that is being subject to classification is text data (as discussed on page 6 in relation to FIG. 1 step 102: a privacy protocol that a user must read and understand and agree to)) comprising:
one or more processors; and a non-transitory, computer-readable medium (“computer-readable storage medium” and “processor”, as mentioned in relation to an embodiment of the inventive classification model training (see numbered page 4, with respect to the taught third and fourth aspects), and see also the discussion relating to FIG. 5 on page 16 for essentially the same) comprising instructions that when executed by the one or more processors cause operations comprising:
receiving, at a device on a computer network (a device coupled to a network, capable of practicing the invention as taught, as discussed in relation to FIG. 5 on page 16), a label modification request for a training datum for a machine learning model (the label comparison and selection process, as noted above and as discussed in accordance with the invention’s first aspect as found on pages 2-3, where each label comparison in terms of accuracy that could result in what is essentially a label correction/modification is akin to a “label modification request” as recited), wherein the label modification request comprises a new label and a datum identifier, and wherein the new label comprises a category of ... data (the label and the data for it are understood to be in relation to a classifier and hence a class, which is understood to be a category as recited, and further any persistent data in a system such as the one discussed in relation to FIG. 5, and the label information for it as taught, would be understood to be stored using a memory scheme as generally understood and widely practiced in the state of the art and would therefore correspond to a memory address (i.e., “datum identifier” as recited));
retrieving, from a label record database, a ... label record ... corresponding to the datum identifier, wherein the ... label record ... comprises a ... pre-existing label ... previously applied to the training datum (as reasoned just above, the label information for it as taught, would be understood to be stored at minimum using a storage/memory scheme as generally understood and widely practiced in the state of the art and would therefore correspond to a storage/memory address (i.e., “datum identifier” as recited));
comparing a pre-existing label ... with the new label (the label comparison and selection process, as noted above and as discussed in accordance with the invention’s first aspect as found on pages 2-3, where each label comparison is in terms of accuracy such that the label that is sufficiently accurate is used).
Bao does not teach that the machine learning model training text data as noted above is machine learning model training chatbot text data and that category for a label as noted above is a category of chatbot data, e.g., as further recited. Rather, the Examiner relies upon HE to teach what Bao otherwise lacks, see e.g., He’s model-driven virtual assistant staging framework to automatically process electronic communication using machine learning techniques (column 1 lines 20-27, column 3 lines 8-22), where the electronic communication may be a text message (column 3 lines 23-26), and the Examiner reasons that a virtual assistant is engaged via text messaging as is contemplated per these teachings is then equivalent to a “chatbot” as recited.
Like Bao, He relates to text-driven and an automation framework that benefits from a machine-learning model framework. Hence, they are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the classifier improvement aspects of Bao to a framework that is operative over an underlying data that is not just text-based as Bao teaches but chat-based as He teaches, with a reasonable expectation of success.
Bao does not teach that the label for the training datum is one of a plurality of pre-existing labels previously applied to the training datum, such that the retrieving step noted above is a retrieval of a plurality of label records for a plurality of pre-existing labels.
Relatedly, Bao also does not teach the label comparison is as follows, per further recitations:
based on determining that the pre-existing label ... differs from the new label (it would be obvious to not compare a label to itself, as that is a waste in computing/processing resources, and hence this condition to further process in accordance with the recited difference is obvious), generating, based on the label record database, (1) a first dataset labeled with the new label and (2) a second dataset labeled with the pre-existing label;
calculating a first plurality of similarity metrics between the training datum and each datum of the first dataset and a second plurality of similarity metrics between the training datum and each datum of the second dataset;
calculating a first average similarity metric based on an average of the first plurality of similarity metrics and a second average similarity metric based on an average of the second plurality of similarity metrics;
comparing the first average similarity metric with the second average similarity metric; and
based on determining that the second average similarity metric is higher than the first average similarity metric, generating a recommendation for a modified label based on the pre-existing label.
Rather, the Examiner relies upon Zhang and Hüllermeier to teach what Bao lacks:
Zhang teaches a comparable label improvement framework based on partial label learning. See, e.g., Abstract and Introduction sections as found on its first page (numbered page 1335). With this approach, a feature vector is associated with a set of candidate labels, and a true label is found therefrom. One specific way this is accomplished is through an average-based approach, such that each candidate label’s model output is obtained and averaged.
Zhang’s consideration of a set of candidate labels is akin to Applicants’ claimed limitations involving a plurality of pre-existing labels and the related feature of a plurality of label records for those. That is to say, for Zhang to provide for the consideration of many labels as taught, then it would necessarily retrieve those labels from some persistent storage sot that they could be compared, evaluated, and selected/discarded in accordance with Zhang’s approach.
Zhang’s average-based approach takes the modeling output for each candidate label and averages them together, to essentially select one candidate label over another. In teaching this, Zhang (Introduction section’s 3rd paragraph, in its last line) cites to Hüllermeier:
Hüllermeier’s section 4.1, on its page 4, teaches in more detail the use of Nearest Neighbor Classification (kNN), which the Examiner understands to be a similarity metric/measurement that is able to differentiate many graphed candidates based on their distance from a prompt/target, such that the nearest distance is understood to be the best classification result for the prompt/target.
Zhang in view of Hüllermeier can then be understood to teach the differentiation of different candidates in a set with respect to their respective distances (i.e., how similar they are, as measured) in relation to a target/prompt, with the explicitly contemplated use case being to find the closest labels to a true/expected label, i.e. the generated recommendation as recited in Applicants’ claim.
Hence, the Examiner reasons that Zhang and Hüllermeier as combined in this way can be used to implement and extend Bao’s label to label comparison to improve labeling in the training of a machine learning model, e.g. by (i) arriving at a comparison that permits the comparison against many such labels among a set of candidate labels and (ii) performing each comparison based on a similarity metric via Hüllermeier’s kNN approach discussed just above.
Like Bao, Zhang and Hüllermeier relate to label selection with the purpose of improving a machine-learning model such as a classifier, e.g. by improving its training via more accurate/improved labeling. Hence, they are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Zhang and Hüllermeier’s aspects, as reasoned above, to more broadly implement Bao’s label improvement feature, with a reasonable expectation of success, such that more than one label can be considered and the label differences can be measured and expressed via a technique known and used in the state of the art, thereby arriving at a determination of a best label for a given input.
Regarding claim 2, the claim is a briefer version of claim 1 discussed above, and is therefore rejected under the same/similar rationale provided above per claim 1.
Regarding claim 3, the claim includes limitations discussed above in relation to claim 1 and is therefore rejected under the same rationale.
Regarding claim 4, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 3, as discussed above. The aforementioned references further teach the additional limitations wherein calculating the first plurality of similarity metrics between the training datum and each datum of the first dataset comprises: generating, for use in a natural language processing model, a vector representation of the training datum and a plurality of vector representations corresponding to data in the first dataset; and in response to inputting the vector representation and the plurality of vector representations into the natural language processing model, generating the first plurality of similarity metrics (both Bao and He contemplate processing data inputs that are text, see the relevant mappings as provided above per claim 1, and hence the vector feature discussed in the 1st full paragraph on Bao’s page 11 is extensible to natural language processing as recited here and is amenable and applicable to the sort of similarity measurements discussed per Zhang and Hüllermeier specifically in relation to claim 1, such that improved labeling by incorporating Zhang and Hüllermeier’s distance and similarity considerations would reasonably make for a better model for Bao as modified).
Regarding claim 5, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 2, as discussed above. The aforementioned references further teach the additional limitations further comprising: based on comparing the first average similarity metric and the second average similarity metric, determining that the first average similarity metric is higher than the second average similarity metric; and based on determining that the first average similarity metric is higher than the second average similarity metric, generating the recommendation for the modified label to include the new label (Hüllermeier’s distance and similarity consideration by way of kNN, as discussed per claim 1, would reasonably provide a solution that is the nearest and hence the most similar to Bao’s target/prompt, with the understanding that nearest per kNN would correspond to highest similarity score as recited here).
Regarding claim 6, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 2, as discussed above. The aforementioned references further teach the additional limitations further comprising: based on comparing the first average similarity metric and the second average similarity metric, determining that the second average similarity metric is higher than the first average similarity metric; and based on determining that the second average similarity metric is higher than the first average similarity metric, generating the recommendation for the modified label to include the pre-existing label (Hüllermeier’s distance and similarity consideration by way of kNN, as discussed per claim 1, would reasonably provide a solution that is the nearest and hence the most similar to Bao’s target/prompt, with the understanding that nearest per kNN would correspond to highest similarity score as recited here).
Regarding claim 8, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 2, as discussed above. The aforementioned references further teach the additional limitations for generating a first feature input for the machine learning model based on the modified label and the training datum; and generating a first output for the machine learning model based on the modified label and the training datum (the labelling as improved in the manner of Bao as modified specifically with Zhang and Hüllermeier is understood to allow for an improved classifier model that can then correlate an input with a label output in a more accurate manner).
Regarding claim 9, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 8, as discussed above. The aforementioned references further teach the additional limitations further comprising: based on the first output, determining a model error indicator; and generating a modified label record in the label record database, wherein the modified label record comprises the model error indicator (Bao’s page 9, 2nd full paragraph discussing a recall rate reflective of when the classifier has failed, i.e., an indication of failure as registered by the system, and it would reason to formally record this instance, not only as to benchmark issues with the classifier and then improve upon them to essentially fix them per Zhang and Hüllermeier’s incorporated aspects, but also as to flag any such label that is then known to be incorrect).
Regarding claim 10, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 2, as discussed above. The aforementioned references further teach the additional limitations further comprising: detecting, in the label record database, an updated label record for a third dataset labeled with the pre-existing label; and calculating a third average similarity metric between the training datum and the third dataset; and based on comparing the third average similarity metric with the first average similarity metric and the second average similarity metric, determining an updated label for the training datum; and generating the recommendation for the modified label based on the updated label (the steps here are essentially additional instances/iterations of the same steps discussed above per claim 1, and hence the reasoning provided above for claim 1 is likewise applicable here).
Regarding claim 11, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 2, as discussed above. The aforementioned references further teach the additional limitations further comprising: detecting, in the label record database, an updated label record for a fourth dataset labeled with the new label; and calculating a fourth average similarity metric between the training datum and the fourth dataset; and based on comparing the fourth average similarity metric with the first average similarity metric and the second average similarity metric, determining an updated label for the training datum; and generating the recommendation for the modified label based on the updated label (the steps here are essentially additional instances/iterations of the same steps discussed above per claim 1, and hence the reasoning provided above for claim 1 is likewise applicable here).
Regarding claim 16, the claim includes limitations discussed above in relation to claim 1 and is therefore rejected under the same rationale.
Regarding claim 17, the claim includes limitations discussed above in relation to claim 3 and is therefore rejected under the same rationale.
Regarding claim 18, the claim includes limitations discussed above in relation to claim 4 and is therefore rejected under the same rationale.
Regarding claim 19, the claim includes limitations discussed above in relation to claim 8 and is therefore rejected under the same rationale.
Regarding claim 20, the claim includes limitations discussed above in relation to claim 9 and is therefore rejected under the same rationale.
6. Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Bao in view of He and further in view of Zhang and Hüllermeier and further yet in view of CN 113254796 A (“Lin”).
Regarding claim 7, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 2, as discussed above. The aforementioned references teach the additional limitations for based on the recommendation for the modified label, determining a modified label record for the training datum, wherein the modified label record comprises a modified label name, and the datum identifier; and generating the modified label record in the label record database (e.g., Bao as modified, if able to correct a label in the training data would then presumably be able to store it, e.g. using a memory/storage identifier in a memory/storage scheme known in the state of the art and typical to a computer as Bao contemplates using (as the Examiner has reasoned similarly in relation to claim 1), such that by storing it, then the model’s improvement is made persistent in the training data via the improved label).
That said, the references listed above appear silent as to the label record also comprising an updated modification timestamp as further recited. Rather, the Examiner relies upon LIN to teach what Bao etc. otherwise lack, see e.g., Lin’s page 2, in the 2nd paragraph under the section The Content of the Invention discussing the use of a timestamp for each label record.
Like Bao etc., Lin relates to a framework that encompasses label management. Hence, they are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Lin’s timestamp feature to implement Bao’s modified framework as having a more complete record of label management such that a basic information such as the timestamp can be used to provide more context to a label’s use, e.g. such as to audit it as Bao contemplates in some portions.
7. Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Bao in view of He and further in view of Zhang and Hüllermeier and further yet in view of U.S. Patent Application Publication No. 2013/0024407 (“Thompson”).
Regarding claim 12, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 2, as discussed above. The aforementioned references do not teach the further limitations for determining whether all data used to generate the modified label for the training datum has been received and in response to determining that not all data used to generate the modified label for the training datum has been received, assigning a first label type to the modified label, and rather the Examiner relies upon THOMPSON to teach what Bao etc. otherwise lack, see e.g., Thompson’s [0048] teaching a catch-all category that permits classification when no proper classification result is arrived at, and it reasons that this could be a classification that could be subject to correction for example in view of more data and more training per Bao’s general directive.
Like Bao, Thompson relate to label selection with the purpose of improving a machine-learning model such as a classifier, e.g. by improving its training via more accurate/improved labeling. Hence, they are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Thompson’s classification aspect as discussed here, as reasoned above, to more broadly implement Bao’s label improvement feature, with a reasonable expectation of success, such that if a classification result is not sufficiently confident then at least there is a dedicated category that could be subject to revision, further review, and ultimately correction under Bao’s directive.
8. Claims 15 is rejected under 35 U.S.C. 103 as being unpatentable over Bao in view of He and further in view of Zhang and Hüllermeier and further yet in view of Lin and Thompson.
Regarding claim 15, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 2, as discussed above. The aforementioned references teach generating, from the label record database, a first plurality of label records associated with the new label and a second plurality of label records associated with the pre-existing label (see the Examiner’s mapping for this as provided above in relation to claim 1),
but do not teach a timestamp or the like, and hence do not teach the further limitations for generating a first plurality of modification timestamps corresponding to the first plurality of label records and a second plurality of modification timestamps corresponding to the second plurality of label records and based on the first plurality of modification timestamps and the second plurality of modification timestamps, determining a first label record for the new label and a second label record for the pre-existing label and based on the first label record and the second label record, determining the first dataset labeled with the new label and the second dataset labeled with the pre-existing label.
Rather, the Examiner relies upon LIN and THOMPSON to teach what Bao etc. otherwise lack, see e.g.:
Lin teaches, in the 2nd paragraph under the section The Content of the Invention discussing the use of a timestamp for each label record.
Like Bao etc., Lin relates to a framework that encompasses label management. Hence, they are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Lin’s timestamp feature to implement Bao’s modified framework as having a more complete record of label management such that a basic information such as the timestamp can be used to provide more context to a label’s use, e.g. such as to audit it as Bao contemplates in some portions.
Thompson teaches, in [0017]-[0019], that relevant content are marked and collected and then used for training, selectively. One of ordinary skill could frame relevancy in terms of a time/temporal importance, e.g. by way of using Lin’s timestamps as incorporated into Bao’s modified framework.
Like Bao, Thompson relate to label selection with the purpose of improving a machine-learning model such as a classifier, e.g. by improving its training via more accurate/improved labeling. Hence, they are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Thompson’s classification aspect as discussed here, as reasoned above, to more broadly implement Bao’s label improvement feature, with a reasonable expectation of success, such that if a classification result is not sufficiently confident then at least there is a dedicated category that could be subject to revision, further review, and ultimately correction under Bao’s directive.
9. Claim 13-14 is rejected under 35 U.S.C. 103 as being unpatentable over Bao in view of He and further in view of Zhang and Hüllermeier and further yet in view of Thompson and Lin.
Regarding claim 13, Bao in view of He and further in view of Zhang and Hüllermeier teach the method of claim 2, as discussed above. The aforementioned references do not teach the additional limitations further comprising:
determining, from the plurality of label records, a record timestamp for a label record corresponding to the pre-existing label;
determining, from the label modification request, a request timestamp for the new label;
based on comparing the record timestamp with the request timestamp, determining a label update rate for the training datum; and
based on determining that the label update rate is above a threshold update rate, generating a warning for display on a user interface.
Rather, the Examiner relies upon LIN and THOMPSON to teach what Bao etc. otherwise lack, see e.g.:
Lin teaches, in the 2nd paragraph under the section The Content of the Invention discussing the use of a timestamp for each label record.
Like Bao etc., Lin relates to a framework that encompasses label management. Hence, they are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Lin’s timestamp feature to implement Bao’s modified framework as having a more complete record of label management such that a basic information such as the timestamp can be used to provide more context to a label’s use, e.g. such as to audit it as Bao contemplates in some portions.
Lin’s timestamping could be used to help define rates such as Bao’s recall rate as discussed on its page 9, which the Examiner would then equate with the recited label update rate.
Thompson teaches, in [0017]-[0019], that relevant content are marked and collected and then used for training, selectively. One of ordinary skill could frame relevancy in terms of a time/temporal importance, e.g. by way of using Lin’s timestamps as incorporated into Bao’s modified framework.
Like Bao, Thompson relate to label selection with the purpose of improving a machine-learning model such as a classifier, e.g. by improving its training via more accurate/improved labeling. Hence, they are similarly directed and therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Thompson’s classification aspect as discussed here, as reasoned above, to more broadly implement Bao’s label improvement feature, with a reasonable expectation of success, such that if a classification result is not sufficiently confident then at least there is a dedicated category that could be subject to revision, further review, and ultimately correction under Bao’s directive.
Regarding claim 14, over Bao in view of He and further in view of Zhang and Hüllermeier and further yet in view of Thompson and Lin teach the method of claim 13, as discussed above. The aforementioned references teach the additional limitation wherein determining, from the label modification request, the request timestamp for the new label comprises: receiving a temporal identifier for a point in time, wherein the temporal identifier comprises a standardized setting for recording times across the computer network; and recording the temporal identifier as the request timestamp (Lin’s page 2, in the 2nd paragraph under the section The Content of the Invention discussing the use of a timestamp for each label record).
Conclusion
10. The prior art made of record and not relied upon is considered pertinent to Applicants’ disclosure:
US 20220414137 A1
US 20220222924 A1
US 20200251188 A1
US 20160379159 A1
US 20220237368 A1
US 20210357303 A1
US 20230315998 A1
US 20200160777 A1
CN 112241626 A
WO 2022227400 A1
Non-Patent Literature “Candidate Label-aware Similarity Graph for Partial Label Data”
Non-Patent Literature “Progressive Identification of True Labels for Partial-Label Learning”
Non-Patent Literature “Learning Soft Labels via Meta Learning”
Non-Patent Literature “Solving the Partial Label Learning Problem: An Instance-based Approach”
Non-Patent Literature “Leveraging Latent Label Distributions for Partial Label Learning”
Non-Patent Literature “Learning from Partial Labels”
11. Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHOURJO DASGUPTA whose telephone number is (571)272-7207. The examiner can normally be reached M-F 8am-5pm CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571 272 4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHOURJO DASGUPTA/Primary Examiner, Art Unit 2144