CTNF 18/425,193 CTNF 96523 Detailed Action Claims 1-6 are currently pending. Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Information Disclosure Statement The information disclosure statements (IDS) submitted on 1/29/2024 and 10/9/2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claims 1, 2 and 5-6 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 1 recites the limitation “… based on a first feature amount that is obtained from first information using the parameter of the first machine learning model” in lines 4-5 and “a correct label indicating first information , the second training data …” in line 9. It is unclear whether the first information in lines 4-5 and the first information in line 9 denotes the same data. Additionally, it is unclear what constitutes “a correct label indicating second information”, because the ‘second information’ is not clearly defined in the claim. Is the ‘second information’ mean the ‘second data’ or ‘third data’? For purpose of the examination, the examiner interprets the limitation to mean: “ first information ” in lines 4-5 and “ first information ” in line 9 denotes the same first information. Claim 5 is a method claim which recites the same feature as the apparatus claim 1, and is rejected for at least the same reasons. Claim 6 is an apparatus claim which recites the same feature as the apparatus claim 1, and is rejected for at least the same reasons. 07-34-05 AIA Claim 2 recites the limitation “ estimating a machine learning of the first data ” in line 3, and “updating the parameter of the first classification model " in line 5 . There is insufficient antecedent basis for this limitation in the claim. Claim 3 recites the limitation “the process further comprises: obtaining an accuracy indicator, based on the prediction accuracy when the third machine learning model classifies the second feature amount and the third feature amount into a first class corresponding to the first information and a second class corresponding to the second information, respectively;” It is unclear what constitutes “when the third machine learning model classifies the second feature amount and the third feature amount into a first class corresponding to the first information and a second class corresponding to the second information” because claim 1 already tells that the second feature amount has a correct label indicating first information and the third feature amount has a correct label indicating second information, and the third machine learning model is already been trained based on those data and labels. Does the limitation mean that the third machine learning model is re-evaluated based on the same training data and the same label? For purpose of the examination, the examiner interprets the limitation to mean: obtaining an accuracy indicator, based on the prediction accuracy when the third machine learning model correctly classifies two different datasets related to the training dataset of the third machine learning model to specific classes, respectively; Claim 4 recites the limitation “the accuracy indicator is a receiver operating characteristic (ROC)/area under the curve (AUC) score” in lines 3-4. It is unclear what constitutes ‘/’ as the slash can mean either "or" or "and" depending on context. Does ‘/’ mean ‘or’ or ‘and’? For purpose of the examination, the examiner interprets the limitation to mean: ROC or AUC . Claim Rejections - 35 USC § 102 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-15 AIA Claim s 1, and 5-6 are rejected under 35 U.S.C. 102( a)(1 ) as being anticipated by Crosby et al. (US 11100373, hereinafter ‘Cros by ’) . Regarding claim 1, Crosby teaches: A non-transitory computer-readable storage medium storing an evaluation program for causing a computer to execute processing comprising: ( [Crosby, claim 5] a local computer having AI, a remote server coupled to the local computer, memory for storing instructions. [col 9, line 50 - col 10, line 10] The predictions generated by the student model are validated by validation module 114a. Evaluating the student model (i.e., the third model) is equivalent to evaluating the trained teacher model (i.e., the second model) because it inherits the learned parameters from the teacher model. [col 9, lines 23-26] Additionally, the student model parameter can be used by the teacher model if the student model meets the performance criteria ) generating a second machine learning model, by updating a parameter of a first machine learning model, based on a first feature amount that is obtained from first information using the parameter of the first machine learning model; ( [Crosby, col 9, lines 9-26] discloses training a teacher model using the learned features (i.e., the first feature amount obtained from first info) from the unsupervised feature extraction module 107a and the curated training subset. Teacher model 400 with its trained learned parameters (i.e., the second machine learning model) is used to generate pseudo labels 401 from the unlabeled data. The teacher model before the training is the first machine learning model ) generating a third machine learning model, based on a first training data and a second training data, the first training data including: a second feature amount that is obtained from a second data based on a parameter of the second machine learning model; and a correct label indicating first information, the second training data including: a third feature amount that is obtained from a third data based on the parameter of the second machine learning model; and a correct label indicating second information; and ( [Crosby, col 9, line 50 - col 10, line 10] A noisy student predictive model 405 (i.e., the third machine learning model) is created based on the weakly supervised labeled data points generated for the unlabeled dataset (i.e., the third feature amount) combined with labeled data points from the curated dataset (i.e., the second feature amount). The ‘labeled data points from the curated dataset’ is the second feature with correct label indicating ‘first info’ because it is also used to train the teacher model and it is hand-labeled as shown in [col 9, lines 9-15] and [col 8, lines 13-14] ) evaluating the second machine learning model, based on the prediction accuracy of the generated third machine learning model. ( [Crosby, col 9, line 50 - col 10, line 10] The predictions generated by the student model are validated by validation module 114a. Evaluating the student model (i.e., the third model) is equivalent to evaluating the trained teacher model (i.e., the second model) because it inherits the learned parameters from the teacher model. [col 9, lines 23-26] Additionally, the student model parameter can be used by the teacher model if the student model meets the performance criteria ) Regarding claim 5, Crosby teaches: An evaluation method implemented by a computer, the evaluation method comprising: ( [Crosby, claim 5] a local computer having AI, a remote server coupled to the local computer, memory for storing instructions ) Claim 5 is a method claim which recites the same feature as the apparatus claim 1, and is rejected for at least the same reasons. Regarding claim 6, Crosby teaches: An accuracy evaluation device comprising: a memory; and a processor coupled to the memory, the processor being configured to perform processing including: ( [Crosby, claim 5] a local computer having AI, a remote server coupled to the local computer, memory for storing instructions ) Claim 6 is an apparatus claim which recites the same feature as the apparatus claim 1, and is rejected for at least the same reasons . Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Crosby in view of Salti et al. ( “Adaptive Appearance Modeling for Video Tracking: Survey and Evaluation”, 2012 , hereinafter ‘Salti’) . Regarding claim 2, Crosby teaches: The non-transitory computer-readable recording medium according to claim 1, wherein the updating includes ( [Crosby, claim 5] a local computer having AI, a remote server coupled to the local computer, memory for storing instructions. [col 9, line 50 - col 10, line 10] The predictions generated by the student model are validated by validation module 114a. Evaluating the student model (i.e., the third model) is equivalent to evaluating the trained teacher model (i.e., the second model) because it inherits the learned parameters from the teacher model. [col 9, lines 23-26] Additionally, the student model parameter can be used by the teacher model if the student model meets the performance criteria ) updating the parameter of the first classification model ( [Crosby, col 9, lines 9-26] discloses training a teacher model using the learned features (i.e., the first feature amount obtained from first info) from the unsupervised feature extraction module 107a and the curated training subset. Teacher model 400 with its trained learned parameters (i.e., the second machine learning model) is used to generate pseudo labels 401 from the unlabeled data. The teacher model before the training is the first machine learning model ) However, Crosby does not specifically disclose: estimating a machine learning of the first data by clustering based on the first feature amount, and updating the parameter of the first classification model based on a third training data having the classification as a correct label of the first data. Salti teaches: estimating a machine learning of the first data by clustering based on the first feature amount, and ( [Salti, page 4338, left col, D. Model Estimation and Update, lines 1-18] and [Salti, page 4338, right col, lines 21-30] disclose estimating the appearance model in each frame by keeping only the subspace centroids in the feature space (i.e., the first feature amount) by clustering the model and retain only the cluster centers ) updating the parameter of the first classification model based on a third training data having the classification as a correct label of the first data. ( [Salti, page 4337, right col, 2 nd para, lines 1-16] discloses updating the appearance model (i.e., parameter of the first classification model) based on a modified set (i.e., third training data) generated by processing the features (i.e., first data) and the labels (i.e., correct label) ) Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Crosby and Salti to use the method of estimating a machine learning model of the first data by clustering of Salti to implement the machine learning concept drift prevention method of the present invention. The suggestion and/or motivation for doing so is to improve the efficiency of the machine learning model while avoiding concept drift caused by overfitting [Salti, page 4346, left col, 2 nd para and Table IV] . 07-21-aia AIA Claim s 3-4 are rejected under 35 U.S.C. 103 as being unpatentable over Crosby in view of Chu et al. ( US 20090106178 , hereinafter ‘Chu’) . Regarding claim 3, Crosby teaches: The non-transitory computer-readable recording medium according to claim 1, wherein the third machine learning model is a binary classifier, and ( [Crosby, claim 5] a local computer having AI, a remote server coupled to the local computer, memory for storing instructions. [col 9, line 50 - col 10, line 10] A noisy student predictive model 405 (i.e., the third machine learning model) is created based on the labeled data points generated for the unlabeled dataset (i.e., the third feature amount) combined with labeled data points from the curated dataset (i.e., the second feature amount). [col 5, lines 16-24] and [Claim 1] indicate that the student model and the teacher model are classifiers ) the process further comprises: obtaining an accuracy indicator, based on the prediction accuracy when the third machine learning model classifies the second feature amount and the third feature amount into a first class corresponding to the first information and a second class corresponding to the second information, respectively; and ( [col 9, line 50 - col 10, line 40] The predictions generated by the student model are validated by validation module 114a. The student model is run on validation test data, which is a subset of the curated dataset aside for validation purposes (i.e., second feature amount), as determined by data splitter 204. The student model generates predictions and validation metrics (whether the student model classifies the input data correctly) and may be trained on additional datasets 200 (third feature amount) to improve the classification accuracy (by checking whether the model classifies the data correctly to specific classes). [col 5, lines 16-24] and [Claim 1] indicate that the student model and the teacher model are classifiers ) evaluating … the second machine learning model ( [col 9, line 50 - col 10, line 10] The predictions generated by the student model are validated by validation module 114a. Evaluating the student model (i.e., the third model) is equivalent to evaluating the trained teacher model (i.e., the second model) because it inherits the learned parameters from the teacher model. [col 9, lines 23-26] Additionally, the student model parameter can be used by the teacher model if the student model meets the performance criteria ) However, Crosby does not specifically disclose: wherein the third machine learning model is a binary classifier, evaluating, based on a relationship between the accuracy indicator and threshold, whether the estimation accuracy of the second machine learning model is degraded. Chu teaches: wherein the third machine learning model is a binary classifier, ( [Chu, 0027] discloses measuring whether the performance of a classification model is decaying by comparing the accuracy to a threshold. Decay metric such as giniDecay may be used. [0061] giniDecay measures the degradation of a binary classification model based on ROC and AUC ) evaluating, based on a relationship between the accuracy indicator and threshold, whether the estimation accuracy of the second machine learning model is degraded . ( [Chu, 0027] discloses measuring whether the performance of a classification model is decaying by comparing the accuracy to a threshold. Decay metric such as giniDecay may be used. [0061] giniDecay measures the degradation of a binary classification model based on ROC and AUC ) Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Crosby and Chu to use the method of evaluating whether the performance of a classifier is degrading based on a threshold of Chu to implement the machine learning concept drift prevention method of the present invention. The suggestion and/or motivation for doing so is to improve the performance of the machine learning model by updating it before it requires large volume of updates [Chu, 0030, 0074 and 0151]. Regarding claim 4, Crosby teaches: The non-transitory computer-readable storage medium according to claim 3, wherein ( [Crosby, claim 5] a local computer having AI, a remote server coupled to the local computer, memory for storing instructions ) the accuracy indicator is … respect to the prediction accuracy of the third machine learning model, and ( [col 10, lines 7-40] The learned student model parameters (i.e., the third machine learning model) are validated by validation module 114a. The validation module runs validation test data using the student model, and then the student model generated validation metrics 500 which are checked against pass/fail criteria ) the evaluating includes evaluating … the second machine learning model ( [col 9, line 50 - col 10, line 10] The predictions generated by the student model are validated by validation module 114a. Evaluating the student model (i.e., the third model) is equivalent to evaluating the trained teacher model (i.e., the second model) because it inherits the learned parameters from the teacher model. [col 9, lines 23-26] Additionally, the student model parameter can be used by the teacher model if the student model meets the performance criteria ) However, Crosby does not specifically disclose: the accuracy indicator is a receiver operating characteristic (ROC)/area under the curve (AUC) score with respect to the prediction accuracy of the third machine learning model, and the evaluating includes evaluating that the estimation accuracy of the second machine learning model is degraded . Chu teaches: the accuracy indicator is a receiver operating characteristic (ROC)/area under the curve (AUC) score with respect to the prediction accuracy of the third machine learning model, and ( [Chu, 0027] discloses measuring whether the performance of a classification model is decaying by comparing the accuracy to a threshold. Decay metric such as giniDecay may be used. [0061] giniDecay measures the degradation of a binary classification model based on ROC and AUC ) the evaluating includes evaluating that the estimation accuracy of the second machine learning model is degraded . ( [Chu, 0027] discloses measuring whether the performance of a classification model is decaying by comparing the accuracy to a threshold. Decay metric such as giniDecay may be used. [0061] giniDecay measures the degradation of a binary classification model based on ROC and AUC ) Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUN KWON whose telephone number is (571)272-2072. The examiner can normally be reached Monday – Friday 7:30AM – 4:30PM ET . Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Kawsar can be reached at (571)270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /JUN KWON/Examiner, Art Unit 2127 /ABDULLAH AL KAWSAR/Supervisory Patent Examiner, Art Unit 2127 Application/Control Number: 18/425,193 Page 2 Art Unit: 2127 Application/Control Number: 18/425,193 Page 3 Art Unit: 2127 Application/Control Number: 18/425,193 Page 4 Art Unit: 2127 Application/Control Number: 18/425,193 Page 5 Art Unit: 2127 Application/Control Number: 18/425,193 Page 6 Art Unit: 2127 Application/Control Number: 18/425,193 Page 7 Art Unit: 2127 Application/Control Number: 18/425,193 Page 8 Art Unit: 2127 Application/Control Number: 18/425,193 Page 9 Art Unit: 2127 Application/Control Number: 18/425,193 Page 10 Art Unit: 2127 Application/Control Number: 18/425,193 Page 11 Art Unit: 2127 Application/Control Number: 18/425,193 Page 12 Art Unit: 2127 Application/Control Number: 18/425,193 Page 13 Art Unit: 2127
Prosecution Insights