DETAILED ACTION
This action is responsive to the claims filed on 03/26/2024. Claims 1-20 are pending for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Chinese Patent Application No. 202111061622, filed on 09/10/2021.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 02/05/2025 and 12/13/2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 4, 10-11, 13, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (CN 111476349 A), hereafter referred to as Xu.
Claim 1: Xu teaches:
A model test method, comprising: (Xu, abstract, “The embodiment of the present application discloses a model testing method and server.”, Xu expressly teaches a method for remotely testing machine-learning/deep-learning models and therefore teaches the claimed model test method.)
receiving first information from a first device, wherein the first information is usable to determine at least one test set; (Xu, page 4, paragraph 12, “The terminal sends a model test request to the server, where the request carries a model file of the model to be tested, a test data set, and parameters and thresholds of the model to be tested.”, The terminal corresponds to the claimed first device and the model-testing server corresponds to the device performing the claimed method. The server receives from the terminal first information containing the test data set and therefore information usable to determine the test set.)
testing an artificial intelligence (AI) model based on the at least one test set, to obtain a test result; and (Xu, page 5, paragraph 9, “The test model is actually the model to be tested. Then, the test data set of the model to be tested is input into the built test model, and each data of the test data set is tested one by one. For example, if it is a face image detection model test, it is to test and recognize each face image in the test data set one by one; if it is a speech recognition model test, it is to test and recognize each voice in the test data set one by one, and so on.”, Xu expressly inputs the received test data set into the model under test and tests the data using that model, thereby testing the AI model based on the test set to obtain a result.)
sending second information to the first device, wherein the second information indicates the test result of the AI model. (Xu, page 6, paragraph 7, “The server sends the above-mentioned model test result to the terminal according to the above-mentioned acquisition request.”, The model-testing server sends the resulting model-test information back to the same terminal that supplied the test information. Accordingly, Xu teaches sending second information indicating the AI-model test result to the first device.)
Claim 2: Xu teaches the limitations of claim 1, Xu further teaches:
The method according to claim 1, wherein the receiving the first information includes receiving indication information of the at least one test set; or the receiving the first information includes receiving a reference signal, and the method further comprises: determining the at least one test set based on the reference signal. (Xu, page 3, paragraph 3, “the test information includes a model file, a test data set and parameters of the model to be tested;”, Under Broadest Reasonable Interpretation, information expressly carrying the particular test data set necessarily indicates which test set is to be employed. Thus, Xu teaches the first expressly recited alternative.)
Claim 4: Xu teaches the limitations of claim 1, Xu further teaches:
The method according to claim 1, wherein the testing the AI model based on the at least one test set, to obtain the test result includes obtaining a first test result, and the first test result is usable to indicate an output that is of the AI model and that is obtained based on the at least one test set; or the first test result is usable to indicate a performance indicator of the AI model obtained by testing the AI model based on the at least one test set. (Xu, page 3, paragraph 13, “After comparing the test label of each test data with the preset label, the test result is obtained by calculation; wherein, the test result includes one or more of the correct rate, the recall rate and the data with wrong prediction.”, Accuracy/correct rate and recall rate are expressly calculated performance indicators of the tested model. Xu therefore teaches the claimed performance-indicator alternative.)
Claim 10: Xu teaches:
A model test method, comprising: sending first information to a second device, wherein the first information is usable to determine at least one test set; and (Xu, page 4, paragraph 12, “The terminal sends a model test request to the server, where the request carries a model file of the model to be tested, a test data set, and parameters and thresholds of the model to be tested.”, From the terminal-side perspective, Xu teaches sending to the testing server first information containing and therefore determining the test set.)
receiving second information from the second device, wherein the second information is usable to indicate a test result of an artificial intelligence (AI) model, and the test result corresponds to the at least one test set. (Xu, page 6, paragraph 7, “The server sends the above-mentioned model test result to the terminal according to the above-mentioned acquisition request.”, The terminal receives the model-test result generated by the server from testing the test data set previously supplied by that terminal.)
Claim 11: Xu teaches the limitations of claim 10, Xu further teaches:
The method according to claim 10, wherein the sending the first information includes sending indication information of the at least one test set; or the first information comprises a reference signal. (Xu, page 3, paragraph 18, “a receiving unit, configured to receive a model test request; wherein, the request carries test information of the model to be tested, and the test information includes a model file, a test data set and parameters of the model to be tested;”, Under BRI, sending the particular test set necessarily communicates information indicating which test set is to be used.)
Claim 13: Xu teaches the limitations of claim 10, Xu further teaches:
The method according to claim 10, wherein the receiving the second information usable to indicate the test result includes receiving the second information indicating a first test result, and the first test result is usable to indicate an output that is of the AI model and that is obtained based on the at least one test set; or the first test result is usable to indicate a performance indicator of the AI model obtained by testing the AI model based on the at least one test set. (Xu, page 3, paragraph 13, “After comparing the test label of each test data with the preset label, the test result is obtained by calculation; wherein, the test result includes one or more of the correct rate, the recall rate and the data with wrong prediction.”, The disclosed correct rate and recall rate constitute performance indicators obtained by testing the AI model with the test set.)
Claim 19: Xu teaches the limitations of claim 10, Xu further teaches:
An apparatus, comprising a memory storing instructions; a processor, connected to the memory, the processor configured to execute the instruction stored in the memory to cause the processor to perform the following: (Xu, page 6, paragraph 23, “The server 400 includes a processor 401, a memory 402, and a communication interface 403. The processing The device 401 , the memory 402 and the communication interface 403 are connected to each other through a bus 404 .”, Xu expressly teaches the claimed processor-and-memory apparatus architecture.)
receiving first information from a first device, wherein the first information is usable to determine at least one test set; (Xu, page 6, paragraph 12, “Receive a model test request; wherein, the request carries test information of the model to be tested, and the test information includes a model file, a test data set and parameters of the model to be tested;”, The processor-controlled server receives from the terminal test information containing the test set.)
testing an artificial intelligence (AI) model based on the at least one test set, to obtain a test result; and (Xu, page 5, paragraph 9, “The test model is actually the model to be tested. Then, the test data set of the model to be tested is input into the built test model, and each data of the test data set is tested one by one. For example, if it is a face image detection model test, it is to test and recognize each face image in the test data set one by one; if it is a speech recognition model test, it is to test and recognize each voice in the test data set one by one, and so on.”, The server processor causes the tested AI/deep-learning model to process the received test data set.)
sending second information to the first device, wherein the second information is usable to indicate the test result of the AI model. (Xu, page 6, paragraph 7, “The server sends the above-mentioned model test result to the terminal according to the above-mentioned acquisition request.”, The testing server returns to the originating terminal information indicating the model-test result.)
Claim 20: Xu teaches the limitations of claim 19, Xu further teaches:
The apparatus according to claim 19, wherein the first information comprises indication information of the at least one test set; or the first information comprises a reference signal, and determination of the at least one test set is based on the reference signal. (Xu, page 4, paragraph 12, “The terminal sends a model test request to the server, where the request carries a model file of the model to be tested, a test data set, and parameters and thresholds of the model to be tested.”, Under BRI, information expressly containing the test data set necessarily indicates the particular test set employed by the testing apparatus, which satisfies one of the alternatives.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 3, 5-7, 12, and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Motohashi et al., (US 20180181875 A1), hereafter referred to as Motohashi.
Claim 3: Xu teaches the limitations of claim 1, Motohashi, in the same field of machine learning further teaches, the following which Xu fails to teach:
The method according to claim 1, wherein the sending the second information includes sending second information that is usable to indicate at least one of the following: AI models participating in the test; a test result corresponding to each AI model; or a test set corresponding to each test result. (Motohashi, paragraph 110, “The model evaluation unit 130 evaluates each of the learning models. A specific evaluation method is described later. It is not necessary for the model evaluation unit 130 to evaluate all the learning models related with the nodes in the semantic hierarchical model. The model evaluation unit 130 performs evaluation for at least a prediction target learning model. [0111] The model selection unit 140 selects a learning model used when performing prediction for a prediction target from a plurality of learning models. The model selection unit 140 selects a higher-order learning model in a stage in which the amount of prediction target data is small, and selects a prediction target learning model, instead of the higher-order learning model, in a process in which the prediction target data is being accumulated. ”, Motohashi teaches separately evaluating models within a plurality of candidate learning models, thereby generating an evaluation result corresponding to the respective models. Incorporating Motohashi’s multiple-model evaluation information into the model-test result already returned by Xu would cause the returned second information to indicate a test result corresponding to each participating AI model.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the model evaluation and selection teachings of Motohashi into the model testing system of Xu. Xu already tests machine-learning models and returns the resulting performance information to a requesting terminal, while Motohashi teaches evaluating multiple learning models, determining whether model performance satisfies a predetermined criterion, and selecting a qualifying model for subsequent prediction. A person of ordinary skill would have been motivated to include Motohashi’s per-model evaluation and selection information in Xu’s test-result exchange so that the requester could use the returned testing information to identify which evaluated model satisfies a desired performance criterion and should be subsequently employed, thereby predictably converting model-test results into an operational model-selection decision.
Claim 5: Xu teaches the limitations of claim 1, Xu further teaches:
The method according to claim 4, further comprising:receiving first indication information from the first device, (Xu, page 6, paragraph 6, “The terminal sends a request for obtaining the model test result to the server according to the above-mentioned indication information.” Xu’s terminal corresponds to the claimed first device, and Xu’s server receives the request sent by the terminal. Therefore, the request received by the server from the terminal teaches receiving first indication information from the first device.)
Motohashi, in the same field of machine learning further teaches, the following which Xu fails to teach:
wherein the first indication information is usable to indicate that each of the AI models participating in the test meets or does not meet a performance goal, is usable to indicate an AI model that meets a performance goal and that is in the AI models participating in the test, is usable to indicate an AI model that does not meet a performance goal and that is in the AI models participating in the test, is usable to indicate an AI model to be subsequently usable by a second device, or is usable to indicate a second device to perform a corresponding operation in a non-AI manner. (Motohashi, paragraph 0111, “The model selection unit 140 selects a learning model used when performing prediction for a prediction target from a plurality of learning models… [0112] The model selection unit 140 outputs the selected learning model to the prediction system 300.”; Paragraph 0112, “The model selection unit 140 outputs the selected learning model to the prediction system 300.”, Paragraph 111 teaches that the model selection unit selects, from a plurality of learning models, a learning model to be used when performing prediction for a prediction target. Motohashi, paragraph [0112], further teaches that the model selection unit “outputs the selected learning model to the prediction system 300.” Thus, Motohashi identifies a selected AI model from among the evaluated learning models and provides that selected AI model to prediction system 300 for subsequent prediction. The selected-model information is therefore usable to indicate an AI model to be subsequently usable by a second device, namely prediction system 300. Because the listed functions of the first indication information are recited in the alternative by “or,” Motohashi’s teaching of the alternative relating to an AI model to be subsequently usable by a second device is sufficient to teach the limitation. Motohashi is not relied upon as teaching the separate alternative of indicating that the second device perform a corresponding operation in a non-AI manner.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Motohashi’s selected-model information into the terminal-to-server communication taught by Xu. Xu teaches receiving information from the terminal after completion of model testing, while Motohashi teaches selecting an evaluated learning model for subsequent use and providing the selected model to a prediction system. A person of ordinary skill would have been motivated to include an indication of the selected model in Xu’s terminal-to-server communication so that the testing server could identify which tested model had been selected for subsequent use by another device, thereby facilitating coordination and deployment of the selected model and producing the predictable result of communicating a model-selection decision through Xu’s existing communication arrangement.
Claim 6: Xu teaches the limitations of claim 1, Motohashi, in the same field of machine learning further teaches, the following which Xu fails to teach:
The method according to claim 1, wherein the testing the AI model based on the at least one test set, to obtain the test result includes obtaining a second test result, and the second test result is usable to indicate that each of the AI models participating in the test meets or does not meet a performance goal, is usable to indicate an AI model that meets a performance goal and that is in the AI models participating in the test, is usable to indicate an AI model that does not meet a performance goal and that is in the AI models participating in the test, is usable to indicate an AI model to be subsequently usable by a second device, or is usable to indicate a second device to perform a subsequent operation in a non-AI manner. (Motohashi, paragraphs [0136]-[0139], teaches evaluating a learning model using N-fold cross-validation, including using a portion of the applicable data as test data, comparing predicted values with actual values, and obtaining evaluation results. In particular, Motohashi, paragraph [0139], teaches that the model evaluation unit “outputs an error average value and an error distribution value.” The error-average value and error-distribution value obtained from testing the learning model constitute a second test result.
Motohashi, paragraph [0140], “the model evaluation unit 130 may evaluate whether both the error average value and the error distribution value, which are calculated using the N-fold cross-validation as the evaluation results of the prediction target learning model, satisfy their respective criterion determined in advance. When both the error average value and the error distribution value satisfy their respective criterion, the model evaluation unit 130 may select the prediction target learning model instead of the higher-order learning model.”, Motohashi further teaches evaluating whether the error-average value and error-distribution value “satisfy their respective criterion” and selecting the prediction-target learning model when the criteria are satisfied. Thus, Motohashi’s second test result is usable to determine that the evaluated prediction-target learning model satisfies predetermined performance criteria and therefore teaches a second test result usable to indicate an AI model that meets a performance goal and that is in the AI models participating in the test. Because the listed indications are recited in the alternative by “or,” Motohashi’s teaching of the alternative relating to an AI model meeting a performance goal is sufficient to teach the limitation. Motohashi is not relied upon as teaching the separate alternative of indicating that a second device perform a subsequent operation in a non-AI manner.)
The motivation for combining Xu with Motohashi is similar to as applied for claim 3 above.
Claim 7: Xu and Motohashi teaches the limitations of claim 6, Xu further teaches
The method according to claim 6, further comprising: receiving second indication information from the first device, (Xu, page 4, paragraph 12, “The terminal sends a model test request to the server, where the request carries a model file of the model to be tested, a test data set, and parameters and thresholds of the model to be tested.”, The terminal corresponds to the claimed first device and the model-testing server corresponds to the device performing the claimed method. The server receives from the terminal first information containing the test data set and therefore information usable to determine the test set.)
Motohashi, in the same field of machine learning further teaches, the following which Xu fails to teach:
wherein the second indication information is usable to indicate the performance goal for the AI model. (Motohashi, Paragraph 203, “the model selection unit selects the higher-order learning model in a stage in which the amount of the target data is small, and selects the target learning model, instead of the higher-order learning model, at a timing at which the evaluation of the target learning model satisfies a predetermined criterion in the process in which the target data is accumulated.”, Motohashi’s predetermined evaluation criterion constitutes a performance goal against which the evaluated model is judged. It would have been obvious to use Motohashi’s model-level evaluation criterion as the threshold information supplied with Xu’s model-test request.)
The motivation for combining Xu with Motohashi is similar to as applied for claim 3 above.
Claim 12 recites limitations substantially similar to claim 3, as such a similar analysis applies.
Claims 14-16 recites limitations substantially similar to claim 5-7, as such a similar analysis applies.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Wei et al., (CN 112306829 A), hereafter referred to as Wei.
Claim 8: Xu teaches the limitations of claim 1,
Wei, in the same field of machine learning further teaches, the following which Xu fails to teach:
The method according to claim 1, further comprising: sending request information to the first device, wherein the request information is usable to request the first information, or is usable to request to test the AI model. (Wei, page 3, paragraph 16, “sending a test data loading request, wherein the test data loading request carries the identity identification information so that the server matches the test data based on the identity identification information; and receiving the test data loading request response and loading the test data.”, Wei teaches a testing device affirmatively requesting test data from another device rather than waiting for the data to be pushed. Applying this known pull-based mechanism would case a testing device to request the first information needed for model testing.)
It would have been obvious to incorporate Wei’s tester-initiated test-data request mechanism into Xu because Xu’s testing server requires test information to perform model testing, while Wei teaches the known alternative of the testing device affirmatively requesting the appropriate test data from a supplying device. Such a modification would predictably permit the tester to obtain the required test information on demand rather than requiring the supplying device to push the information without a preceding request.
Claim 17 recites limitations substantially similar to claim 8, as such a similar analysis applies.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Wei, as applied for claims 8 and 17 above, and in further view of Wang et al., (US 20200050951 A1), hereafter referred to as Wang.
Claim 9: Xu and Wei teaches the limitations of claim 8, Wang, in the same field of machine learning further teaches, the following which Xu and Wei fails to teach:
The method according to claim 8, wherein the sending the request information includes sending indication information of an input format of the AI models participating in the test. (Wang, paragraph 50, “a model requester 306 (shown in FIGS. 3 and 4) generates a specification (such as input data format, number of output classes, etc.) of a machine learning model for which the model requester wants to accomplish a machine learning task. At 504, the model requester 306 performs one or more steps of preliminary training of the machine learning model, using its own local dataset. Then, at 506 the model requester 306 sends the specification to other edge nodes 304 (generally, any desired number of edge nodes such as 304 a, 304 b, 304 c . . . )”, Wang expressly teaches transmitting to remote devices a machine-learning-model specification identifying the model’s input data format. Incorporating that known information into a test request would enable the supplying device to identify test data compatible with the participating model’s input requirements.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further include Wang’s input-data-format specification in the test-data request into the systems of Xu and Wei because Wang expressly teaches sending a machine-learning model specification including its input data format to remote nodes, which then determine whether their available data matches that specification. Including the participating model’s input format in Xu and Wei’s test-data request would predictably enable the supplying device to identify and return test data compatible with the model being tested and avoid providing incompatible test information.
Claim 18 recites limitations substantially similar to claim 9, as such a similar analysis applies.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Mattson, P., Reddi, V. J., Cheng, C., Coleman, C., Diamos, G., Kanter, D., ... & Wu, C. J. (2020). MLPerf: An industry standard benchmark suite for machine learning performance. IEEE Micro, 40(2), 8-16.
US20200293930A1
US20210081614A1
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HYUNGJUN B YI whose telephone number is (703)756-4799. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached on (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.B.Y./Examiner, Art Unit 2124
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights