Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Specification
The specification filed on May 15, 2025 is accepted.
Drawings
The drawings filed on July 18, 2025 are accepted.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/31/2026 was filed after the mailing date of the application no. 19/209671. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al (hereinafter Li) (CN 113610230) (attached English translation is used for examination) in view of Wang (CN 117743340) (attached English translation is used for examination).
Regarding claim 1, 9 and 17 Li teaches (Li on [page 1 lines 1-10] teaches method for training prediction model);
A computing system including one or more processors and one or more storage devices, the one or more storage devices, individually or collectively, having computer executable instructions stored thereon, the computer executable instructions, when executed by the one or more processors, enabling the one or more processors to, individually or collectively, execute actions comprising: (Li on [page 8 text associated with Fig 8] teaches computer readable instructions stored in memory executed by processor to perform operation);
A non-transitory storage medium having computer executable instructions stored thereon, the computer executable instructions, when executed by one or more processors, enabling the one or more processors to, individually or collectively, execute actions comprising (Li on [page 8 text associated with Fig 8] teaches computer readable instructions stored in memory executed by processor to perform operation);
obtaining a test set, the test set including test data, one or more auxiliary test results corresponding to the test data, and label information corresponding to the one or more auxiliary test results (Li on [page 1 last para and page 2 1st para] teaches obtaining a test sample, wherein the test sample includes test data and a test tag corresponding to the test data, the test data includes data related to the transaction service, and the test tag includes a historical service processing amount of the target service in the target time period. Further teaches inputting the test data into each prediction model of the plurality of prediction models to obtain respective service processing volume test results of the plurality of prediction models i.e., one or more auxiliary test result from plurality of predication models and determining the target prediction model based on the service processing quantity test results of the plurality of prediction models and the test labels i.e., label information corresponding to one or more auxiliary results);
the test data including data of one or more different modalities (Li on [page 5 lines 5-10] teaches the test data may include at least one of weather data, date data, time data, geographic location data, or service type data. i.e., one or more different modalities);
and the one or more auxiliary test results each being a result outputted by an auxiliary assessment model after the test data are separately inputted into the auxiliary assessment model (Li on [page 1 last para and page 2 1st para] teaches inputting the test data into each prediction model of the plurality of prediction models to obtain respective service processing volume test results of the plurality of prediction models i.e., one or more auxiliary test result from plurality of predication models and determining the target prediction model based on the service processing quantity test results of the plurality of prediction models and the test labels. See on [page 6 step S302] teaches the test data is input into each prediction model of the plurality of prediction models to obtain respective service processing volume test results of the plurality of prediction models. See also [page 7 text associated with Fig 5] inputting the feature data 501 to the target prediction model A, inputting the feature data 501 into the target prediction model B and the input to the target prediction model C, and obtaining a prediction result A, a prediction result B, and a prediction result C, respectively, comparing the prediction result A, the prediction result B, and the prediction result C with the actual result of the service processing amount, and determining whether the prediction result A, the prediction result B, and the prediction result C are smaller than or equal to the preset determination threshold);
inputting the test data into a target (Li on [page 6 step S402] teaches the feature data is input into the target prediction model to obtain a prediction result. Further teaches obtaining a service processing quantity actual result of the to-be-predicted service in the to-be-predicted time period; and determining a final prediction model from the plurality of target prediction models based on the service processing quantity prediction result and the service processing quantity actual result of the to-be-predicted service in respective to-be-predicted time periods of the plurality of target prediction models, where the final prediction model includes one. See also [page 7 text associated with Fig 5] inputting the feature data 501 to the target prediction model A, inputting the feature data 501 into the target prediction model B and the input to the target prediction model C, and obtaining a prediction result A, a prediction result B, and a prediction result C, respectively, comparing the prediction result A, the prediction result B, and the prediction result C with the actual result of the service processing amount, and determining whether the prediction result A, the prediction result B, and the prediction result C are smaller than or equal to the preset determination threshold);
and determining, among the one or more auxiliary test results, a target auxiliary test result matching the test result, determining label information corresponding to the test result based on label information corresponding to the target auxiliary test result (Li on [page 6 step S303] teaches the respective service processing quantity test results of the plurality of prediction models may be compared with the test label, to determine whether the comparison threshold is less than or equal to the preset comparison threshold, and if the determination result is yes, the prediction model corresponding to the service processing quantity test result is determined as the target prediction model. See on [page 6 last para and page 7 1st para] teaches the service processing volume prediction result of the to-be-predicted service maybe compared with the actual result of the service processing amount in the to-be-predicted time period of each of the plurality of target prediction models, to determine whether the comparison is less than or equal to the preset determination threshold, and if the determination result is yes, the target prediction model corresponding to the service processing quantity prediction result is determined as the final prediction model. See also [page 7 text associated with Fig 5] inputting the feature data 501 to the target prediction model A, inputting the feature data 501 into the target prediction model B and the input to the target prediction model C, and obtaining a prediction result A, a prediction result B, and a prediction result C, respectively, comparing the prediction result A, the prediction result B, and the prediction result C with the actual result of the service processing amount, and determining whether the prediction result A, the prediction result B, and the prediction result C are smaller than or equal to the preset determination threshold).
Although Li teaches inputting the test data in machine learning based predication model, but fails to explicitly teach inputting the test data in large model and determining a risk assessment result of the target large model based on the label information corresponding to the test result, however Wang from analogous art teaches
inputting the test data into a target large model to obtain a test result corresponding to the test data (Wang on [page 2 2nd para] teaches inputting test data in large model);
and determining a risk assessment result of the target large model based on the label information corresponding to the test result (Wang on [page 8 para 6-8] teaches the task test result output by the large model can be analyzed according to the candidate reference information, when the task test result is matched with the candidate reference information, the task test result can be judged, and the task test result satisfies the task execution result reference condition. As an example, the content of the prompt filled in the blank item "note item" in the above example can be determined as the candidate reference information of the candidate prompt, that is, the words "insufficient", "slight", "poor" (i.e., label information) and so on will not appear in the "7" in the above example. In this example, it is possible to analyze the task test result output by the large model based on that "7" does not appear "insufficient", "slight", "poor" and so on, and identify whether there are "insufficient", "slight", "poor" and so on in the task test result, when it is identified that there is no word such as " insufficient ", " slightly ", " poor " and so on in the task test result, it can be judged that the task test result is matched with the candidate reference information in the candidate prompt so as to judge that the task test result satisfies the preset task execution result reference condition).
Thus, it would have been obvious to one ordinary skill in the art before the effective filing date to implement the teaching of Wang into the teaching of Li by inputting the test data into a target large model to obtain a test result and determining risk assessment of test result. One would be motivated to do so in order to determine target model to perform task based on test risk result (Wang [page 1-2]).
Regarding claims 2, 10 and 18 the combination of Li and Wang teaches all the limitations of claims 1, 9 and 17 respectively, Li further teaches separately inputting the test data into different auxiliary assessment models to obtain auxiliary test results corresponding to the test data and outputted by each auxiliary assessment model of the different auxiliary models, the different auxiliary assessment models including one or more of an auxiliary large model or a machine learning model (Li on [page 1 last para and page 2 1st para] teaches inputting the test data into each prediction model of the plurality of prediction models to obtain respective service processing volume test results of the plurality of prediction models i.e., one or more auxiliary test result from plurality of predication models and determining the target prediction model based on the service processing quantity test results of the plurality of prediction models and the test labels. See on [page 6 step S302] teaches the test data is input into each prediction model of the plurality of prediction models to obtain respective service processing volume test results of the plurality of prediction models. See also [page 7 text associated with Fig 5] inputting the feature data 501 to the target prediction model A, inputting the feature data 501 into the target prediction model B and the input to the target prediction model C, and obtaining a prediction result A, a prediction result B, and a prediction result C, respectively, comparing the prediction result A, the prediction result B, and the prediction result C with the actual result of the service processing amount, and determining whether the prediction result A, the prediction result B, and the prediction result C are smaller than or equal to the preset determination threshold);
Wang teaches determining label information corresponding to each auxiliary test result (Wang on [page 8 para 6-8] teaches the task test result output by the large model can be analyzed according to the candidate reference information, when the task test result is matched with the candidate reference information, the task test result can be judged, and the task test result satisfies the task execution result reference condition. As an example, the content of the prompt filled in the blank item "note item" in the above example can be determined as the candidate reference information of the candidate prompt, that is, the words "insufficient", "slight", "poor" (i.e., label information) and so on will not appear in the "7" in the above example. In this example, it is possible to analyze the task test result output by the large model based on that "7" does not appear "insufficient", "slight", "poor" and so on, and identify whether there are "insufficient", "slight", "poor" and so on in the task test result, when it is identified that there is no word such as " insufficient ", " slightly ", " poor " and so on in the task test result, it can be judged that the task test result is matched with the candidate reference information in the candidate prompt so as to judge that the task test result satisfies the preset task execution result reference condition).
Thus, it would have been obvious to one ordinary skill in the art before the effective filing date to implement the teaching of Wang into the teaching of Li by inputting the test data into a target large model to obtain a test result and determining risk assessment of test result. One would be motivated to do so in order to determine target model to perform task based on test risk result (Wang [page 1-2]).
Regarding claims 3, 11 and 19 the combination of Li and Wang teaches all the limitations of claims 2, 10 and 18 respectively, Wang further teaches wherein the determining the label information corresponding to each auxiliary test result includes: constructing prompt information based on each auxiliary test result, and inputting the prompt information into an annotation large model to perform annotation processing on each auxiliary test result to obtain the label information corresponding to each auxiliary test result, the label information including one or more of risky, risk-free, simple refusal, reasonable refusal, or irrelevant (Wang on [page 4 step S102] teaches constructing prompt and inputting prompt into large model. See on [page 8 para 6-8] teaches the task test result output by the large model can be analyzed according to the candidate reference information, when the task test result is matched with the candidate reference information, the task test result can be judged, and the task test result satisfies the task execution result reference condition. As an example, the content of the prompt filled in the blank item "note item" in the above example can be determined as the candidate reference information of the candidate prompt, that is, the words "insufficient", "slight", "poor" (i.e., label information) and so on will not appear in the "7" in the above example. In this example, it is possible to analyze the task test result output by the large model based on that "7" does not appear "insufficient", "slight", "poor" and so on, and identify whether there are "insufficient", "slight", "poor" and so on in the task test result, when it is identified that there is no word such as " insufficient ", " slightly ", " poor " and so on in the task test result, it can be judged that the task test result is matched with the candidate reference information in the candidate prompt so as to judge that the task test result satisfies the preset task execution result reference condition).
Thus, it would have been obvious to one ordinary skill in the art before the effective filing date to implement the teaching of Wang into the teaching of Li by inputting the test data into a target large model to obtain a test result and determining risk assessment of test result. One would be motivated to do so in order to determine target model to perform task based on test risk result (Wang [page 1-2]).
Regarding claims 4, 12 and 20 the combination of Li and Wang teaches all the limitations of claims 1, 9 and 17 respectively, Li further teaches wherein the obtaining the test set used includes: crawling test data from the Internet by using a network crawler; obtaining test data generated by a tester; or obtaining test data generated by using a test data generation device (Li on [page 4 lines 20-30] teaches The server 105 May be a server that provides various services, such as a background management server (merely an example) that provides support for a user through a website browsed by the terminal devices 101, 102, 103. The background management server may analyze the received user request and other data related to the transaction service, and feedback the processing result (for example, a webpage, information, or data obtained or generated according to the user request) to the terminal device);
and wherein the test data include one or more of text data, image data, audio data, or video data (Li on [page 5 lines 5-10] teaches the test data may include at least one of weather data, date data, time data, geographic location data, or service type data. i.e., one or more different modalities).
Regarding claims 5 and 13 the combination of Li and Wang teaches all the limitations of claims 1 and 9 respectively, Li further teaches wherein the determining the target auxiliary test result matching the test result includes: determining a similarity between the test result and each auxiliary test result in the one or more auxiliary test results; and determining an auxiliary test result with a similarity greater than a threshold as the target auxiliary test result (Li on [page 6 step S303] teaches the respective service processing quantity test results of the plurality of prediction models may be compared with the test label, to determine whether the comparison threshold is less than or equal to the preset comparison threshold, and if the determination result is yes, the prediction model corresponding to the service processing quantity test result is determined as the target prediction model. See on [page 6 last para and page 7 1st para] teaches the service processing volume prediction result of the to-be-predicted service maybe compared with the actual result of the service processing amount in the to-be-predicted time period of each of the plurality of target prediction models, to determine whether the comparison is less than or equal to the preset determination threshold, and if the determination result is yes, the target prediction model corresponding to the service processing quantity prediction result is determined as the final prediction model. See also [page 7 text associated with Fig 5] inputting the feature data 501 to the target prediction model A, inputting the feature data 501 into the target prediction model B and the input to the target prediction model C, and obtaining a prediction result A, a prediction result B, and a prediction result C, respectively, comparing the prediction result A, the prediction result B, and the prediction result C with the actual result of the service processing amount, and determining whether the prediction result A, the prediction result B, and the prediction result C are smaller than or equal to the preset determination threshold);
Regarding claim 6 and 14 the combination of Li and Wang teaches all the limitations of claims 5 and 13 respectively, Li further teaches wherein the determining the similarity between the test result and each auxiliary test result includes: determining, by using a similarity calculation rule, the similarity between the test result and each auxiliary test result; or separately inputting the test result and each auxiliary test result into a similarity model to obtain the similarity between the test result and each auxiliary test result (Li on [page 6 step S303] teaches the respective service processing quantity test results of the plurality of prediction models may be compared with the test label, to determine whether the comparison threshold is less than or equal to the preset comparison threshold, and if the determination result is yes, the prediction model corresponding to the service processing quantity test result is determined as the target prediction model. See on [page 6 last para and page 7 1st para] teaches the service processing volume prediction result of the to-be-predicted service maybe compared with the actual result of the service processing amount in the to-be-predicted time period of each of the plurality of target prediction models, to determine whether the comparison is less than or equal to the preset determination threshold, and if the determination result is yes, the target prediction model corresponding to the service processing quantity prediction result is determined as the final prediction model. See also [page 7 text associated with Fig 5] inputting the feature data 501 to the target prediction model A, inputting the feature data 501 into the target prediction model B and the input to the target prediction model C, and obtaining a prediction result A, a prediction result B, and a prediction result C, respectively, comparing the prediction result A, the prediction result B, and the prediction result C with the actual result of the service processing amount, and determining whether the prediction result A, the prediction result B, and the prediction result C are smaller than or equal to the preset determination threshold).
Regarding claims 7 and 15 the combination of Li and Wang teaches all the limitations of claims 1 and 9 respectively, Wang further teaches wherein the determining the target auxiliary test result matching the test result includes: obtaining a keyword included in the test result; and searching, among the one or more auxiliary test results, for an auxiliary test result including the keyword, and using the auxiliary test result including the keyword as the target auxiliary test result (Wang on [page 7 lines 1-20] teaches The key phrase cannot be a simple noun or adjective, and each phrase has at least three words, so it needs to experience advantages, characteristics, advantages and part of speech, and it does not need to appear words such as "insufficient", "slight", "poor" and so on, and link the key words with comma. Step 3: outputting the json string, not outputting the original text, not outputting each step, only outputting the final josn string, and the josn string format is (" text abstract ": string, " key word ": string. See on [page 10 last 3 para] teaches the abstract accuracy rate is the probability that the abstract in the output result of the large model is matched with the abstract reference condition in the candidate prompt, the key word accuracy is the probability that the key word part in the output result of the large model matches with the key word reference condition in the candidate prompt, the output result qualification rate is the comprehensive evaluation of the large model generation result, which not only satisfies the format is correct, but also satisfies the abstract qualification, the probability that the whole result obtained by the key word is qualified is satisfied).
Thus, it would have been obvious to one ordinary skill in the art before the effective filing date to implement the teaching of Wang into the teaching of Li by inputting the test data into a target large model to obtain a test result and determining risk assessment of test result. One would be motivated to do so in order to determine target model to perform task based on test risk result (Wang [page 1-2]).
Regarding claim 8 and 16 the combination of Li and Wang teaches all the limitations of claims 1 and 9 respectively, Li further teaches further comprising: separately inputting, in response to detecting that the test data in the test set are updated, updated test data in the test set into the different auxiliary assessment models to obtain an updated auxiliary test result corresponding to the updated test data, and determining updated auxiliary test result, and performing risk assessment on a large model based on the test set as updated data (Li on [page 6 step S402 and page 7 lines 1-10] teaches the feature data is input into the target prediction model to obtain a prediction result. Further teaches obtaining a service processing quantity actual result of the to-be-predicted service in the to-be-predicted time period; and determining a final prediction model from the plurality of target prediction models based on the service processing quantity prediction result and the service processing quantity actual result of the to-be-predicted service in respective to-be-predicted time periods of the plurality of target prediction models, where the final prediction model includes one).
Wang further teaches determining updated label information corresponding to each updated auxiliary test result (Wang on [page 8 4th last para] teaches the prompt information corresponding to the abnormal information is obtained, and the candidate reference information is updated according to the prompt information to modify the candidate prompt to obtain the modified candidate prompt).
Thus, it would have been obvious to one ordinary skill in the art before the effective filing date to implement the teaching of Wang into the teaching of Li by inputting the test data into a target large model to obtain a test result and determining risk assessment of test result. One would be motivated to do so in order to determine target model to perform task based on test risk result (Wang [page 1-2]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ji et al (CN 117609015) is directed towards a unit test method, a unit test large model adjusting method and device, relating to the technical field of computer, especially relates to the technical field of software test, unit test, large model, artificial intelligence and so on.
Yang et al (CN 116503157) is directed towards a risk identification method, comprising the following steps: obtaining a second data set, wherein the second data set comprises at least one second sample data; performing automatic characteristic engineering to the second sample data to obtain the target sample data; performing incremental learning on the reference model through the target sample data to obtain an updated model; performing performance test to the reference model and the updating model to obtain the reference test result and the updating test result; determining a target model according to the reference test result and the updated test result.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOEEN KHAN whose telephone number is (571)272-3522. The examiner can normally be reached 7AM-5PM EST M-TH Alternate Fridays.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Shewaye Gelagay can be reached at (571)272-4219. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOEEN KHAN/Primary Examiner, Art Unit 2436
