DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15 are rejected under 35 U.S.C. 101
because the claimed invention is directed to an abstract idea without significantly
more.
When considering subject matter eligibility under 35 U.S.C. 101, it must be
determined whether the claim is directed to one of the four statutory categories of
invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the
claim does fall within one of the statutory categories, the second step in the analysis is
to determine whether the claim is directed to a judicial exception (Step 2A). The Step 2A
analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined
whether or not the claims recite a judicial exception (e.g., mathematical concepts,
mental processes, certain methods of organizing human activity). If it is determined in
Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the
second prong (Step 2A, Prong 2), where it is determined whether or not the claims
integrate the judicial exception into a practical application. If it is determined at step 2A,
Prong 2 that the claims do not integrate the judicial exception into a practical
application, the analysis proceeds to determining whether the claim is a patent-eligible
application of the exception (Step 2B). If an abstract idea is present in the claim, any
element or combination of elements in the claim must be sufficient to ensure that the
claim integrates the judicial exception into a practical application, or else amounts to
significantly more than the abstract idea itself. Applicant is advised to consult the 2019
PEG for more details of the analysis.
Step 1
According to the first part of the analysis, in the instant case, claims 1-7, 8-9, 11-15, 10 are directed to a device, a method and a medium of ML model evaluation. Thus, each of the claims falls within one of the four statutory categories (i.e. process, machine, manufacture, or composition of matter). Step 2A,
Step 2A, Prong 1
Following the determination of whether or not the claims fall within one of the four
categories (Step 1), it must be determined if the claims recite a judicial exception (e.g.
mathematical concepts, mental processes, certain methods of organizing human
activity) (Step 2A, Prong 1). In this case, the claims are determined to recite a judicial
exception as explained below.
Regarding Claims 1, 8 and 10 these claims recite
generate a plurality of second machine learning models that are different from a first machine learning model subject to performance evaluation; and evaluate the first machine learning model on a basis of prediction labels, the prediction labels being output by inputting same data to the first machine learning model and to each of the second machine learning models.
The claims recite a mental process. As set forth in MPEP 2106.04(a)(2)(III)(C), “Claims can recite a mental process even if they are claimed as being performed on a computer”. These are recited at a high level such that they could be performed mentally, and they are also disclosed as a human user performing these functions, simply using a computer as a tool-see spec, [0011]-[0017], etc. Fig. 1. Thus, the claim recites abstract ideas.
Step 2A, Prong 2
Following the determination that the claims recite a judicial exception, it must be
determined if the claims recite additional elements that integrate the exception into a
practical application of the exception (Step 2A, Prong 2). In this case, after considering
all claim elements individually and as an ordered combination, it is determined that the
claims do not include additional elements that integrate the exception into a practical
application of the exception as explained below.
In Prong Two, a claim is evaluated as a whole to determine whether the recited judicial exception is integrated into a practical application of that exception. A claim is not “directed to” a judicial exception, and thus is patent eligible, if the claim as a whole integrates the recited judicial exception into a practical application of that exception. A claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. MPEP 2106.04(d). The claims recite an abstract idea and further the claims as a whole does not integrate the recited judicial exception into a practical application of the exception. A claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. MPEP 2106.04(d).
Regarding Claims 1, 8 and 10 these claims
This limitation recites using one or more neural networks as a tool to perform an
abstract idea, which is not indicative of integration into a practical application. MPEP 2106.05(f).)
This limitation is understood to be generic computer equipment and mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.0S(f))
Step 2B
Based on the determination in Step 2A of the analysis that the claims are
directed to a judicial exception, it must be determined if the claims contain any element
or combination of elements sufficient to ensure that the claim amounts to significantly
more than the judicial exception (Step 2B). In this case, after considering all claim
elements individually and as an ordered combination, it is determined that the claims do
not include additional elements that are sufficient to amount to significantly more than
the judicial exception for the same reasons given above in the Step 2A, Prong 2
analysis. Furthermore, each additional element identified above as being insignificant
extra-solution activity is also well-known, routine, conventional as described below.
Claims 1, 8 and 10: The claims do not include additional elements, alone or in combination, that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements amount to no more than generic computing components and field of use/technological environment which do not amount to significantly more than the abstract idea. The underlying concept merely receives information, analyzes it, and store the results of the analysis – this concept is not meaningfully different than concepts found by the courts to be abstract (see Electric Power Group, collecting information, analyzing it, and displaying certain results of the collection and analysis; see Cybersource, obtaining and comparing intangible data; see Digitech, organizing information through mathematical correlations; see Grams, diagnosing an abnormal condition by performing clinical tests and thinking about the results; see Cyberfone, using categories to organize store and transmit information; see Smartgene, comparing new and stored information and using rules to identify options). The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as a combination do not amount to significantly more than the abstract idea. For example, claim 1 recites the additional elements of “generating...”, “evaluating…” These elements are recited at a high level of generality and are well-understood, routine, and conventional activities in the computer art. Generic computers performing generic computer functions, without an inventive concept, do not amount to significantly more than the abstract idea. Looking at the elements as a combination does not add anything more than the elements analyzed individually. Therefore, these claims do not amount to significantly more than the abstract idea itself.
Step 2A/2B Prong 2 Dependent Claims
Regarding to claim 2, 9
Claim 2, 9 merely recite other additional elements that selecting the ML models which performing generic functions that when looking at the elements as a combination does not add anything more than the elements analyzed individually. Therefore, these claims also do not amount to significantly more than the abstract idea itself. These claims are not patent eligible.
Regarding to claim 3, 11
Claim 3, 11 merely recite other additional elements that selecting the ML model based on dissimilarity which performing generic functions that when looking at the elements as a combination does not add anything more than the elements analyzed individually. Therefore, these claims also do not amount to significantly more than the abstract idea itself. These claims are not patent eligible.
Regarding to claim 4, 12
Claim 4, 12 merely recite other additional elements that extracting and selecting the ML model based on dissimilarity which performing generic functions that when looking at the elements as a combination does not add anything more than the elements analyzed individually. Therefore, these claims also do not amount to significantly more than the abstract idea itself. These claims are not patent eligible.
Regarding to claim 5, 13
Claim 5, 13 merely recite other additional elements that selecting the ML model based on dissimilarity which performing generic functions that when looking at the elements as a combination does not add anything more than the elements analyzed individually. Therefore, these claims also do not amount to significantly more than the abstract idea itself. These claims are not patent eligible.
Regarding to claim 6, 14
Claim 6, 14 merely recite other additional elements that selecting the ML model based on minimum spanning tree which performing generic functions that when looking at the elements as a combination does not add anything more than the elements analyzed individually. Therefore, these claims also do not amount to significantly more than the abstract idea itself. These claims are not patent eligible.
Regarding to claim 7, 15
Claim 7, 15 merely recite other additional elements that evaluating the ML model based on matching degree between the labels which performing generic functions that when looking at the elements as a combination does not add anything more than the elements analyzed individually. Therefore, these claims also do not amount to significantly more than the abstract idea itself. These claims are not patent eligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 7-10, 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by NUSHI et al. (Nushi) US 20200349395.
In regard to Claim 1, Nushi disclose A model evaluation device comprising: ([0012] model evaluation system)
at least one memory configured to store instructions; and at least one processor configured to execute instructions to: ([0096]-[0098] memory, processor and instruction)
generate a plurality of second machine learning models that are different from a first machine learning model subject to performance evaluation; (Fig. 4B, [0025][0051] [0062]-[0070] employ second ML models (408, 410) different from first ML model (404) with performance evaluation) and
evaluate the first machine learning model on a basis of prediction labels, the prediction labels being output by inputting same data to the first machine learning model and to each of the second machine learning models. (Fig. 4B, [0017][0022]-[0024] [0032]-[0036] [0061]-[0070] evaluate the first ML model based on the output label by inputting the same test instance to the first ML and the second ML models)
In regard to Claim 2, Nushi disclose The model evaluation device according to claim 1,
Nushi disclose wherein the at least one processor is configured to execute the instructions to:
from the generated second machine learning models, select a predetermined number of the second machine learning models on a basis of dissimilarity, based on a preset reference, of the prediction labels output from the second machine learning models between the second machine learning models; (Fig. 4B, [0022]-[0024] [0032]-[0036][0048]-[0052] [0057]-[0070][0078] based on the similarity measure between different features (Euclidean distance) means dissimilarity of the output labels with feature clusters and identify a predetermined number of known features, for example, from the ML models to select the model based on ranking with the max. confidence value respective to outputs of individual models with the feature clusters generated) and
evaluate the first machine learning model on a basis of the prediction labels, the prediction labels being output by inputting the same data to the first machine learning model and to each of the selected second machine learning models. (Fig. 4B, [0017][0022]-[0024] [0032]-[0036] [0061]-[0070] evaluate the first ML model based on the output label by inputting the same test instance to the first ML and the second ML models)
In regard to Claim 7, Nushi disclose The model evaluation device according to claim 1,
Nushi disclose wherein the at least one processor is configured to execute the instructions to evaluate the first machine learning model on a basis of a matching degree between a correct label that is the prediction label output from each of the second machine learning models and the prediction label output from the first machine learning model. ([0022]-[0024] [0032]-[0035] [0044] [0047]-[0052] [0061]-[0070] [0075]-[0083] evaluate the first model based on the matching or not with strength of correlation between the output label from the second models and identified as the correct output by comparing the output label with the ground truth data and the output label from the first model)
In regard to claims 8-9, 15, claims 8-9, 15 are method claims corresponding to the device claims 1-2, 7 above and, therefore, are rejected for the same reasons set forth in the rejections of claims 1-2, 7.
In regard to claim 10, claim 10 is a medium claim corresponding to the device claim 1 above and, therefore, is rejected for the same reasons set forth in the rejections of claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3-6, 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over NUSHI et al. (Nushi) US 20200349395 in view of Hu et al. (Hu) US 11100406.
In regard to Claim 3, Nushi disclose The model evaluation device according to claim 2,
But Nushi fail to explicitly disclose “wherein the at least one processor is configured to execute the instructions to select the second machine learning models in such a manner that the dissimilarity between the predetermined number of the second machine learning models becomes higher.”
Hu disclose wherein the at least one processor is configured to execute the instructions to select the second machine learning models in such a manner that the dissimilarity between the predetermined number of the second machine learning models becomes higher. (col. 1, line 34-col. 2, line 32, col. 4, line 23-40, col. 5, line 3-42, col. 7, line 15-27, col. 13, line 28-col. 14, line 16, claim 1, select the model based on the dissimilarity values between the nodes with higher dissimilarity value)
It would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made to incorporate Hu‘s knowledge network platform into Nushi’s invention as they are related to the same field endeavor of ML model selection. The motivation to combine these arts, as proposed above, at least because Hu‘s model selection based on the differences between the models would help to provide model selection method into Nushi’s system. Therefore it would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made that providing model selection based on the difference between the models would help to improve accuracy of the ML model training.
In regard to Claim 4, Nushi disclose The model evaluation device according to claim 2,
But Nushi fail to explicitly disclose “wherein the at least one processor is configured to execute the instructions to: from among the generated second machine learning models, extract two second machine learning models in which the dissimilarity between the second machine learning models is lower compared with other second machine learning models, and select the second machine learning model by selecting either one of the extracted two second machine learning models.”
Hu disclose wherein the at least one processor is configured to execute the instructions to: from among the generated second machine learning models, extract two second machine learning models in which the dissimilarity between the second machine learning models is lower compared with other second machine learning models, and select the second machine learning model by selecting either one of the extracted two second machine learning models(col. 1, line 34-col. 2, line 32, col. 4, line 23-40, col. 5, line 3-42, col. 7, line 15-27, col. 13, line 28-col. 14, line 16, claim 1, select the model pair based on the dissimilarity values between the nodes with lower dissimilarity value and select the proper one)
It would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made to incorporate Hu‘s knowledge network platform into Nushi’s invention as they are related to the same field endeavor of ML model selection. The motivation to combine these arts, as proposed above, at least because Hu‘s model selection based on the differences between the models would help to provide model selection method into Nushi’s system. Therefore it would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made that providing model selection based on the difference between the models would help to improve accuracy of the ML model training.
In regard to Claim 5, Nushi disclose The model evaluation device according to claim 4,
But Nushi fail to explicitly disclose “wherein the at least one processor is configured to execute the instructions to select one of the extracted two second machine learning models, the one having a higher dissimilarity with another one of the generated second machine learning models.”
Hu disclose wherein the at least one processor is configured to execute the instructions to select one of the extracted two second machine learning models, the one having a higher dissimilarity with another one of the generated second machine learning models. (col. 1, line 34-col. 2, line 32, col. 4, line 23-40, col. 5, line 3-42, col. 7, line 15-27, col. 13, line 28-col. 14, line 16, claim 1, select the model based on the dissimilarity values between the nodes with higher dissimilarity value)
It would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made to incorporate Hu‘s knowledge network platform into Nushi’s invention as they are related to the same field endeavor of ML model selection. The motivation to combine these arts, as proposed above, at least because Hu‘s model selection based on the differences between the models would help to provide model selection method into Nushi’s system. Therefore it would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made that providing model selection based on the difference between the models would help to improve accuracy of the ML model training.
In regard to Claim 6, Nushi disclose The model evaluation device according to claim 2,
Nushi fail to explicitly disclose “wherein the at least one processor is configured to execute the instructions to, from the generated second machine learning models, further select the second machine learning model on a basis of a minimum spanning tree in a graph in which each of the generated second machine learning models is represented as a node and the dissimilarity between the second machine learning models is represented as a weight of a side linking the nodes.”
Hu disclose wherein the at least one processor is configured to execute the instructions to, from the generated second machine learning models, further select the second machine learning model on a basis of a minimum spanning tree in a graph in which each of the generated second machine learning models is represented as a node and the dissimilarity between the second machine learning models is represented as a weight of a side linking the nodes. (col. 1, line 34-col. 2, line 32, col. 7, line 15-27, col. 13, line 28-col. 14, line 16, claim 1, select the model with the minimum sum of weight of each link along the path between the model nodes, (shirted path) the models are represented by the nodes and dissimilarity between the nodes are represented as a weight of linking nodes)
It would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made to incorporate Hu‘s knowledge network platform into Nushi’s invention as they are related to the same field endeavor of ML model selection. The motivation to combine these arts, as proposed above, at least because Hu‘s model selection based on the differences between the models would help to provide model selection method into Nushi’s system. Therefore it would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made that providing model selection based on the difference between the models would help to improve accuracy of the ML model training.
In regard to claims 11-14, claims 11-14 are method claims corresponding to the device claims 3-6 above and, therefore, are rejected for the same reasons set forth in the rejections of claims 3-6.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure.
U.S. Patent Documents PATENT DATE INVENTOR(S) TITLE
US 20200005183 A1 2020-01-02 TANIMOTO et al.
EVALUATION METHOD FOR LEARNING MODELS, TRAINING METHOD, DEVICE, AND PROGRAM
TANIMOTO et al. disclose a device, a method, and a program which allow learning models to be appropriately evaluated or trained. The evaluation device according to an aspect performs the steps of: (A) obtaining, using checking data, a first execution result based on a first learning model as an exemplar model; (B) obtaining, using the checking data, a second execution result based on a second learning model; (C) determining whether or not the first and second execution results satisfy a logical formula; and (D) comparing, using a Bayesian statistical model checking method, respective behaviors of the first and second learning models with each other on the basis of a result of the determination in the step (C).… see abstract.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XUYANG XIA whose telephone number is (571)270-3045. The examiner can normally be reached Monday-Friday 8am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at 571-272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
XUYANG XIA
Primary Examiner
Art Unit 2143
/XUYANG XIA/ Primary Examiner, Art Unit 2143
Prosecution Insights