DETAILED ACTION
This action is responsive to the following communications: Original Application filed on October 12, 2023. All references to this application refer to the U.S. Patent Application Publication No. 2024/0169220 A1.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-12 are pending in this case. Claims 1 and 7 are the independent claims. Claims 1-12 are rejected.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. Applicants have perfected priority to Japanese Patent Application No. 2022-186181, filed on November 22, 2022.
Claim Objections
Claims 1 and 7 are objected to because of the following informalities:
In the third limitation of claim 1, the limitation recites “…and register the generated data in template management information.” This should recite “…and register the relation data in template management information.”
In the seventh limitation of claim 1, the limitation recites “…an evaluation result based on the evaluation method corresponding to the evaluation method data…” This should recite “…an evaluation result based on an evaluation method corresponding to the evaluation method data…”
In the third limitation of claim 7, the limitation recites “…and registering the generated data in template management information.” This should recite “…and registering the relation data in template management information.”
In the seventh limitation of claim 7, the limitation recites “…an evaluation result based on the evaluation method corresponding to the evaluation method data…” This should recite “…an evaluation result based on an evaluation method corresponding to the evaluation method data…”
Appropriate corrections are required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
With regard to claim 1,
Step 2A, Prong 1
This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim “recites” a judicial exception when the judicial exception is “set forth” or “described” in the claim.
Claim 1 recites:
1. A computer system, comprising:
a computer including a processor, a storage device connected to the processor, and a network interface connected to the processor, wherein
the computer system is accessibly connected to a model management information for managing model data including model-related items, risk assessment management information for managing risk assessment data including items related to model evaluation viewpoints, and evaluation method management information for managing evaluation method data including items related to evaluation methods, and configured to:
generate, as relation data, association of the model data, the risk assessment data, and the evaluation method data, included in a template defining a content of model evaluation, and register the generated data in template management information;
when receiving an evaluation request including information about a model to be evaluated, by referring to the model management information, search for the model data of the model to be evaluated; search for the relation data associated with the searched model data;
generate the template based on the searched relation data;
store, in association with the relation data, an evaluation result based on the evaluation method corresponding to the evaluation method data associated with the searched relation data; and
generate a report based on the generated template and the evaluation result.
The broadest reasonable interpretation of the bolded limitations above are directed to a mental process able to be performed in the human mind or by a human using pen and paper. A human can generate relation data of a model, including association, risk, and evaluation data of the model, record the relation data in a template, receive a request for data, retrieve requested data, and present the requested data to the user via a template. A human can complete these tasks mentally or with pen and paper.
Step 2A, Prong 1 (Yes).
Step 2A, Prong 2
This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception or whether the claim is “directed to” the judicial exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d).
The additional elements in this claim are the following:
a computer including a processor, a storage device connected to the processor, and a network interface connected to the processor;
Model management information for managing model data information including model-related items;
Risk assessment management information for managing risk assessment data including items related to model evaluation viewpoints;
Evaluation method management information for managing evaluation method data including items related to evaluation methods;
The first element is recited at a high level of generality and thus is a generic computer system performing computer functions. See MPEP 2106.04(d). The additional elements are information structures (model management, risk assessment management, evaluation method management, evaluation result). Information structures themselves do not improve functioning of a computer. See MPEP 2106.04(d)(I).
Even when viewed in combination the additional elements do not integrate the recited judicial exception into a practical application.
Step 2A, Prong 2 (Yes).
Step 2B
This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP 2106.05.
As explained with respect to Step 2A, the additional elements related to a generic computer system comprising generic computer components (e.g., processor, memory, network interface) and various information structures. As presented in the Specification, the computer system is a generic computer system (see Specification, paragraphs 0032-0039). Additionally, the model management information, risk assessment management information, and evaluation method management information, as recited in the Specification is mere information (see Specification, paragraphs 0040-0041). Additionally, the storage and retrieval of information has been determined by the courts as well-understood, routine, and conventional activities. See MPEP 2106.05(d)(II). Therefore, the additional elements cannot provide an inventive concept, even when considered in combination. See MPEP 2106.05(f).
Step 2B (Yes).
Claim 1 is ineligible.
With respect to independent claim 7,
These claims are a method embodiment of Claim 1 and are rejected under a similar rationale. The computer system, including a processor, a storage device connected to the processor, and a network interface connected to the processor, recited in these claims are also generic computing components.
Claim 7 is ineligible.
Dependent Claims:
Claims 2-6 and 8-12: These claims only recite further abstract ideas (mental processes) concerning searching, retrieving, comparing, and storing data, and thus are ineligible.
To expedite a complete examination of the instant application, the claims rejected above under 35 U.S.C. 101, as relating to judicial exceptions without significantly more, are further rejected as set forth below in anticipation of amendments to these claims to place them within the four statutory categories of invention.
Examiner’s Note
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the Examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicants are advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the Examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-12 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. 2022/0351051 A1, filed by Goto as U.S. National Stage Entry on February 9, 2022, and published on November 3, 2022 (hereinafter Goto), in view of U.S. Patent Application Publication No. 2019/0340518 A1, filed by Merrill et al., on April 25, 2019, and published on November 7, 2019 (hereinafter Merrill).
With respect to independent claim 1, Goto discloses a computer system, comprising:
A computer including a processor, a storage device connected to the processor, and a network interface connected to the processor, wherein the computer system is accessibly connected to a model management information for managing model data including model-related items…and evaluation method management information for managing evaluation method data including items related to evaluation methods, and configured to: Goto discloses a computer system comprising a processors, memory, and network interface connected to a database that stores model information including model data, risk data, and evaluation metrics (see Goto, Figs. 1-3 and 13-19; see also, Goto, paragraphs 0048-0053 [describing the architecture of Fig. 1, including template info for displaying information and the analysis system which analyzes the model, its inputs and outputs, and generates outputs of the predictive model; the template model includes item definitions, algorithm definitions, and view definitions], 0060 [describing the configuration of the analysis system of Fig. 2], 0063-0070[ describing the computer system architecture of Fig. 3, including processor, memory, and network interface], 0150-0159 [describing templates generally, including information about each template, such as template ID, name, outline, solution, engine type, objective variable, output variable, and item definitions], and 0168-0187 [describing the completed templates displayed in Figs. 13-19]).
Goto does not appear to expressly disclose risk assessment management information for managing risk assessment data including items related to model evaluation viewpoints.
However, Merrill teaches generating metadata and semantic data associated with predictive models (see Merrill, Figs. 2 and 11; see also, Merrill, paragraphs 0033 [describing how model information is generated, including model scoring information and risk data for models], 0035 [describing how the risk management documentation is automatically generated], 0041 [repository stores data about models, including parameters and tuning variables, scores, model evaluation and model analysis], 0042 [describing the formula for generating a score for each model based on actual vs. predicted performance], and 0139-0174 [describing the process of executing the Model Risk Management module to extract and store the risk data associated with the model]).
Accordingly, it would have been obvious to one of ordinary skill in the art, having the teachings of Goto and Merrill before him before the effective filing date of the claimed invention, to modify the system of Goto to incorporate generating and storing model risk assessment information as taught by Merrill. One would have been motivated to make such a combination because this assists in comparing predictive models using metrics, as taught by Merrill (see Merrill, paragraph 0004 [“There is a need in the machine learning field for new and useful systems for storing, updating, managing and using machine learning model metadata and statistics across the machine learning model lifecycle. There is a further need for automated tools that operate on machine learning model metadata and statistics to automate cumbersome, manual, time-consuming, and error-prone steps of the model development and verification process. Tools are also needed to generate the required model risk management documentation and analysis artifacts.”]).
Goto, as modified by Merrill, further teaches the system configured to:
Generate, as relation data, association of the model data, the risk assessment data, and the evaluation method data, included in a template defining a content of model evaluation, and register the generated data in template management information; Goto further teaches generating relationship data information for the various aspects of the model (model, risk, evaluation) that is associated with each model (see Goto, Figs. 12-19; see also, Goto, paragraphs 0129-0137 [Describing the report generated in Fig. 12, which includes sections for model details, evaluation metrics, variables, outputs, explanation variables, etc.]; see also, Goto, paragraphs 0150-0159 and 0168-0187, described supra). Merrill further teaches generating the risk assessment data (see Merrill, Figs. 2 and 11; see also, Merrill, paragraphs 0139-0174, described supra).
When receiving an evaluation request including information about a model to be evaluated, by referring to the model management information,
Search for the model data of the model to be evaluated; Goto further teaches a search screen interface for searching models (see Goto, Fig. 6; see also, Goto, paragraphs 0081-0085 [describing the template search interface and results]).
Search for the relation data associated with the searched model data; Goto describes searching for related model data (see Goto, Fig. 6; see also, Goto, paragraphs 0081-0085, described supra).
Generate the template based on the searched relation data; Goto further teaches generating the output report using the template and the model information data (see Goto, Fig. 12; see also, Goto, paragraphs 0081-0085, 0129-0137, 0168-0187, described supra).
Store, in association with the relation data, an evaluation result based on the evaluation method corresponding to the evaluation method data associated with the searched relation data; Merrill further teaches storing evaluation results with model data (see Merrill, paragraph 0042, described supra).
Generate a report based on the generated template and the evaluation result; Goto further teaches generating the output report using the template and the model information data (see Goto, Fig. 12; see also, Goto, paragraphs 0081-0085, 0129-0137, 0168-0187, described supra).
With respect to dependent claim 2, Goto, as modified by Merrill, teaches the computer system according to claim 1, as described above.
Goto and Merrill further teach the system configured to:
When receiving a comparative evaluation request including information for specifying a first model and a second model for comparison with each other, by referring to the template management information, Merrill further teaches using the evaluation metric to compare different models (see Merrill, paragraph 0052 [evaluation metric can be used for model comparison and champion/challenger analysis]; see also, Merrill, paragraphs 0146-0149, described supra, claim 1).
Search for first relation data associated with the model data of the first model; Goto further teaches searching for data associated with a first model (see Goto, Fig. 6; see also, Goto, paragraphs 0081-0085, described supra, claim 1).
Generate second relation data of the second model based on the first relation data and register the generated data in the template management information; Merrill further teaches generating data about a second model (or second version of the first model) (see Merrill, paragraphs 0042 and 0146-0149, described supra, claim 1).
Store, in association with the first relation data, an evaluation result of the evaluation method corresponding to the evaluation method data included in the first relation data of the first model; Merrill further teaches storing the evaluation data with the model data (see Merrill, paragraph 0042, described supra, claim 1).
Store, in association with the second relation data, an evaluation result of the evaluation method corresponding to the evaluation method data included in the second relation data of the second model; Merrill further teaches storing the evaluation data with the model data (see Merrill, paragraph 0042, described supra, claim 1).
Compare the first model and the second model based on the evaluation result of the first model and the evaluation result of the second model, and generate a report on the result of the comparison; Merrill further teaches comparing the first and second models based on their respective evaluation scores, and generating a report (see Merrill, paragraph 0042, described supra, claim 1).
With respect to dependent claim 3, Goto, as modified by Merrill, teaches the computer system according to claim 1, as described above.
Goto further teaches the system, configured to:
When receiving an evaluation viewpoint recommendation request including information about characteristics of a model, by referring to the model management information,
Search for the model data of the model having a specified characteristic; Goto further teaches searching for a model using specific characteristics (see Goto, paragraphs 0081-0085, described supra, claim 1).
By referring to the template management information, search for the relation data associated with the searched model data; Goto further teaches searching for the specified model data (see Goto, paragraphs 081-0085, described supra, claim 1).
Generate recommendation information to display the evaluation method data associated with the searched relation data; Goto further teaches generating model recommendation information based on the search criteria (see Goto, paragraphs 0081-0085, described supra, claim 1).
With respect to dependent claim 4, Goto, as modified by Merrill, teaches the computer system according to claim 1, as described above.
Goto and Merrill further teach the system, configured to:
When receiving an evaluation method recommendation request including information about an evaluation viewpoint, by referring to the risk assessment management information,
Search for the risk assessment data corresponding to a specified evaluation viewpoint; Goto further teaches searching for a specific model based on the requested evaluation viewpoint (see Goto, paragraphs 0081-0085, described supra, claim 1). Additionally, Merrill further teaches storing and retrieving risk assessment and evaluation data (see Merrill, paragraphs 0041, 0042, and 0139-0174, described supra, claim 1).
By referring to the template management information, search for the relation data associated with the searched risk assessment data; Goto further teaches searching for a specific model based on the requested evaluation viewpoint (see Goto, paragraphs 0081-0085, described supra, claim 1). Additionally, Merrill further teaches storing and retrieving risk assessment data (see Merrill, paragraphs 0041, 0042, and 0139-0174, described supra, claim 1).
Generate recommendation information to display the evaluation method data associated with the searched relation data; Goto further teaches retrieving for a specific model based on the requested evaluation viewpoint (see Goto, paragraphs 0081-0085, described supra, claim 1). Additionally, Merrill further teaches storing/retrieving risk and evaluation data (see Merrill, paragraphs 0041, 0042, and 0139-0174, described supra, claim 1; see also, Merrill, paragraph 0052, described supra, claim 2).
With respect to dependent claim 5, Goto, as modified by Merrill, teaches the computer system according to claim 1, as described above.
Goto and Merrill further teaches the system
Wherein the model data includes, as an item, information on an improvement method of the model, Merrill further teaches the model data includes an evaluation metric which can be used for comparison or improvement of models (see Merrill, paragraph 0042, described supra, claim 1; see also, Merrill, paragraph 0052, described supra, claim 2).
Wherein the computer system is configured to: when receiving an improvement method recommendation request including information about an evaluation method, by referring to the evaluation method management information,
Search for the evaluation method data corresponding to a specified evaluation method; Goto further teaches searching for a specific model based on the requested evaluation viewpoint (see Goto, paragraphs 0081-0085, described supra, claim 1). Additionally, Merrill further teaches storing and retrieving risk assessment data (see Merrill, paragraphs 0041, 0042, and 0139-0174, described supra, claim 1).
By referring to the template management information, search for the relation data associated with the searched evaluation method data; Goto further teaches searching for a specific model based on the requested evaluation viewpoint (see Goto, paragraphs 0081-0085, described supra, claim 1). Additionally, Merrill further teaches storing and retrieving risk assessment data (see Merrill, paragraphs 0041, 0042, and 0139-0174, described supra, claim 1).
Acquire the model data associated with the searched relation data from the model management information; Goto further teaches retrieving a specific model based on the searched model (see Goto, paragraphs 0081-0085, described supra, claim 1). Additionally, Merrill further teaches retrieving risk assessment data (see Merrill, paragraphs 0041, 0042, and 0139-0174, described supra, claim 1).
Generate recommendation information to display information about an improvement method of the model included in the acquired model data; Merrill further teaches generating recommendation information about an improvement method (see Merrill, paragraphs 0042 and 0146-0149, described supra, claim 1; see also, Merrill, paragraph 0052, described supra, claim 2).
With respect to dependent claim 6, Goto, as modified by Merrill, teaches the computer system according to claim 1, as described above.
Goto further teaches the system, configured to:
Present the generated template to a user; Goto further teaches presenting the generated template/report to a user (see Goto, Figs. 13-19; see also, Goto, paragraphs 0168-0187, described supra, claim 1).
Provide an interface for correcting the relation data; Goto further teaches providing an interface for correcting the model data (see Goto, Fig. 7; see also, Goto, paragraphs 0053 [describing how users can input data], 0076 [the user terminal allows for user input of model specification, data, etc.], 0087-0093 [describing how the system acquires input about models], and 0094-0096 [describing Fig. 7, and how the user can update data using drag and drop]).
Independent claim 7, and its respective dependent claims 8-12, recite a model evaluation method executed by the computer system recited in independent claim 1, and its respective dependent claims 2-6. Accordingly, independent claim 7, and its respective dependent claims 8-12, are rejected under the same rationales used to reject independent claim 1, and its respective dependent claims 2-6, which are incorporated herein.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicants’ disclosure. See PTO-892.
It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
Any inquiry concerning this communication or earlier communications from the Examiner should be directed to ERIC J. BYCER whose telephone number is (571) 270-3741. The Examiner can normally be reached Monday - Thursday 9am-6pm, and alternate Fridays 9am-5pm.
Examiner interviews are available via a variety of formats. See MPEP § 713.01. To schedule an interview, Applicants are encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/InterviewPractice.
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, MATT ELL can be reached on (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center to authorized users only. Should you have questions about access to the USPTO patent electronic filing system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/ERIC J. BYCER/
Primary Examiner
Art Unit 2141
Prosecution Insights