DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 6/23/26 has been entered.
Notice to Applicant
The following is a Non-Final Office Action responsive to Applicant’s communication of 6/23/26 in response to the Non-Final rejection of 12/10/25. Claims 1-20 are pending and have been rejected below.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e. an abstract idea) without reciting significantly more.
Step One - First, pursuant to step 1 in MPEP 2106.03, the claim 1 is directed to a method which is a statutory category.
Step 2A, Prong One - MPEP 2106.04 - The claim 1 recites–
“A… method comprising:
selecting at least one process;
identifying a plurality of metrics for said at least one process;
automatically calculating a performance envelope for at least one of said plurality of metrics said performance envelope comprising: an upper control limit (UCL), a lower control limit (LCL), and a plurality of different boundary values over a range of time periods for said UCL and said LCL;
continually monitoring, in real-time, data related to said at least one process for any data related to said plurality of metrics, and … updating said performance envelope for at least one of said plurality of metrics (Applicant’s example of Metrics in FIG. 5 and [0027-0029] include: send rate, offer acceptance rate, approval rate, authorization approval rate; [0079-0081] – bureau pass rate, credit underwriting rules for “bureau pass rate”; “underwriting rules” relative to errors);
automatically storing any of said data related to said plurality of metrics in an alert …;
automatically analyzing said stored data for any data points outside of said performance envelope for said at least one of said plurality of metrics;
wherein said at least one of said plurality of metrics are further monitored by a client utilizing said at least one process (Applicant’s specification [0019] as published states “The term “client” refers to a retailer, brand, business, IT department, IT manager, customer, manager, combination thereof, or the like, that would utilize the monitoring and alerting system for one or more of their own processes. [0064] as published states “In one embodiment, direct alert 210 is a real-time alert that will provide an (automated and/or curated) email, text, message, or other pop-up type alert directly to a client or a designated user or users (e.g., process manager, owner, designated IT personnel, or the like);
automatically generating an alert for one or more identified data points;
receiving a response for each of said identified data points, said response comprising: a description of a remediation performed in a user digestible response; and
updating, in said alert …, said identified data points with said response.”
As drafted, this is, under its broadest reasonable interpretation, within the Abstract idea grouping of “certain methods of organizing human activity” (e.g. commercial interactions (marketing – examples of “letter send rate, offer acceptance rate, same day offer acceptance rate” 571-574 in FIG. 5) or “managing interactions between people (following rules or instructions))” or “fundamental economic practices – (mitigating risk)” where the client (such as a business or manager) is monitoring the metrics (e.g. related to a credit/loan approval, “make offer rate 562”, “offer acceptance rate” 572”, Bureau Pass rate in [0079-0081] as published and FIG. 5) as well as directed to the Abstract idea grouping including “mathematical relationships” as here we are performing a series of mathematical operations- first, identifying metrics, calculating a performance envelope, including a statistical upper control limit and lower control limit for different time periods ([0023] as published – any process metrics or business KPIs; [0028] approval rate; [0031] bureau pass rate), analyzing data for performance outside an envelope, generating an alert where a response for the identified data points has a description of a remediation ([0021] as published - In general, the tool will analyze data related to the one or more processes and their underlying business metrics and provide real-time or near real-time user digestible information based on the results of the analysis. [0073] – personnel to respond to and remediate an alert). Accordingly, claim 1 is directed to an abstract idea because it is for commercial interactions for businesses in a series of metrics for a business process (e.g. a credit/loan approval and offers, along with approval rate metrics in some examples; FIG. 5, [0027-0030] as published - the various metrics relate to financial processes – accounts, “make offer rate”, “send rate”, approvals) and doing a series of mathematical calculations and analysis steps to calculate KPI (key performance indicators) along with analyzing data points outside of a threshold/envelope.
Step 2A, Prong Two - MPEP 2106.04 - This judicial exception is not integrated into a practical application. In particular, the claim 1 recites additional elements that are:
A computer-implemented method comprising:
…
continually monitoring, in real-time, data related to said at least one process for any data related to said plurality of metrics, and “automatically and dynamically” updating said performance envelope for at least one of said plurality of metrics;
automatically storing any of said data related to said plurality of metrics in an alert database;
updating, in said alert database, said identified data points with said response.”
(MPEP 2106.05f applies –the claim involves a computer in preamble, and is considered “apply it [the abstract ideas – certain methods of organizing human activity and math relationships] on a computer”; merely uses a computer as a tool to perform an abstract idea; the “storing” of data and a “database” is viewed as “field of use” (MPEP 2106.05h)).
Accordingly, the additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim also fails to recite any improvements to another technology or technical field, improvements to the functioning of the computer itself, use of a particular machine, effecting a transformation or reduction of a particular article to a different state or thing, and/or an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception. See 84 Fed. Reg. 55. The claim is directed to an abstract idea.
Step 2B in MPEP 2106.05 - The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of a computing system, is treated as MPEP 2106.05(f) (Mere Instructions to Apply an Exception – “Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible.” Alice Corp., 134 S. Ct. at 235)). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept.
The limitations of “”storing” data in a “database” is also a conventional computer functions – See MPEP 2106.05d(II)(iv) - Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334.
The claim fails to recite any improvements to another technology or technical field, improvements to the functioning of the computer itself, use of a particular machine, effecting a transformation or reduction of a particular article to a different state or thing, adding unconventional steps that confine the claim to a particular useful application, and/or meaningful limitations beyond generally linking the use of an abstract idea to a particular environment. See 84 Fed. Reg. 55. The claim is not patent eligible. Viewed individually or as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself.
Independent claim 8 is a statutory category at step one (article of manufacture). Claim 8 further recites:
“A non-transitory computer-readable medium for storing instructions, said instructions comprising: one or more instructions which, when executed by one or more processors, cause one or more processors” which are additional elements treated as “apply it [abstract idea] on a computer” (MPEP 2106.05f) at step 2a, prong two and step 2B. The remaining limitations are similar to claim 1 and are rejected for the same reasons at step 2a, prong one; step 2a, prong two, and step 2B. Examiner notes in claim 8, this means each step is by a “computer” and the recitations of “automatically” and the new “automatically and dynamically” are viewed as just “apply it [abstract idea] on a computer.” Examiner reminds Applicant that relying on the speed of the computer itself for business processes, like credit/loan approvals and financial metrics here, is viewed as similar to “MPEP 2106.05(a)(1)(I) Examples… not sufficient so show an improvement in computer-functionality: ii. Accelerating a process of analyzing audit log data when the increased speed comes solely from the capabilities of a general-purpose computer, FairWarning IP, LLC v. Iatric Sys., 839 F.3d 1089, 1095.
Independent claim 15 further recites:
“A computer system comprising: a memory; a display; and at least one processor, said at least one processor configured to” which are additional elements treated as “apply it [abstract idea] on a computer” (MPEP 2106.05f) at step 2a, prong two and step 2B. The remaining limitations are similar to claim 1 and claim 8 and are rejected for the same reasons at step 2a, prong one; step 2a, prong two, and step 2B.
Claims 2, 9, 16 narrow the abstract idea by naming the metrics used, e.g. KPI metrics.
Claims 3, 10, 17 narrow the abstract idea by having further mathematical relationships including normal behavior of attributes, and thresholds.
Claims 4, 11, 18 narrow the abstract idea by having further mathematical relationships including normal behavior of attributes, and value ranges.
Claims 5, 12 recite the limitations of claims 3 and 4 and are rejected for the same reasons.
Claims 6, 13 narrow the abstract idea by having further analysis and giving information to the user if no remediation is needed for the alert.
Claims 7, 14 narrow the abstract idea by having further analysis and giving information to the user if remediation was performed that includes a description of remediation.
Claim 19 narrows the abstract idea by giving alert of analysis to a user. It also recites additional elements to extent client is a 2nd computer. At step 2a, prong two and step 2B, this is treated as “apply it [abstract idea] on a computer” (MPEP 2106.05f).
Claim 20 is rejected for similar reasons as claim 19, and further recites a destination of software of the computer, where alert is given to “interactive dashboard application.” At this time, this only gives the destination of where alert goes to, and is also treated as “apply it [abstract idea] on a computer” (MPEP 2106.05f) and field of use (MPEP 2106.05h).
Therefore, the claim(s) are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
For more information on 101 rejections, see MPEP 2106.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Gopinathan (US 2015/0142713), Lowry, (US 2012/0123994) and Merrill (US 2019/0340518).
Concerning claim 1, Gopinathan discloses:
A method (Gopinathan see par 16 - method are disclosed that provide an adaptive, automated neural network setup for model creation and deployment, which continuously learns from its output and re-creates itself whenever required; his platform can be applied to any type of business that uses data from any event requiring a decision as an output, including (but not limited to) consumer credit, business credit… and many others) comprising:
selecting at least one process (Gopinathan see par 16 - method are disclosed that provide an adaptive, automated neural network setup for model creation and deployment, which continuously learns from its output and re-creates itself whenever required This platform amalgamates various processes that can essentially build the decision making system of a process or an entire collection of processes (such as required to operate an entire business), including but not limited to variable generation, model building, optimization and fine-tuning, portfolio performance evaluation, report generation and alert triggering mechanism);
identifying a plurality of metrics for said at least one process (Gopinathan – see par 50 - Since the data is always fetched from the Master Data Manager whenever a new model is build or an existing one is rebuilt, or the performance of a certain business metric has to be evaluated, this data needs to be synced into the Data Manager. The Master Data Manager 110 may be a master repository of all system data. The system also may have an Automated Report Generation Module 103 that takes data from Master Data Manager 110, and applies trigger alerts and distribution shift alarms along with reports on the performance of key business metrics. In a lending environment, for example, there is some expected minimum number of applications for loans, percentage of approvals, maximum risk cutoffs, fraud thresholds, etc that should be satisfied.);
automatically calculating a performance envelope for at least one of said plurality of metrics (Gopinathan – see par 50 - The system also may have an Automated Report Generation Module 103 that takes data from Master Data Manager 110, and applies trigger alerts and distribution shift alarms along with reports on the performance of key business metrics. In a lending environment, for example, there is some expected minimum number of applications for loans, percentage of approvals, maximum risk cutoffs, fraud thresholds, etc that should be satisfied. If any of these are not met, an alert is triggered by this system to notify for human intervention. See par 52 - If the performance of any of the deployed models or business metrics falls below a predefined threshold, it sends a request to the Model Building Module 105 to rebuild the underperforming model(s) or to a Decision Logic Module 108 to improve the corresponding decision logic; The decision logic might depend upon the underperforming models or be affected directly by the new data. The decision logic may be modified to make the threshold fit into the expected values.)
Gopinathan discloses setting an upper bound for acceptable range for controlling credit risk (See par 79).
Lowry discloses:
said performance envelope comprising: an upper control limit (UCL), a lower control limit (LCL), and a plurality of different boundary values over a range of time periods for said UCL and said LCL (Lowry – see par 45 - determining the upper control limit for the metric in step 203 of the method of FIG. 2, the computing system may access and load information stored in one or more data tables (e.g., to determine which function is to be used in calculating the upper control limit, and to determine how that particular function is defined). see par 46-47 - In step 204, a lower control limit may be determined for the metric. In one or more arrangements, determining a lower control limit for a metric may be similar to determining an upper control limit for a metric, except that a different modeling function may be selected and used in calculating the lower control limit and/or different data tables (e.g., data tables describing the function to be used in calculating the lower control limit) may be consulted accordingly by the computing system; see par 70, FIG. 5 - In addition, user interface 500 may include additional lines 502 and 503, which may graphically represent previous values of the upper control limit of the metric and the lower control limit of the metric, respectively; (FIG. 5 shows Upper control limit 502 and lower control limit 503 with changing values for each month in the graph) By reviewing the chart illustrated in user interface 500, an analyst for a financial institution, for instance, may be able to better understand the behavior of a particular metric and its control limit functions, which may enable the analyst to select better modeling functions for the metric and improve the calculations underlying both the metric and its various data quality evaluation functions)
Gopinathan and Lowry disclose:
continually monitoring, in real-time, data related to said at least one process for any data related to said plurality of metrics and automatically and dynamically updating said performance envelope for at least one of said plurality of metrics ([0023] as published –“metrics” is any process metrics or business KPIs; [0028] approval rate; [0031] bureau pass rate
Gopinathan discloses the limitations – see par 52 - The Model Management Module 106 may constantly monitor the performance levels of the models and business metrics with each new transaction. If the performance of any of the deployed models or business metrics falls below a predefined threshold, it sends a request to the Model Building Module 105 to rebuild the underperforming model(s) or to a Decision Logic Module 108 to improve the corresponding decision logic. For example, in a lending business, a lending decision may be made by making use of a set of models. The repayment behavior of the borrowers from latest vintages may be compared with the predictions of the models and if the performance parameters show a drop from estimated values, then the Model Management Module may trigger a request to the Model Building Module to rebuild models that are underperforming; The decision logic might depend upon the underperforming models or be affected directly by the new data. The decision logic may be modified to make the threshold fit into the expected values; see par 53- In the above lending example, where the profit model was rebuilt, the evaluation of performance of the models and trigger for the rebuild is given by the Model Management Module. The decision logic may also be revisited to check for any scope of optimization. With a modification of the models and other business rules, the system may generate a completely different score combination and decision logic;
see also Lowry – see par 51 - In step 206, a normalized quality score may be calculated for the metric. In one or more arrangements, this calculation may be based on the latest actual value of the metric, the forecast, the upper control limit, and/or the lower control limit. By computing such a normalized quality score for the metric, an organization, such as a financial institution, may be able to compare the data quality for the particular metric with the data quality of other metrics, which may enable the organization to target certain metrics for improvement; see par 70, FIG. 5 - In addition, user interface 500 may include additional lines 502 and 503, which may graphically represent previous values of the upper control limit of the metric and the lower control limit of the metric, respectively; (FIG. 5 shows Upper control limit 502 and lower control limit 503 with changing values for each month in the graph; see par 85 - by implementing one or more aspects of the disclosure, an organization, such as a financial institution, may be able to rapidly add hundreds or thousands of metrics (or more) to a collection of metrics being monitored by the organization);
automatically storing any of said data related to said plurality of metrics in an alert database (Gopinathan – see par 53 - The decision logic (comprising of the profit model score, a third party credit score and a fraud rule) is altered accordingly and the new thresholds are computed. Both the Model Building Module 105 and Model Management Module 106 use data from the Master Data Manager 110. See par 62, FIG. 3 - In the Near Real Time Processor 102, offline decision outcomes data 304, and data from other additional external sources 302, enters a Data Pre-Processing module 242, where complex variable computations, which include (but are not limited to) dynamic risk computations, etc, are carried out. External sources may include (but are not limited to) data collected from social networks, etc. The data then may be synched with the information received from the Variable Generation Module 104, which contains dependent variables (DVs) and independent variables (IDVs); see par 63, FIG. 1, 3 – Near Real Time Processor 102 enables generation of reports in a Reports Data Mart 306);
automatically analyzing said stored data for any data points outside of said performance envelope for said at least one of said plurality of metrics (Gopinathan – see par 52 - The Model Management Module 106 may constantly monitor the performance levels of the models and business metrics with each new transaction. A (new) transaction or event may be a request or an exchange or transfer of a tangible product or service for an asset/money/payment/promise of payment between one or more parties, for example a loan application from a customer to a lender, loan acceptance by lender. Another example may be a credit/debit card transaction for purchase of goods, acceptance of customer details by the retailer and delivery of the purchased entity. If the performance of any of the deployed models or business metrics falls below a predefined threshold; For example, in a lending business, a lending decision may be made by making use of a set of models. The repayment behavior of the borrowers from latest vintages may be compared with the predictions of the models and if the performance parameters show a drop from estimated values, then the Model Management Module may trigger a request to the Model Building Module to rebuild models that are underperforming;
see also Lowry –see par 52 - calculating a normalized quality score for the metric may include comparing the latest actual value of the metric with the forecast, the upper control limit, and/or the lower control limit to evaluate how closely the latest actual value matched the forecast and/or to determine whether and to what extent the latest actual value approached or exceeded the upper or lower control limit. see par 58 - if it is determined in optional step 208 that the latest actual value of the metric did exceed the upper control limit or the lower control limit for the metric, then in optional step 209, the metric may be added to an issue log; see par 59 - such an issue log may include a listing of various metrics, as well as detailed information about each metric to assist an analyst in assessing what remedial actions may be necessary to improve the data quality of the particular metric; see par 72, FIG. 7 - data table 700 may include, for a plurality of metrics, the most recent actual value of each metric, the most recent forecast value for each metric, the most recent lower control limit value for each metric, the most recent upper control limit value for each metric, the most recent quality score for each metric, and/or the date at which this quality score was determined.);
wherein said at least one of said plurality of metrics are further monitored by a client utilizing said at least one process (Applicant’s specification [0019] as published states “The term “client” refers to a retailer, brand, business, IT department, IT manager, customer, manager, combination thereof, or the like, that would utilize the monitoring and alerting system for one or more of their own processes. [0064] as published states “In one embodiment, direct alert 210 is a real-time alert that will provide an (automated and/or curated) email, text, message, or other pop-up type alert directly to a client or a designated user or users (e.g., process manager, owner, designated IT personnel, or the like).”
Gopinathan discloses the limitations based on broadest reasonable interpretation in light of the specification – a person/party/business/manager can receive the metrics by alert to manually monitor a process with metrics - see par 52 - The Model Management Module 106 may constantly monitor the performance levels of the models and business metrics with each new transaction. Another example may be a credit/debit card transaction ; see par 63 - The Near Real Time Processor 102 may also enable the generation of various types of reports on a continuous basis in a Reports Data Mart 306. The Master data Manager 110 provides data to the Automated Reports Generation Module 103 which computes various business performance metrics concerning, but not limited to profit, bad debt, acquisition, model performance, champion/challenger performance; In the system, alarms and alerts are triggered based on the comparative analysis against data from a previous day, previous month, previous year, previous market conditions or any such conditions specified. This module (103 report generation) constantly measures… and monitors other such diagnostic metrics. A case is also flagged for human intervention. The human operator may interfere whenever he/she so desires);
automatically generating an alert for one or more identified data points (Gopinathan – see FIG. 6, par 76 - the Model Management Module 106 may also synchronize the model score data, for offline validations, with the Modeling Data Mart and sends updated specifications with the new models and variables to the Automated Reports Generation Module 103 (part of the Near Real Time Module 102), which returns the metrics for all existing variables/models. The Model Management Module 106 may also continuously monitor the performance of the business strategy currently deployed and if the performance falls below a pre-specified threshold, automatically triggers rebuilds of the predictive models involved in the underperforming strategy. The Model Management Module 106 also may be able to identify the part of the strategy (model/set of models) that is under-performing and triggers an alert to the Model Building Module 105 to start a rebuilding process for the corresponding models; see par 78 - module 107 also may evaluate how the predictive models perform in the different challenger segments, and generates rebuild triggers for the model management module if the performance falls below expected or specified thresholds. After a sufficient number of transactions (varies by business product and specific challenger being tested), a successful challenger strategy becomes a part of the current business strategy in production; else it is either modified further or flagged for human intervention. See par 79 - In a lending environment, for example, if one wants to control credit risk and sets an upper bound for the acceptable range, then this module would take into account all the models, business rules, test segments, etc, to generate an optimized strategy which would approve only those application whose predicted cumulative credit risk falls below this upper bound. For example, in the case of John's loan application, the decision logic comprises of a profit model score, a third party credit score and a fraud alert and a threshold may be determined based on these three values. The Decision Logic Module 108 may also trigger a rebuild (giving few constraints) of certain models that are a part of the strategy. The final decision logic (including model, IDV, and DecisionLogic specs) arrived at will be reviewed and presented in a common format (across the system) that will enable automatic deployment into the Real Time Decision Engine 101. See par 80 - In an embodiment of the AMP system used for consumer credit, the Decision Logic Module 108 creates a business strategy based on constrained optimization where the objective function is (to maximize) Total Profit from the portfolio. Constraints typically are in the form of available lending capital and reserve requirements and specific conditions on metrics related to bad debt and unit profit (i.e. individual loan or customer level profit), and the inputs are the different predictive models built for each aspect of the portfolio (risk, conversion, profitability, income, etc). The business strategy typically ends up as a set of mathematical rules derived from the models that are applied to each transaction. The derived business strategy, along with the underlying predictive models and variables is then passed onto the Pre-Production Testing 109, to be eventually deployed into the Real Time Decision Engine 101.
see also Lowry – see par 59 - For example, in addition to identifying each metric by name and/or some other identifier, the issue log also may include, for each metric, historical performance information (e.g., past actual values for the metric, past forecasted values for the metric, etc.), one or more charts and graphs, information about the forecasting functions used for the particular metric, and/or other information. Using this information, an organization, such as a financial institution, may be able to identify ways to improve the data quality of a particular metric, such as by modifying the functions used to calculate the forecast and control limits for the metric, by changing the data sources and/or sample sizes used in calculating the metric, and/or by taking other actions.);
receiving a response for each of said identified data points (Gopinathan – see par 63 - The Near Real Time Processor 102 may also enable the generation of various types of reports on a continuous basis in a Reports Data Mart 306. The Master data Manager 110 provides data to the Automated Reports Generation Module 103 which computes various business performance metrics concerning, but not limited to profit, bad debt, acquisition, model performance, champion/challenger performance. This module constantly measures correlations between different independent variables, correlation between independent variables and dependent variables, applies the Kolmogorov-Smirnov statistical distribution test across all variables to identify significant distributional shifts over various time intervals, and monitors other such diagnostic metrics. If an anomaly is found, this module automatically triggers corrective action invoking the Model Management Module 106 (which then triggers model rebuilds for affected variables); See par 76 - the models that are deployed in the Real Time Decision Engine 101 can be set to be rebuilt every day (or even every hour/minute, depending on the business) and adapt dynamically to the ever changing environment. In a lending environment, the models in place may predict risk involved in a transaction. Marketing associated with the incoming traffic might change, resulting in a different behavior of the applicants from the one predicted by the risk model. If this model is built again using data from recent transactions, it should take into account the bias caused by the marketing campaign and predict risk more accurately henceforth.), said response comprising: a description of a remediation performed in a user digestible format (First, Examiner notes that the content of the “description” here is not entitled to patentable weight – see MPEP 2111.05 - , where the claim as a whole is directed to conveying a message or meaning to a human reader independent of the intended computer system, and/or the computer-readable medium merely serves as a support for information or data, no functional relationship exists. For example, a claim to a memory stick containing tables of batting averages, or tracks of recorded music, utilizes the intended computer system merely as a support for the information. Such claims are directed toward conveying meaning to the human reader rather than towards establishing a functional relationship between recorded data and the computer.
Nonetheless, for compact prosecution, art still applied:
For claim interpretation - Applicant specification [0021] as published states “In general, the tool will analyze data related to the one or more processes and their underlying business metrics and provide real-time or near real-time user digestible information based on the results of the analysis.”; [0076-0077] as published states “the self-service RCA path 431 and the managed RCA path 432 (which could also therefore include the hybrid path) include a research 436 feature used to research any data that has successfully passed through the prioritize/filter 435 component. In one embodiment, the report includes all the data that has successfully passed through the prioritize/filter 435 component as well as any research found by the research 436 feature
Gopinathan discloses the limitations based on broadest reasonable interpretation in light of the specification – See par 76 - The Model Management Module 106 may also continuously monitor the performance of the business strategy currently deployed and if the performance falls below a pre-specified threshold, automatically triggers rebuilds of the predictive models involved in the underperforming strategy. The Model Management Module 106 also may be able to identify the part of the strategy (model/set of models) that is under-performing and triggers an alert to the Model Building Module 105 to start a rebuilding process for the corresponding models; see par 78 - module 107 also may evaluate how the predictive models perform in the different challenger segments, and generates rebuild triggers for the model management module if the performance falls below expected or specified thresholds. After a sufficient number of transactions (varies by business product and specific challenger being tested), a successful challenger strategy becomes a part of the current business strategy in production; else it is either modified further or flagged for human intervention;
see also Lowry – see par 54 - a normalized quality score lesser than negative one or greater than positive one may indicate that some remedial action is required. For example, such a score may be indicative of a problem with data collection (e.g., some samples were uncounted, miscounted, etc.) or forecasting (e.g., the functions defining the forecast, upper control limit, and/or lower control limit do not accurately model the trend), or it may suggest that some substantive change has occurred (e.g., something in the substance or nature of the metric itself has changed; for instance, a retailer being measured has gone out of business, a workforce reduction has occurred, etc.). see par 69 - By considering the graph illustrated in user interface 400, an analyst associated with a financial institution, for instance, may be able to quickly and easily determine which metrics require remediation for data quality issues. For example, it may be seen that bar 401 extends beyond the control limit line at +1.0, which may indicate that the metric corresponding to bar 401 has exceeded its upper control limit. Accordingly, based on the graph, it may be determined that remedial action should be taken to address and/or correct the data quality issues associated with that metric.)
It is unclear if Gopinathan discloses a “response” as best understood, and a successful strategy for getting performance above a threshold can be “flagged for human intervention” (See par 78).
Merrill is also applied here as it discloses:
receiving a “response” for each of said identified data points, said “response comprising: a description of a remediation performed in a user digestible format” (Merrill – See par 28 - embodiments disclosed herein provide libraries and tools that allow an operator to perform modeling and analysis tasks that record data in a knowledge graph, including tasks such as allowing an operator to flag unreliable variables and features, for instance: flagging of features that lead to disparities in approval rate between protected classes and a baseline;… including comparison of distributions of each feature in the population that would be newly approved by the model with the distributions of the same feature within the population of applicants that would have been approved by both the old model and the new model; See par 74 - In one embodiment, the disparate impact analysis includes approval rates among population segments such as protected classes and a baseline, determination of which features contribute to the approval rate disparity, and an analysis of each feature's economic impact. In one embodiment, the disparate impact analysis includes the disposition of a variable's suitability for inclusion in the model (included or suppressed) and the reason for the disposition (including a business justification, quantification of harm, and natural language representing the reasoning behind the disposition, and attribution of the judgment to an identified responsible person). See par 172 - These machine-readable relations enable new and useful automated analysis and reporting: a query to the knowledge graph may now answer if a detected anomaly matters (is this feature important to the model and what is the economic impact of the anomaly) and how to fix it).
Gopinathan, Lowry, and Merrill disclose:
updating, in said alert database, said identified data points with said response (Gopinathan – See par 53 - In the above lending example, where the profit model was rebuilt, the evaluation of performance of the models and trigger for the rebuild is given by the Model Management Module. The decision logic may also be revisited to check for any scope of optimization. With a modification of the models and other business rules, the system may generate a completely different score combination and decision logic. In this case, since only one model is considered, for the sake of simplicity let model M2 replaces model M1. The decision logic (comprising of the profit model score, a third party credit score and a fraud rule) is altered accordingly and the new thresholds are computed. see par 54 - In a lending business, from time to time, new models may be built, improved strategies and algorithms may be developed, third party services may get updated and improved versions may have to be put into use for better results and more accurate estimates. See par 76 – newly build profit model with new variables; The Model Management Module 106 may also synchronize the model score data, for offline validations, with the Modeling Data Mart and sends updated specifications with the new models and variables to the Automated Reports Generation Module 103 (part of the Near Real Time Module 102), which returns the metrics for all existing variables/models
Merrill - see par 44 - In one embodiment, a dashboard of anomalous model inputs and scores is presented to an operator so they can inspect the outliers and take action to retrain the model or ignore the alerts and add the anomalous data to the baseline data for the modeling system. In one embodiment, each addition to baseline data is tracked in a knowledge graph so that the history of modifications may be preserved and rolled back, should the operator choose to undo past modifications. In this way an intelligent monitoring system may be constructed that learns from human judgment as alerts are cleared or actioned; See par 172 - These machine-readable relations enable new and useful automated analysis and reporting: a query to the knowledge graph may now answer if a detected anomaly matters (is this feature important to the model and what is the economic impact of the anomaly) and how to fix it (it came from this feature which is comprised of these other inputs that come from these data stores using this function to combine them.)
Gopinathan, Lowry and Merrill are analogous art as they are directed to analyzing data/metrics for financial/credit/loan applications (see Gopinathan Abstract, par 50; Lowry Abstract par 33; Merrill Abstract, par 35 – fair lending, par 163 - features related to loans). 1) Gopinathan discloses setting an upper bound for acceptable range for controlling credit risk (See par 79). Lowry improves upon Gopinathan by disclosing determining upper and lower control limits for metrics, where the upper and lower limits can change over time (e.g. months) (See par 45-47, 70, FIG. 5). One of ordinary skill in the art would be motivated to further include determining upper and lower control limits for metrics, where the upper and lower limits can change over time to efficiently improve upon the upper bounds of acceptable range for credit risk in Gopinathan. 2) Gopinathan discloses triggering corrective actions of model rebuilds when anomalies found (See par 63), rebuilding risk model based on more recent transactions and identifying underperforming strategies as well as flagging strategies for human intervention (See par 52, 76, 78). Lowry discloses determining remediation is needed for a metric of a financial institution for issues/corrections (See par 69). Merrill improves upon Gopinathan and Lowry by disclosing flagging features leading to disparities in approval rates, then use of a new model to check different distributions of features (See par 28), ability to possibly add anomalous data to baseline data for modeling/training, determining features contributing to an “approval rate disparity” and a reason for including in a model (See par 74), providing reporting on “how to fix” an anomaly (See par 44, 172). One of ordinary skill in the art would be motivated to further include having detection of features relative to approval rates that can result in modifying the model and detection of anomalous data and how to fix it or change models/training data to efficiently improve upon the rebuilding of models based on recent transactions in Gopinathan and the remediation of metrics in Lowry.
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the rebuilding of models in the lending environment in Gopinathan (See par 50, 63, 76), to further include determining upper and lower control limits as disclosed in Lowry, to further include fixing/modifying models/training related to loan approvals as disclosed in Merrill, since the claimed invention is merely a combination of old elements, and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable and there is a reasonable expectation of success.
Concerning independent claim 8, Gopinathan, Lowry, and Merrill disclose:
A non-transitory computer-readable medium for storing instructions, said instructions comprising: one or more instructions which, when executed by one or more processors, cause one or more processors to (Gopinathan – See par 37- In a software implementation of the AMP 100, each component shown in FIG. 1 may be implemented as a plurality lines of computer code that may be stored on a computer readable medium, such as a CD, DVD, flash memory, persistent storage device, cloud computing storage and then may be executed by a processor.).
The remaining limitations are similar to claim 1 above and are rejected for the same reasons over Gopinathan, Lowry, and Merrill.
Concerning independent claim 15, Gopinathan, Lowry, and Merrill disclose:
A computer system (Gopinathan – see par 37 – computer system) comprising:
a memory (Gopinathan – See par 37- In a software implementation of the AMP 100, each component shown in FIG. 1 may be implemented as a plurality lines of computer code that may be stored on a computer readable medium, such as a CD, DVD, flash memory, persistent storage device, cloud computing storage and then may be executed by a processor).
Gopinathan discloses having reports (See par 50, 63).
Merrill discloses:
a display (Merrill – par 188 – FIG. 3 – device with display device and user input device; see par 31 – displaying output to operator).
Gopinathan discloses:
at least one processor, said at least one processor configured to: (Gopinathan – See par 37- In a software implementation of the AMP 100, each component shown in FIG. 1 may be implemented as a plurality lines of computer code that may be stored on a computer readable medium, such as a CD, DVD, flash memory, persistent storage device, cloud computing storage and then may be executed by a processor.).
The remaining limitations are similar to claim 1 above and are rejected for the same reasons over Gopinathan, Lowry, and Merrill. In addition, Merrill improves upon the computer with reporting in Gopinathan by explicitly having a “display”.
Concerning claims 2, 9, and 16 Gopinathan disclose:
The computer-implemented method of claim 1 wherein said plurality of metrics are selected from a group consisting of: system performance metrics, key performance indicators (KPI) metrics, and a combination of said system performance metrics and said KPI metrics (Gopinathan see par 63 - The Master data Manager 110 provides data to the Automated Reports Generation Module 103 which computes various business performance metrics concerning, but not limited to profit, bad debt, acquisition, model performance, champion/challenger performance.)
Concerning claims 3, 10, and 17, Gopinathan discloses that in a lending environment, there is “expected” minimum number of applications for loans, percentage of approvals, maximum risk cutoffs, fraud thresholds, etc that should be satisfied (See par 50), that when a model is underperforming, loan decision logic may be modified to make the threshold fit into expected values (See par 52), and that “The decision logic (comprising of the profit model score, a third party credit score and a fraud rule) is altered accordingly and the new thresholds are computed” (See par 53).
Lowry discloses:
The computer-implemented method of claim 1 wherein said performance envelope comprises:
one or more automatically and dynamically developed thresholds indicative of the end of a range of normal behavior for one or more process attributes (Lowry – see par 42 - once a forecast has been calculated for a metric, it may be desirable to determine and/or set control limits around that forecast, so as to define some measures against which the accuracy of the forecast may be evaluated (once the actual latest value of the metric is eventually determined). In one or more arrangements, the control limits thus may define a range within which variance from the forecast may be deemed acceptable. Notably, different metrics may have different control limits, as further discussed below. For example, for some metrics, a variation of plus or minus ten percent from the forecast may be within acceptable control limits. see par 54 - a normalized quality score lesser than negative one or greater than positive one may indicate that some remedial action is required. For example, such a score may be indicative of a problem with data collection; see par 59 - the issue log also may include, for each metric, historical performance information (e.g., past actual values for the metric, past forecasted values for the metric, etc.), one or more charts and graphs, information about the forecasting functions used for the particular metric, and/or other information. Using this information, an organization, such as a financial institution, may be able to identify ways to improve the data quality of a particular metric, such as by modifying the functions used to calculate the forecast and control limits for the metric, by changing the data sources and/or sample sizes used in calculating the metric, and/or by taking other actions.); see par 70, FIG. 5 - In addition, user interface 500 may include additional lines 502 and 503, which may graphically represent previous values of the upper control limit of the metric and the lower control limit of the metric, respectively; (FIG. 5 shows Upper control limit 502 and lower control limit 503 with changing values for each month in the graph)) selected from a group consisting of: activities, usage, results, and queries (Lowry – see par 54 - a normalized quality score lesser than negative one or greater than positive one may indicate that some remedial action is required. For example, such a score may be indicative of a problem with data collection (e.g., some samples were uncounted, miscounted, etc.) or forecasting (e.g., the functions defining the forecast, upper control limit, and/or lower control limit do not accurately model the trend), or it may suggest that some substantive change has occurred (e.g., something in the substance or nature of the metric itself has changed; for instance, a retailer being measured has gone out of business, a workforce reduction has occurred, etc).
Gopinathan discloses that in a lending environment, there is “expected” minimum number of applications for loans, percentage of approvals, maximum risk cutoffs, fraud thresholds, etc that should be satisfied (See par 50), that when a model is underperforming, loan decision logic may be modified to make the threshold fit into expected values (See par 52), and that “The decision logic (comprising of the profit model score, a third party credit score and a fraud rule) is altered accordingly and the new thresholds are computed” (See par 53). Lowry improves upon Gopinathan and Merrill by disclosing data modifying control limits for metrics and improving data quality of a metric by considering data collection, samples, and accuracy for modeling the trends. One of ordinary skill in the art would be motivated to further include modifying control limits for metrics and improving data quality of a metric to efficiently improve upon the “new thresholds” being computed for profit model purposes in Gopinathan and the fixing of data related to detected anomalies in Merrill.
Concerning claims 4, 11, and 18 Gopinathan, Merrill, and Way disclose:
The computer-implemented method of claim 1 wherein said performance envelope comprises: one or more automatically and dynamically developed value ranges indicative of a normal behavior for one or more process attributes (Lowry – see par 56 - In optional step 207, a comparison chart may be generated. In one or more arrangements, the comparison chart may include the normalized quality score for the metric and at least one other normalized quality score for at least one other metric. For example, such a comparison chart may take the form of a bar graph, and because the quality scores underlying the bars displayed in the graph may be normalized, the heights of the various bars may indicate the data quality of the various metrics in relation to each other. Using such a graph, an organization, such as a financial institution, may be able to quickly and easily identify data quality issues in particular metrics, even if tens of thousands metrics are being analyzed and evaluated. see par 70 - In addition, user interface 500 may include additional lines 502 and 503, which may graphically represent previous values of the upper control limit of the metric and the lower control limit of the metric, respectively. By reviewing the chart illustrated in user interface 500, an analyst for a financial institution, for instance, may be able to better understand the behavior of a particular metric and its control limit functions, which may enable the analyst to select better modeling functions for the metric and improve the calculations underlying both the metric and its various data quality evaluation functions (e.g., the forecasting method for the metric, the functions used in calculating the control limits for the metric, etc.); see par 70, FIG. 5 - In addition, user interface 500 may include additional lines 502 and 503, which may graphically represent previous values of the upper control limit of the metric and the lower control limit of the metric, respectively; (FIG. 5 shows Upper control limit 502 and lower control limit 503 with changing values for each month in the graph)) selected from a group consisting of: activities, usage, results, and queries (Lowry – see par 54 - a normalized quality score lesser than negative one or greater than positive one may indicate that some remedial action is required. For example, such a score may be indicative of a problem with data collection (e.g., some samples were uncounted, miscounted, etc.) or forecasting (e.g., the functions defining the forecast, upper control limit, and/or lower control limit do not accurately model the trend), or it may suggest that some substantive change has occurred (e.g., something in the substance or nature of the metric itself has changed; for instance, a retailer being measured has gone out of business, a workforce reduction has occurred, etc).
It would have been obvious to combine Gopinathan, Merrill, and Lowry for the same reasons as claim 1 and 3 above.
Concerning claims 5 and 12, Gopinathan disclose:
The computer-implemented method of claim 1 wherein said performance envelope comprises:
one or more automatically and dynamically developed thresholds for one or more process attributes (Lowry – see par 42 - once a forecast has been calculated for a metric, it may be desirable to determine and/or set control limits around that forecast, so as to define some measures against which the accuracy of the forecast may be evaluated (once the actual latest value of the metric is eventually determined). In one or more arrangements, the control limits thus may define a range within which variance from the forecast may be deemed acceptable. Notably, different metrics may have different control limits, as further discussed below. For example, for some metrics, a variation of plus or minus ten percent from the forecast may be within acceptable control limits. see par 54 - a normalized quality score lesser than negative one or greater than positive one may indicate that some remedial action is required. For example, such a score may be indicative of a problem with data collection; see par 59 - the issue log also may include, for each metric, historical performance information (e.g., past actual values for the metric, past forecasted values for the metric, etc.), one or more charts and graphs, information about the forecasting functions used for the particular metric, and/or other information. Using this information, an organization, such as a financial institution, may be able to identify ways to improve the data quality of a particular metric, such as by modifying the functions used to calculate the forecast and control limits for the metric, by changing the data sources and/or sample sizes used in calculating the metric, and/or by taking other actions; see par 70, FIG. 5 - In addition, user interface 500 may include additional lines 502 and 503, which may graphically represent previous values of the upper control limit of the metric and the lower control limit of the metric, respectively; (FIG. 5 shows Upper control limit 502 and lower control limit 503 with changing values for each month in the graph)) selected from a group consisting of: activities, usage, results, and queries; and
one or more automatically and dynamically developed value ranges for one or more process attributes selected from said group consisting of: activities, usage, results, and queries (Lowry – see par 54 - a normalized quality score lesser than negative one or greater than positive one may indicate that some remedial action is required. For example, such a score may be indicative of a problem with data collection (e.g., some samples were uncounted, miscounted, etc.) or forecasting (e.g., the functions defining the forecast, upper control limit, and/or lower control limit do not accurately model the trend), or it may suggest that some substantive change has occurred (e.g., something in the substance or nature of the metric itself has changed; for instance, a retailer being measured has gone out of business, a workforce reduction has occurred, etc).
The same citations from claims 1, 3 and 4 above apply and it would have been obvious to combine Gopinathan, Merrill, and Lowry for the same reasons as claims 1, 3 and 4 above.
Concerning claims 6 and 13, Gopinathan and Merrill disclose:
The computer-implemented method of claim 1 further comprising:
analyzing said alert for one of said identified data points (Gopinathan – see par 76 - The Model Management Module 106 also may be able to identify the part of the strategy (model/set of models) that is under-performing and triggers an alert to the Model Building Module 105 to start a rebuilding process for the corresponding models; See also Merrill );
determining if said remediation is needed for said alert based on said analysis of said alert (Gopinathan – See par 50 - In a lending environment, for example, there is some expected minimum number of applications for loans, percentage of approvals, maximum risk cutoffs, fraud thresholds, etc that should be satisfied. If any of these are not met, an alert is triggered by this system to notify for human intervention; see par 78 - After a sufficient number of transactions (varies by business product and specific challenger being tested), a successful challenger strategy becomes a part of the current business strategy in production; else it is either modified further or flagged for human intervention;
See also Merrill - see par 44 - In one embodiment, a dashboard of anomalous model inputs and scores is presented to an operator so they can inspect the outliers and take action to retrain the model or ignore the alerts and add the anomalous data to the baseline data for the modeling system; See par 172 - These machine-readable relations enable new and useful automated analysis and reporting: a query to the knowledge graph may now answer if a detected anomaly matters (is this feature important to the model and what is the economic impact of the anomaly) and how to fix it); and
Gopinathan discloses that if rules are not satisfied, an alert is triggered for human intervention (See par 50) and that there is a pre-production testing engine to identify “any” issues (See par 56). Merrill discloses ignoring alerts and adding anomalous data to baseline data for modeling (See par 44).
Moore discloses:
providing, as said response, a no remediation performed message if no remediation is needed (Lowry – see par 67 - FIG. 4 illustrates an example user interface for analyzing data quality for a plurality of metrics according to one or more aspects described herein; see FIG. 4, par 69 - by considering the graph illustrated in user interface 400, an analyst associated with a financial institution, for instance, may be able to quickly and easily determine which metrics require remediation for data quality issues. For example, it may be seen that bar 401 extends beyond the control limit line at +1.0, which may indicate that the metric corresponding to bar 401 has exceeded its upper control limit. Accordingly, based on the graph, it may be determined that remedial action should be taken to address and/or correct the data quality issues associated with that metric. … On the other hand, it may be seen that bars 402-409 fall within the control limit lines, which may indicate that the data quality levels for these metrics are within acceptable ranges.).
It would have been obvious to combine Gopinathan, Lowry, and Merrill for the same reasons as claim 1 above. Gopinathan discloses that if rules are not satisfied, an alert is triggered for human intervention (See par 50) and that there is a pre-production testing engine to identify “any” issues (See par 56). Merrill discloses ignoring alerts and adding anomalous data to baseline data for modeling (See par 44). Lowry improves upon Gopinathan and Merrill by disclosing analyzing plurality of metrics and indicating that some metrics are within acceptable ranges and need no remediation (See par 67-69). One of ordinary skill in the art would be motivated to further include analyzing plurality of metrics and indicating that some metrics are within acceptable ranges and need no remediation to efficiently improve upon the alert triggered for human intervention for issues in Gopinathan and the fixing of data related to detected anomalies in Merrill.
Concerning claims 7 and 14, Gopinathan and Merrill disclose:
The computer-implemented method of claim 1 further comprising:
analyzing said alert for one of said identified data points (Gopinathan – see par 76 - The Model Management Module 106 also may be able to identify the part of the strategy (model/set of models) that is under-performing and triggers an alert to the Model Building Module 105 to start a rebuilding process for the corresponding models; See also Merrill);
determining if said remediation is needed for said alert based on said analysis of said alert (Gopinathan – See par 50 - In a lending environment, for example, there is some expected minimum number of applications for loans, percentage of approvals, maximum risk cutoffs, fraud thresholds, etc that should be satisfied. If any of these are not met, an alert is triggered by this system to notify for human intervention.
See also Merrill - see par 44 - In one embodiment, a dashboard of anomalous model inputs and scores is presented to an operator so they can inspect the outliers and take action to retrain the model or ignore the alerts and add the anomalous data to the baseline data for the modeling system);
performing said remediation (Gopinathan – see par 63 - This module constantly measures correlations between different independent variables, correlation between independent variables and dependent variables, applies the Kolmogorov-Smirnov statistical distribution test across all variables to identify significant distributional shifts over various time intervals, and monitors other such diagnostic metrics. If an anomaly is found, this module automatically triggers corrective action invoking the Model Management Module 106 (which then triggers model rebuilds for affected variables); see par 78 - module 107 also may evaluate how the predictive models perform in the different challenger segments, and generates rebuild triggers for the model management module if the performance falls below expected or specified thresholds. After a sufficient number of transactions (varies by business product and specific challenger being tested), a successful challenger strategy becomes a part of the current business strategy in production; else it is either modified further or flagged for human intervention;
see also Merrill - see par 44 - In one embodiment, the alerting system sends emails, SMS messages, and other forms of alerts to operators based on the numeric score, in order to alert the operator that the model may be encountering data or behaving in ways that are unsafe. In one embodiment, a dashboard of anomalous model inputs and scores is presented to an operator so they can inspect the outliers and take action to retrain the model or ignore the alerts and add the anomalous data to the baseline data for the modeling system); and
providing, as said response, said description of said remediation performed in said user digestible format (Gopinathan – See par 74 - In one embodiment, the disparate impact analysis includes approval rates among population segments such as protected classes and a baseline, determination of which features contribute to the approval rate disparity, and an analysis of each feature's economic impact. In one embodiment, the disparate impact analysis includes the disposition of a variable's suitability for inclusion in the model (included or suppressed) and the reason for the disposition (including a business justification, quantification of harm, and natural language representing the reasoning behind the disposition; see FIG. 6, par 76 - the Model Management Module 106 may also synchronize the model score data, for offline validations, with the Modeling Data Mart and sends updated specifications with the new models and variables to the Automated Reports Generation Module 103 (part of the Near Real Time Module 102), which returns the metrics for all existing variables/models. The Model Management Module 106 may also continuously monitor the performance of the business strategy currently deployed and if the performance falls below a pre-specified threshold, automatically triggers rebuilds of the predictive models involved in the underperforming strategy. The Model Management Module 106 also may be able to identify the part of the strategy (model/set of models) that is under-performing and triggers an alert to the Model Building Module 105 to start a rebuilding process for the corresponding models; see par 78 - module 107 also may evaluate how the predictive models perform in the different challenger segments, and generates rebuild triggers for the model management module if the performance falls below expected or specified thresholds. After a sufficient number of transactions (varies by business product and specific challenger being tested), a successful challenger strategy becomes a part of the current business strategy in production;
Merrill – see par 69 - In some embodiments, the system 101 includes an MRM module for automated generation of a data dictionary that includes a table and an associated narrative. See par 74 - In one embodiment, the disparate impact analysis includes approval rates among population segments such as protected classes and a baseline, determination of which features contribute to the approval rate disparity, and an analysis of each feature's economic impact. In one embodiment, the disparate impact analysis includes the disposition of a variable's suitability for inclusion in the model (included or suppressed) and the reason for the disposition (including a business justification, quantification of harm, and natural language representing the reasoning behind the disposition; See par 172 - These machine-readable relations enable new and useful automated analysis and reporting: a query to the knowledge graph may now answer if a detected anomaly matters (is this feature important to the model and what is the economic impact of the anomaly)).
It would have been obvious to combine Gopinathan, Lowry and Merrill for the same reasons as claim 1 above.
Concerning claim 19, Gopinathan discloses:
The computer system of claim 15, wherein said alert comprises: a direct alert automatically provided directly to said client, the direct alert selected from a group consisting of: an email, a text, a message, and a pop-up type alert (Gopinathan – see par 63 - The Near Real Time Processor 102 may also enable the generation of various types of reports on a continuous basis in a Reports Data Mart 306. The Master data Manager 110 provides data to the Automated Reports Generation Module 103 which computes various business performance metrics concerning, but not limited to profit, bad debt, acquisition, model performance, champion/challenger performance (“report” discloses “message”);
Merrill – see par 28 - Some embodiments disclosed herein provide libraries and tools that allow an operator to perform modeling and analysis tasks that record data in a knowledge graph, including tasks such as allowing an operator to flag unreliable variables and features, for instance: features that are largely missing, extremely low variance, or that have large shifts in distribution over time; flagging of features that lead to disparities in approval rate between protected classes and a baseline… , comparison of approval rates between protected classes, determination of which features contribute to disparity, and a quantification of a feature's contribution to the approval rate disparity as well as their contribution to the economics of the model; and other analysis; see par 44, 178-179 – model monitoring module… stored in the knowledge graph to generate an alert; alert is email or text message to operators).
It would have been obvious to combine Gopinathan, Lowry, and Merrill for the same reasons as claim 1 above.
Concerning claim 20, Gopinathan, Lowry, and Merrill disclose:
The computer system of claim 15, wherein said alert comprises: a summarizing alert automatically provided to an interactive dashboard application (Merrill – see par 44 - In one embodiment, the alerting system sends emails, SMS messages, and other forms of alerts to operators based on the numeric score, in order to alert the operator that the model may be encountering data or behaving in ways that are unsafe. In one embodiment, a dashboard of anomalous model inputs and scores is presented to an operator so they can inspect the outliers and take action to retrain the model or ignore the alerts and add the anomalous data to the baseline data for the modeling system).
It would have been obvious to combine Gopinathan, Lowry, and Merrill for the same reasons as claim 1 above. Merrill improves upon the alerts and reports of Gopinathan and the remediation of metrics in Lowry by explicitly disclosing having a “dashboard” with a plurality of information (e.g. model inputs, scores, outliers).
Response to Arguments
Applicant’s arguments filed 6/23/26 have been fully considered but they are not persuasive and/or are moot in view of the new rejections.
With regards to 101, Applicant argues that the preamble now states “computer-implemented method” and therefore the claim is a practical application under 2a, prong two. Remarks, page 10-11. In response, Examiner respectfully disagrees. When viewing limitations individually or in combination, just adding “by a computer” does not make a claim eligible. See MPEP 2106.05f (apply it [abstract idea] on a computer]; MPEP 2106.05(a)(1)(I) Examples… not sufficient so show an improvement in computer-functionality: ii. Accelerating a process of analyzing audit log data when the increased speed comes solely from the capabilities of a general-purpose computer, FairWarning IP, LLC v. Iatric Sys., 839 F.3d 1089, 1095. The metrics here is 0023] as published – any process metrics or business KPIs; [0028] approval rate; [0031] bureau pass rate for financial lending.
With regards to step 2A, prong 1, Applicant appears to be arguing that based on MPEP 2111.01 that “limitations from the specification are not read into the claims,” and therefore, the claim is not directed to an abstract idea because it says “metrics” and “performance envelope.” Remarks, page 12. In response, Examiner respectfully disagrees. First, for 101 Patent Subject Matter Eligibility Analysis, MPEP 2106(II) states “It is essential that the broadest reasonable interpretation (BRI) of the claim be established prior to examining a claim for eligibility” and “In Mentor Graphics, the court interpreted the claims in light of the specification…” In addition, MPEP 2111.01 focuses on “reasonable” interpretation and also states “"[C]laims are not to be read in a vacuum, and limitations therein are to be interpreted in light of the specification in giving them their ‘broadest reasonable interpretation.” Here, Applicant does not have limitations narrowing the claims; the metrics in the examples throughout the specification are related to a credit/loan approval and various approval and financial offer rates in many embodiments, which is why “mitigating risk” is also a reasonable Abstract Idea grouping here. Applicant’s arguments are not persuasive.
Applicant then argues that the claim only “involves” mathematical concepts, and no mathematical concept is recited. Remarks, page 12-13. In response, Examiner respectfully disagrees. The claim includes calculating metrics, calculating the statistical properties of upper control limits and lower control limits, analyzing data points “outside” performance envelope which are the upper and lower thresholds of “upper control limit” and “lower control limit”.’
Applicant then argues the claim is not directed to abstract idea of “fundamental economic practices” of “mitigating risk” because the claim has “no reference to loan approval or monitoring any metric related to loan approval.” Remarks, page 14. In response, Examiner respectfully disagrees. The metrics in the examples throughout the specification are related to a credit/loan approval and various approval and financial offer rates in many embodiments (e.g. [0079-0081] as published, FIG. 5, which is why “mitigating risk” is also a reasonable Abstract Idea grouping here. See also MPEP 2106(II) “Evaluating eligibility based on the BRI also ensures that patent eligibility under 35 U.S.C. 101 does not depend simply on the draftsman’s art. Alice, 573 U.S. 208, 224, 110 USPQ2d at 1984, 1985.” Merely refraining from saying the metrics are related to “finance/offers/acceptance rates/bureau pass rates” in the claim, but having a specification where those are often the examples, does not make the claim eligible.
Applicant then argues the claim is not directed to abstract idea of “certain methods of organizing human activity” of “marketing” or “managing interactions between people (following rules or instructions”. Remarks, page 15. In response, Examiner respectfully disagrees. The metrics in the examples throughout the specification are related to an offer acceptance rate and approval rates, along with backlogs and statuses for alerts that need to be worked on and reviewed (See FIG. 4, [0084-0085]), which is why “marketing” (offers) and “managing interactions, following rules or instructions (for backlogs and reviewing work of people) is also a reasonable Abstract Idea grouping here. See also MPEP 2106(II) “Evaluating eligibility based on the BRI also ensures that patent eligibility under 35 U.S.C. 101 does not depend simply on the draftsman’s art. Alice, 573 U.S. 208, 224, 110 USPQ2d at 1984, 1985.” Merely refraining from saying the metrics are related to “finance/offers/acceptance rates/bureau pass rates” in the claim, but having a specification where those are often the examples, does not make the claim eligible.
With regards to 103, Applicant argues the new limitations. Remarks, page 17-19. The arguments are moot over the new rejections necessitated by the amendments.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Sohn et al., “Control Chart Application to Monitoring Microfinance Institution’s Mission Change for Women Borrowers Under Adverse Economy,” 2016, International Journal of Industrial Engineering, Vol. 23, No. 6, pages 382-398 – directed to using control charts with upper and lower control limits to investigate process for analyzing female borrower rates at MFI (Microfinance Institutions) (See page 386, 1st paragraph).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IVAN R GOLDBERG whose telephone number is (571)270-7949. The examiner can normally be reached 830AM - 430PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anita Coupe can be reached at 571-270-3614. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IVAN R GOLDBERG/Primary Examiner, Art Unit 3619
Prosecution Insights