Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Status of the Application
The following is a Final Office Action.
In response to Examiner's communication of 3/19/2026, Applicant responded on 6/22/2026. Amended claim 11. Cancelled claims 17, 18, 20. Added claims 21-23.
Claims 1-16, 19, 21-23 are pending in this application and have been examined.
Response to Amendment
Applicant's amendments to claims 11 are not sufficient to overcome the 35 USC 101 rejections set forth in the previous action.
Applicant's amendments to claims 11 are not sufficient to overcome the prior art rejections set forth in the previous action.
Response to Arguments – 35 USC § 101
Applicant’s arguments with respect to the rejections have been fully considered, but they are not persuasive.
Applicant submits, “…This characterization is in error. Claim 1 is not directed to a mental process or method of organizing human activity, but instead to a specific and technologically grounded machine learning architecture that addresses a technical problem in predictive modeling-namely, inconsistent feature contributions in trained models. The claim recites a structured sequence of operations involving a first tree-based machine learning model, identification of feature value ranges exhibiting inconsistent contributions, and the training of a second tree-based model specifically configured to resolve those inconsistencies through feature interaction analysis. This is not analogous to human reasoning or pen- and-paper activity; rather, it requires the operation of trained machine learning models, decision path analysis within those models, and iterative model refinement based on statistical contribution behavior-processes that cannot practically be performed mentally and that depend on computer-implemented model execution. Moreover, the claim integrates any alleged abstract concept into a practical application. The recited approach improves the functioning of machine learning systems themselves by introducing a mechanism to detect and resolve instability in feature contributions-an issue that directly affects model reliability, interpretability, and performance. Unlike generic data analysis, the claimed invention modifies the operation of predictive models by introducing a second-stage model trained on a subset of data defined by conflict conditions, thereby improving prediction quality and explainability. This constitutes a concrete technological improvement to machine learning systems, consistent with case law such as Enfish, which prohibits oversimplifying claims by ignoring their specific technical implementation…The Examiner's assertion that the claim elements are merely "data gathering" or "outputting" is likewise incorrect. The core of the claim lies not in input or output, but in the transformation and structured analysis of data through model-specific operations, including decision path evaluation and conditional retraining. Furthermore, the claim does not merely recite generic computer implementation; it specifies particular machine learning techniques (tree-based models), identifies technical conditions (inconsistent feature contribution within value ranges exceeding a threshold), and prescribes a specific solution (training a second model to identify a resolving feature). These are not routine or conventional computer functions, but rather a specialized algorithmic improvement. Under Step 2B, even if the claim were considered to involve an abstract idea, the ordered combination of elements clearly amounts to significantly more. The interaction between the first and second models, the targeted extraction of data subsets based on model-derived inconsistency, and the generation of a structured knowledge base capturing resolved feature interactions together constitute an inventive concept. The Examiner's reliance on generalized statements from the specification regarding conventional computing environments does not demonstrate that the claimed combination itself is conventional, as required by Berkheimer. Accordingly, claim 1 is directed to patent-eligible subject matter. The Examiner will note that, for expediting identification of allowable subject matter, claim 11 has been amended. The amendments follow the Examiner's suggestions during the interview, including clarifying the technical operation of the machine learning models and expressly tying the claimed inconsistency determination to decision paths of the trained tree-based machine learning model. In particular, the amendments further clarify that the inconsistency is determined across the one or more decision paths, thereby emphasizing that the claimed functionality is based on internal structural behavior of the trained model. As discussed during the interview, the Examiner indicated that additional specificity regarding how the model operates, including use of decision paths, would be helpful in advancing the application. See Interview Summary herein…As amended, claim 11 is not directed to a mental process. The claim requires training a tree-based machine learning model and performing analysis based on decision paths within that trained model. The determination that a feature exhibits inconsistency "across the one or more decision paths" reflects a computation performed on multiple model traversal paths, requiring aggregation and comparison of model-derived contributions across distinct execution structures. As discussed with the Examiner, such operations depend on the internal structure of a trained model and therefore cannot practically be performed in the human mind or with pen and paper. See Interview Summary herein. The Examiner's prior characterization of the claim as a mental process is therefore inconsistent with the explicitly recited requirement that inconsistency is determined based on trained model decision paths and evaluated across those paths. The claim is also not directed to organizing human activity. While the output of the system relates to risk-related predictions, the focus of the claim is on improving the functioning of machine learning models. Specifically, the claim addresses the technical problem of inconsistent feature contributions that vary depending on decision paths within a model. As noted during the interview, the Examiner acknowledged that reciting more detail regarding model operation and structure would better reflect the technical nature of the invention. See Interview Summary herein. The claimed solution- identifying inconsistency across decision paths and resolving that inconsistency using a second model- operates at the level of model internals and feature interaction behavior, not at the level of human decision-making or organization of activities. Even if the Examiner were to assert that the claim involves an abstract concept, the claim clearly integrates any such concept into a practical application. The claim recites a specific sequence involving a first trained model, decision path-based evaluation of feature contributions, identification of inconsistency across those paths within a defined value range, and the use of a second model to resolve that inconsistency. The addition of "across the one or more decision paths" underscores that the analysis is tied directly to model structure and execution. This constitutes a concrete technological solution that improves the reliability and interpretability of machine learning predictions. As discussed during the interview, providing such implementation-specific detail was identified as a path toward allowance. Under Step 2B, the claim recites significantly more than any alleged abstract idea. The ordered combination training a first model, evaluating feature contributions across multiple decision paths, identifying inconsistency within a value range, training a second model to resolve that inconsistency, and generating a knowledge base-is not routine or conventional. In particular, requiring inconsistency across decision paths introduces a structural and model-specific analysis not present in conventional approaches that rely solely on thresholds or aggregate scores. The Examiner has not demonstrated that this particular arrangement is well-understood, routine, or conventional. See Interview Summary herein. Accordingly, claim 11, as amended, is directed to patent-eligible subject matter. The Examiner will note that claim 19 recites subject matter parallel to independent claim 1, but in the form of computer-readable media containing executable program code. Accordingly, the arguments presented with respect to claim 1 apply equally here and are incorporated by reference. As with claim 1, claim 19 is not directed to a mental process or method of organizing human activity. The claim requires execution of program code that trains and applies tree-based machine learning models, evaluates decision paths within those models, detects inconsistency in feature contributions within a value range, and uses a second model to resolve that inconsistency. These operations are inherently tied to execution by a computer system and require manipulation of data structures and model internals that cannot practically be performed in the human mind. Further, the claim is directed to a technical improvement in machine learning systems, namely detection and resolution of inconsistent feature contributions across decision paths within trained models. The computer-readable media claim format reinforces that the invention is implemented through machine-executable instructions rather than mental reasoning. As discussed with respect to claim 1, the claimed invention operates on model internals and improves predictive modeling behavior, not human activity. Even if the presence of an abstract idea were assumed, claim 19 integrates any such idea into a practical application by reciting a specific sequence of machine learning operations executed by a computer system. Under Step 2B, the claim recites significantly more than any alleged abstract idea for the same reasons discussed for claim 1, including the non-conventional ordered combination of decision- path-based inconsistency detection and resolution via a second model. Accordingly, claim 19 is directed to patent-eligible subject matter. The Examiner will note that claims 21-23 have been added to further expedite identification of allowable subject matter. These claims directly reflect the Examiner's suggestions during the interview to introduce additional technical detail regarding the machine learning models, including how feature contributions are evaluated, how the second model is trained, and how model behavior is expressed in structured outputs. See Interview Summary herein. Specifically, claim 21 recites aggregation of feature contributions across decision paths, thereby explicitly tying the determination of inconsistency to internal model structure. Claim 22 recites training the second model on a subset of data corresponding to a value range identified through model analysis, thereby introducing a more precise and technical training mechanism. Claim 23 recites generation of a model-derived explanation reflecting resolution of inconsistency, reinforcing that the claimed output is not merely descriptive but is derived from the operation of the machine learning models. These additional limitations further demonstrate that the claimed invention is not directed to a mental process or method of organizing human activity. Rather, the claims require execution of trained machine learning models, analysis of decision paths, and evaluation of feature interaction behavior across those paths. As discussed during the interview, such model-specific operations were identified by the Examiner as important in demonstrating technical character and overcoming eligibility concerns. See Interview Summary herein. The added limitations also reinforce that any alleged abstract idea is integrated into a practical application. The claims now explicitly recite model-internal aggregation, targeted retraining on a data subset, and structured explanation derived from model behavior, all of which are concrete computational steps that improve machine learning model operation and interpretability. Further, under Step 2B, the recited elements-particularly aggregation across decision paths and subset-based retraining-reflect a non-conventional approach to handling inconsistent feature contributions in predictive modeling. These are not routine computer functions, but rather specialized techniques grounded in model structure and behavior. As suggested during the interview, the introduction of such technical specificity supports a conclusion of eligibility. See Interview Summary herein. Accordingly, claims 21-23 are directed to patent-eligible subject matter...” The Examiner respectfully disagrees.
Examiner respectfully notes, while Applicant and Examiner discussed this application and Applicant agreed to make amendments during the interview. Applicant did not make any amendments to Claim 1 and 19, ignoring Examiner’s recommendations from the interview. And, Applicant’s amendments to Claim 11 and the lack of amendments to Claim 1 and 19 are not sufficient to overcome the 35 USC 101 rejections. Examiner further notes, the newly added Claims 21-23 recite abstract limitations that further narrow the recited abstract ideas recited in Claim 1, under Step 2A Prong1 and do not recite any new additional elements beyond the abstract ideas under Step 2A Prong2, thus does not integrate the recited abstract ideas into a practical application under Step 2A Prong2 or amount to significantly more under Step 2B.
Further, unlike the Enfish, by Applicant’s own disclosure and admission, the claims indeed recite and direct to, …problem in predictive modeling-namely, inconsistent feature contributions in trained models…the models are used to predict project success rates…to solve complex problems with increasing accuracy and efficiency…model using historical project delivery data as input data to generate one or more risk-related predictions for project performance…evaluates the feature values in the new project data and matches them to the predefined ranges in the first and second feature contribution tables…improves predictive modeling behavior… introducing a mechanism to detect and resolve instability in feature contributions-an issue that directly affects model reliability, interpretability, and performance…identifies conditions (inconsistent feature contribution within value ranges exceeding a threshold), and prescribes a specific solution (training a second model to identify a resolving feature)… a specialized algorithmic improvement… identifying inconsistency across decision paths and resolving that inconsistency using a second model- operates at the level of model internals and feature interaction behavior…model is trained and used for risk analysis. The primary (or first) model focuses on initial risk predictions, while the secondary model addresses specific inconsistencies identified during the primary analysis…, which is a problem directed to, organizing human activity (i.e. human modeling and predicting project risks, i.e. hedging risk), and a mental process (i.e. human modeling and predicting project risks, i.e. hedging risk), as established in Step 2A Prong 1. This problem does not specifically arise in the realm of computer technology, but rather, this problem existed and was addressed long before the advent of computers. Thus, the claims do not recite a technical improvement to a technical problem. Additionally, pursuant to the broadest reasonable interpretation, as an ordered combination, each of the additional elements are computing elements recited at high level of generality implementing the abstract idea, and thus, are no more than applying the abstract idea with generic computer components performing extra solution activities, gathering data and outputting data, and generally linked to a technical environment, i.e. computer, machine learning. Therefore, as a whole, the additional elements do not integrate the abstract ideas into a practical application in Step 2A Prong 2 (apply it and general link) or amount to significantly more in Step 2B (apply it and WURC).
Even novel and newly discovered judicial exceptions are still exceptions, despite their novelty. July 2015 Update, p. 3; see SAP America Inc. v. Investpic, LLC, No. 2017-2081, slip op. at 2 (Fed Cir. May 15, 2018).
Simply reciting specific limitations that narrow the abstract idea does not make an abstract idea non-abstract. 79 Fed. Reg. 74631; buySAFE Inc. v. Google, Inc., 765 F.3d 1350, 1355 (2014); see SAP America at p. 12. As discussed in SAP America, no matter how much of an advance the claims recite, when “the advance lies entirely in the realm of abstract ideas, with no plausibly alleged innovation in the non-abstract application realm,” “[a]n advance of that nature is ineligible for patenting.” Id. at p. 3.
The Supreme Court has identified a number of concepts falling within the “certain methods of organizing human activity” grouping as abstract ideas. In particular, in Alice, the Court concluded that the use of a third party to mediate settlement risk is a ‘‘fundamental economic practice’’ and thus an abstract idea. 573 U.S. at 219–20, 110 USPQ2d at 1982. In addition, the Court in Alice described the concept of risk hedging identified as an abstract idea in Bilski as ‘‘a method of organizing human activity’’. Id. Previously, in Bilski, the Court concluded that hedging is a ‘‘fundamental economic practice’’ and therefore an abstract idea. 561 U.S. at 611–612, 95 USPQ2d at 1010.
Claims can recite a mental process even if they are claimed as being performed on a computer. The Supreme Court recognized this in Benson, determining that a mathematical algorithm for converting binary coded decimal to pure binary within a computer’s shift register was an abstract idea. The Court concluded that the algorithm could be performed purely mentally even though the claimed procedures “can be carried out in existing computers long in use, no new machinery being necessary.” 409 U.S at 67, 175 USPQ at 675. See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699 (concluding that concept of “anonymous loan shopping” recited in a computer system claim is an abstract idea because it could be “performed by humans without a computer”).
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015).
TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.
[E]xamples where the courts have found the additional elements to be mere instructions to apply an exception, because they do no more than merely invoke computers or machinery as a tool to perform an existing process include:
i. A commonplace business method or mathematical algorithm being applied on a general purpose computer, Alice Corp. Pty. Ltd. V. CLS Bank Int’l, 573 U.S. 208, 223, 110 USPQ2d 1976, 1983 (2014); Gottschalk v. Benson, 409 U.S. 63, 64, 175 USPQ 673, 674 (1972); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015);
ii. Generating a second menu from a first menu and sending the second menu to another location as performed by generic computer components, Apple, Inc. v. Ameranth, Inc., 842 F.3d 1229, 1243-44, 120 USPQ2d 1844, 1855-57 (Fed. Cir. 2016);
iii. A process for monitoring audit log data that is executed on a general-purpose computer where the increased speed in the process comes solely from the capabilities of the general-purpose computer, FairWarning IP, LLC v. Iatric Sys., 839 F.3d 1089, 1095, 120 USPQ2d 1293, 1296 (Fed. Cir. 2016);
iv. A method of using advertising as an exchange or currency being applied or implemented on the Internet, Ultramercial, Inc. v. Hulu, LLC, 772 F.3d 709, 715, 112 USPQ2d 1750, 1754 (Fed. Cir. 2014);
v. Requiring the use of software to tailor information and provide it to the user on a generic computer, Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1370-71, 115 USPQ2d 1636, 1642 (Fed. Cir. 2015);
[A] claimed process covering embodiments that can be performed on a computer, as well as embodiments that can be practiced verbally or with a telephone, cannot improve computer technology. See RecogniCorp, LLC v. Nintendo Co., 855 F.3d 1322, 1328, 122 USPQ2d 1377, 1381 (Fed. Cir. 2017) (process for encoding/decoding facial data using image codes assigned to particular facial features held ineligible because the process did not require a computer).
Examples that the courts have indicated may not be sufficient to show an improvement in computer-functionality:
i. Generating restaurant menus with functionally claimed features, Ameranth, 842 F.3d at 1245, 120 USPQ2d at 1857;
ii. Accelerating a process of analyzing audit log data when the increased speed comes solely from the capabilities of a general-purpose computer, FairWarning IP, LLC v. Iatric Sys., 839 F.3d 1089, 1095, 120 USPQ2d 1293, 1296 (Fed. Cir. 2016);
iii. Mere automation of manual processes, such as using a generic computer to process an application for financing a purchase, Credit Acceptance Corp. v. Westlake Services, 859 F.3d 1044, 1055, 123 USPQ2d 1100, 1108-09 (Fed. Cir. 2017) or speeding up a loan-application process by enabling borrowers to avoid physically going to or calling each lender and filling out a loan application, LendingTree, LLC v. Zillow, Inc., 656 Fed. App'x 991, 996-97 (Fed. Cir. 2016) (non-precedential);
iv. Recording, transmitting, and archiving digital images by use of conventional or generic technology in a nascent but well-known environment, without any assertion that the invention reflects an inventive solution to any problem presented by combining a camera and a cellular telephone, TLI Communications, 823 F.3d at 611-12, 118 USPQ2d at 1747;
vi. Instructions to display two sets of information on a computer display in a non-interfering manner, without any limitations specifying how to achieve the desired result, Interval Licensing LLC v. AOL, Inc., 896 F.3d 1335, 1344-45, 127 USPQ2d 1553, 1559-60 (Fed. Cir. 2018);
vii. Providing historical usage information to users while they are inputting data, in order to improve the quality and organization of information added to a database, because “an improvement to the information stored by a database is not equivalent to an improvement in the database’s functionality,” BSG Tech LLC v. Buyseasons, Inc., 899 F.3d 1281, 1287-88, 127 USPQ2d 1688, 1693-94 (Fed. Cir. 2018); and
viii. Arranging transactional information on a graphical user interface in a manner that assists traders in processing information more quickly, Trading Technologies v. IBG LLC, 921 F.3d 1084, 1093-94, 2019 USPQ2d 138290 (Fed. Cir. 2019).
Response to Arguments – Prior Art
Applicant’s arguments with respect to the rejections have been fully considered, but they are not persuasive.
Applicant submits, “…This combination is improper and fails to disclose or suggest the key limitations of claim 1. As an initial matter, McKenna, on current review, is directed to fraud detection using machine learning models that generate risk scores based on input features. While it may disclose training models and producing scores, it does not teach or suggest identifying a value range of a feature within which the feature's contribution exhibits inconsistency exceeding a threshold. The Examiner's reliance on "thresholds," "correlative scores," or "discrepancy" concepts in McKenna does not equate to the claimed requirement of analyzing feature contributions across decision paths and identifying instability within specific value ranges. These are distinct technical concepts: a threshold used to classify risk is fundamentally different from detecting inconsistent directional contribution of a feature within a defined value range. Similarly, McKenna, on current review, does not disclose or suggest training a second machine learning model specifically to identify a second feature that resolves the inconsistent contribution of the first feature within that range. The cited portions of McKenna describing a second model receiving outputs from a first model merely reflect a hierarchical or ensemble scoring approach, not a targeted retraining based on conflict conditions. The claimed invention requires isolating data corresponding to the inconsistent range and performing feature interaction analysis to resolve contribution instability-an approach not present in McKenna…The Examiner's reliance on Chang is also insufficient. Chang, on current review, relates generally to improving machine learning efficiency, such as by feature selection or reducing resource overhead. It does not address the problem of inconsistent feature contributions within value ranges, nor does it teach training a second model to resolve such inconsistencies through feature interaction. The generalized motivation to improve performance or scalability does not provide a rationale for modifying McKenna to include the specific conflict detection and resolution mechanism recited in claim 1. The proposed combination therefore relies on hindsight, using Applicant's disclosure as a blueprint. Additionally, the claimed "knowledge base table" comprising the first feature, second feature, value range, and their combination is not taught or suggested by the prior art. While McKenna, on current review, may disclose storing scores and reason codes, it does not disclose constructing a structured knowledge base that explicitly captures conditional feature interactions derived from resolving contribution inconsistencies. This is a distinct data structure tied to the specific methodology of the invention. In summary, neither McKenna alone nor in combination with Chang teaches or suggests the critical limitations of claim 1, including (i) identifying inconsistent feature contributions within a defined value range based on decision path analysis, (ii) training a second model on a subset of data defined by that inconsistency to identify a resolving feature, and (iii) generating a knowledge base capturing the interaction between features and value ranges. The Examiner's reasoning does not establish a sufficient motivation to combine the references to arrive at the claimed invention, and the rejection is therefore improper...The Examiner will note that, for expediting identification of allowable subject matter, claim 11 has been amended. These amendments follow the Examiner's suggestions discussed during the interview by clarifying the operation of the machine learning models and explicitly reciting that inconsistency in feature contributions is determined across the one or more decision paths of the trained first model. As discussed during the interview, the Examiner indicated that clarifying the role of the model and narrowing the claim to more specific technical behavior would better distinguish over the prior art. See Interview Summary herein. The amendments preserve the claim scope while more clearly defining the technical framework. As amended, the rejection over McKenna in view of Chang remains improper. McKenna, on current review, discloses machine learning-based risk or fraud prediction, including training models and generating scores based on features. However, McKenna does not teach or suggest identifying a value range of a feature based on decision paths within a trained tree-based model, wherein the feature's contributions exhibit inconsistency across the one or more decision paths. The amended limitation requires that inconsistency be derived from variation in feature contributions across different model traversal paths, which is fundamentally different from McKenna's use of thresholds, correlations, or discrepancies. As discussed during the interview, the Examiner acknowledged that clarifying how the claim operates at the model level could distinguish over the cited art. See Interview Summary herein. Further, McKenna, on current review, does not disclose or suggest determining inconsistency across decision paths as a basis for further model refinement. The cited portions referring to discrepancies or thresholds do not evaluate how a feature's contribution varies across different decision path contexts within a tree-based model. The amended claim now expressly requires that inconsistency arises from variation across decision paths, introducing a structural requirement not present in McKenna. Additionally, McKenna does not teach or suggest training a second machine learning model to identify a second feature that resolves the inconsistency observed across decision paths in the first model. The multiple models discussed in McKenna relate to layered scoring or output aggregation, not to resolving internal inconsistency in feature contributions identified in a first model. As clarified through the amendments, the second model is specifically used to resolve path-dependent inconsistency, further distinguishing the claim. Chang, on current review, is directed to optimizing machine learning performance, such as through feature removal or efficiency improvements. Chang does not address inconsistency in feature contributions across decision paths, nor does it suggest analyzing intra-model path variation. As noted in the interview discussion, generalized motivations to improve efficiency do not provide a basis for modifying McKenna to include the claimed decision-path-based inconsistency detection and resolution framework. See Interview Summary herein. The Examiner's rationale therefore relies on impermissible hindsight. Finally, neither McKenna nor Chang teaches or suggests generating a knowledge base table comprising the first feature, second feature, value range, and their combination in the context of resolving inconsistency across decision paths. While McKenna may disclose reason codes or scores, it does not produce a structured representation capturing feature interaction behavior derived from decision-path- based analysis. Accordingly, the cited references fail to teach or suggest the limitations of identifying feature contribution inconsistency across decision paths, training a second model to resolve that inconsistency, and generating a structured knowledge base reflecting that resolution. The rejection does not establish a prima facie case of obviousness, and withdrawal of the §103 rejection of claim 11 is respectfully requested. Claim 19 stands in parallel with claim 1 and recites the same substantive limitations in computer- readable media form. Accordingly, the arguments presented with respect to claim 1 are incorporated by reference and apply equally here. As discussed with respect to claim 1, the rejection over McKenna in view of Chang fails to teach or suggest key limitations of the claim. McKenna, on current review, does not disclose identifying a value range for a feature based on decision paths within a trained tree-based model, nor determining that feature contributions exhibit inconsistency within that range. Similarly, McKenna does not teach training a second model to identify a feature that resolves such inconsistency. Chang, on current review, fails to remedy these deficiencies, as it is directed to general optimization of machine learning models rather than analysis of feature contribution behavior across decision paths. The fact that claim 19 is directed to computer-readable media does not alter the substantive analysis, as the underlying operations performed by the program code remain unchanged. The cited references do not disclose or suggest the claimed sequence of operations, including decision-path-based identification of inconsistency and resolution of such inconsistency through a second model, nor do they disclose generating a knowledge base table capturing the resulting feature interaction. Accordingly, for the reasons set forth with respect to claim 1, the cited references fail to establish a prima facie case of obviousness as to claim 19, and withdrawal of the §103 rejection is respectfully requested. The Examiner will note that claims 21-23 have been added in view of the interview discussion to further clarify the technical aspects of the claimed invention and to distinguish over the cited references. As discussed during the interview, adding specificity regarding how the machine learning models operate-particularly with respect to decision paths, data subsets, and feature interaction-was identified as a path toward allowance. See Interview Summary herein. As added, claim 21 requires aggregation of feature contributions across decision paths and identification of inconsistency based on variation in magnitude or direction across those paths. McKenna, on current review, does not disclose or suggest evaluating feature contributions across multiple decision paths or identifying inconsistency in that manner. McKenna's use of thresholds or discrepancy detection does not involve aggregation or comparison of feature contributions across model traversal paths. Claim 22 further requires that the second model be trained on a subset of data corresponding to the identified value range. McKenna, on current review, does not disclose training a secondary model on a data subset defined by model-derived conditions such as value-range-specific inconsistency. Rather, McKenna generally applies models to full datasets or aggregated data, not targeted subsets derived from decision path analysis. Claim 23 recites associating feature combinations with a model-derived explanation indicating resolution of inconsistency. Neither McKenna nor Chang teaches or suggests generating explanations tied specifically to resolution of feature contribution inconsistency within a defined value range. While McKenna may output reason codes, such outputs do not reflect the claimed structured explanation derived from resolving model-internal inconsistency. Chang, on current review, similarly fails to disclose these limitations. Chang is directed to feature reduction or efficiency improvements and does not address decision-path-based contribution analysis, subset-based retraining, or explanation of resolved feature interactions. As discussed during the interview, generalized motivations such as improving efficiency do not provide a basis for modifying McKenna to arrive at these specific limitations. See Interview Summary herein. Accordingly, claims 21-23 add further technical distinctions over the cited references, including decision-path aggregation, subset-based retraining, and structured explanation of resolved feature interaction. These limitations are not taught or suggested by the prior art and would not have been obvious to one of ordinary skill. The § 103 rejection should therefore be withdrawn with respect to these claims….” The Examiner respectfully disagrees.
Examiner respectfully notes, while Applicant and Examiner discussed this application and Applicant agreed to make amendments during the interview. Applicant did not make any amendments to Claims 1 and 19, ignoring Examiner’s recommendations from the interview. As such, Examiner’s original interpretation remains unchanged for Claims 1 and 19. And, Applicant’s minor amendments to narrow Claim 11 slightly, do not alter Examiner’s original interpretation.
Examiner further respectfully notes, Applicant is requiring a particular narrow and tailored reading of the broadly recited claims that is not required by the broadest reasonable interpretation. Additionally, Applicant argues features and limitations that are not required by the broadly recited claims. And as such, under the broadest reasonable interpretation, the broadly recited claims do not distinguish from the cited references, McKenna in view of Chang. And Applicant’s arguments are not persuasive. Thus, Applicant’s amendments to Claim 11 and the lack of amendments to Claim 1 and 19 are not sufficient to overcome the prior art rejections. See office action below for Examiner’s detailed reasoning and rationale.
Further, for example, Applicant argues, “…the claimed "knowledge base table" comprising the first feature, second feature, value range, and their combination is not taught or suggested by the prior art… it does not disclose constructing a structured knowledge base that explicitly captures conditional feature interactions derived from resolving contribution inconsistencies.…neither McKenna nor Chang teaches or suggests generating a knowledge base table comprising the first feature, second feature, value range, and their combination in the context of resolving inconsistency across decision paths… generating a structured knowledge base reflecting that resolution… nor do they disclose generating a knowledge base table capturing the resulting feature interaction….”. However, the claims clearly recite: “…generating a knowledge base table comprising at least one of the first input feature, the second input feature, the first value range for the first input feature, and the combination of the first input feature and the second input feature...” Thus, Applicant’s arguments requiring features not required by the claims, are not persuasive. And, under the broadest reasonable interpretation, McKenna teaches:
generating a knowledge base table comprising at least one of the first input feature, the second input feature, the first value range for the first input feature, and the combination of the first input feature and the second input feature. (in at least [0103] a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7. [0129] The data sources may be compared using a tiered matching algorithm, including fuzzy matching. For example, the fraud detection computer system 120 may receive the normalized and transformed data and apply fuzzing matching algorithm to the data. When similarities are detected above a similarity threshold (e.g., 90 out of 100 potential match score), the fraud detection computer system 120 may cluster this data to identify individual dealer users. The individual dealer user may be assigned a new dealer user identifier to correspond with the combined data records. [0140] At 280, a second output may be determined. For example, the fraud detection computer system 120 may determine the second output as the score from the trained second ML model. In some examples, the second level score may be correlated with a user identifier associated with the dealer user device and stored with the scores data store 152. The application score may be retrieved in response to a search query of the data store. In some examples, the second level score may be provided in an application report or electronic message to a user device. [0215] The fraud detection computer system 120 may apply the output of the aggregation process 950 as input to the ML model. For example, multiple application scores may correspond with an input feature to determine a likelihood of fraud associated with the dealer user. In some examples, the number of application scores above a score threshold (e.g., high risk) may be compared to a risk threshold (e.g., more than half). The output of the ML model may determine a score corresponding with the dealer user and/or other information described herein. [0228] At 1160, the application score, reason code(s), and action(s) may be provided. For example, the fraud detection computer system 120 may provide the second score, the one or more reason codes, and the one or more actions to the dealer user device or a lender user device via a communication network. [0229] the second score may be provided as input to compute a first score for a future application associated with a borrower user device. In other examples, the second score may be provided in a dealer user report, as illustrated with FIG. 10. The report may be used to determine a likelihood that an application associated with the dealer user includes a likelihood of fraud.[0238] At 1270, the second score may be provided. For example, the fraud detection computer system 120 may provide the score to the lender device. The score may correspond with the lender user device and the dealer user device associated with the application data that provided the input to the ML model to generate the score(s).)
Additionally, respectfully, Applicant’s argument requires that the each of the features of supporting references are bodily incorporated into primary reference that teach and every element is individually taught by a single reference. However, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981). The test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one single or in all of the references. See id. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See id.; In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Claim Rejections – 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-16, 19, 21-23 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 1 (similarly 19) recites, “A method, comprising:
training a first tree-based … model using historical project delivery data as input data to generate one or more risk-related predictions for project performance;
determining a plurality of input features from the historical project delivery data;
identifying, based on one or more decision paths within the first tree-based … model, a first value range for a first input feature, among the plurality of input features, wherein a contribution of the first input feature to one or more of the risk-related predictions, within the first value range, is determined, and the contributions of the first input feature exhibit inconsistency that exceeds a defined threshold;
training a second tree-based … model using the historical project delivery data as input data to identify a second input feature, wherein a combination of the first input feature and the second input feature resolves the inconsistent contributions of the first input feature, within the first value range; and
generating a knowledge base table comprising at least one of the first input feature, the second input feature, the first value range for the first input feature, and the combination of the first input feature and the second input feature.”
Claim 11 recites, “…
training a first tree-based … model using historical project delivery data as input data to generate one or more risk-related predictions for project performance;
determining a plurality of input features from the historical project delivery data;
identifying, based on one or more decision paths of the trained first tree-based … model, a first value range for a first input feature, among the plurality of input features, wherein a contribution of the first input feature to one or more of the risk-related predictions, within the first value range, is determined, and the contributions of the first input feature exhibit inconsistency that exceeds a defined threshold across the one or more decision paths;
training a second tree-based … model using the historical project delivery data as input data to identify a second input feature, wherein a combination of the first input feature and the second input feature is configured to resolve the inconsistent contributions of the first input feature within the first value range; and
generating a knowledge base table comprising at least one of the first input feature, the second input feature, the first value range for the first input feature, and the combination of the first input feature and the second input feature…”
Analyzing under Step 2A, Prong 1:
The limitations regarding, …training a first tree-based … model using historical project delivery data as input data to generate one or more risk-related predictions for project performance; determining a plurality of input features from the historical project delivery data; identifying, based on one or more decision paths within the first tree-based … model, a first value range for a first input feature, among the plurality of input features, wherein a contribution of the first input feature to one or more of the risk-related predictions, within the first value range, is determined, and the contributions of the first input feature exhibit inconsistency that exceeds a defined threshold; training a second tree-based … model using the historical project delivery data as input data to identify a second input feature, wherein a combination of the first input feature and the second input feature resolves the inconsistent contributions of the first input feature, within the first value range; and generating a knowledge base table comprising at least one of the first input feature, the second input feature, the first value range for the first input feature, and the combination of the first input feature and the second input feature.… training a first tree-based … model using historical project delivery data as input data to generate one or more risk-related predictions for project performance; determining a plurality of input features from the historical project delivery data; identifying, based on one or more decision paths of the trained first tree-based … model, a first value range for a first input feature, among the plurality of input features, wherein a contribution of the first input feature to one or more of the risk-related predictions, within the first value range, is determined, and the contributions of the first input feature exhibit inconsistency that exceeds a defined threshold across the one or more decision paths; training a second tree-based … model using the historical project delivery data as input data to identify a second input feature, wherein a combination of the first input feature and the second input feature is configured to resolve the inconsistent contributions of the first input feature within the first value range; and generating a knowledge base table comprising at least one of the first input feature, the second input feature, the first value range for the first input feature, and the combination of the first input feature and the second input feature…, under the broadest reasonable interpretation, can include a human using their mind and using pen and paper to perform the above identified limitations, therefore, the claims are directed to a mental process.
Further, …training a first tree-based … model using historical project delivery data as input data to generate one or more risk-related predictions for project performance; determining a plurality of input features from the historical project delivery data; identifying, based on one or more decision paths within the first tree-based … model, a first value range for a first input feature, among the plurality of input features, wherein a contribution of the first input feature to one or more of the risk-related predictions, within the first value range, is determined, and the contributions of the first input feature exhibit inconsistency that exceeds a defined threshold; training a second tree-based … model using the historical project delivery data as input data to identify a second input feature, wherein a combination of the first input feature and the second input feature resolves the inconsistent contributions of the first input feature, within the first value range; and generating a knowledge base table comprising at least one of the first input feature, the second input feature, the first value range for the first input feature, and the combination of the first input feature and the second input feature.… training a first tree-based … model using historical project delivery data as input data to generate one or more risk-related predictions for project performance; determining a plurality of input features from the historical project delivery data; identifying, based on one or more decision paths of the trained first tree-based … model, a first value range for a first input feature, among the plurality of input features, wherein a contribution of the first input feature to one or more of the risk-related predictions, within the first value range, is determined, and the contributions of the first input feature exhibit inconsistency that exceeds a defined threshold across the one or more decision paths; training a second tree-based … model using the historical project delivery data as input data to identify a second input feature, wherein a combination of the first input feature and the second input feature is configured to resolve the inconsistent contributions of the first input feature within the first value range; and generating a knowledge base table comprising at least one of the first input feature, the second input feature, the first value range for the first input feature, and the combination of the first input feature and the second input feature…, are human modeling and predicting project risks, which are hedging risk, which are fundamental economic principles or practices, therefore the claims, are directed to certain methods of organizing human activities.
Accordingly, the claims are directed to a mental process, certain methods of organizing human activities, and thus, the claims are directed to an abstract idea under the first prong of Step 2A.
Analyzing under Step 2A, Prong 2:
This judicial exception is not integrated into a practical application under the second prong of Step 2A.
In particular, the claims recite the additional elements beyond the recited abstract idea identified under Step 2A, Prong 1, such as:
Claim 1, 11, 19: machine learning (ML), A system, comprising: one or more memories collectively containing one or more programs; and one or more processors, wherein the one or more processors are configured to, individually or collectively, One or more computer-readable media containing, in any combination, computer program code that, when executed by a computer system
, and pursuant to the broadest reasonable interpretation, as an ordered combination, each of the additional elements are computing elements recited at high level of generality implementing the abstract idea, and thus, are no more than applying the abstract idea with generic computer components.
Further, these additional elements generally link the abstract idea to a technical environment, namely the environment of a computer.
Additionally, with respect to, “…training…as input…”, “…extracting…”, “…generating…”, “…updating…”, “…outputting…”, these elements do not add a meaningful limitations to integrate the abstract idea into a practical application because they are extra-solution activity, pre and post solution activity - i.e. data gathering – “…training…as input…”, “…extracting…”, data output – “…generating…”, “…updating…”, “…outputting…”
Analyzing under Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under Step 2B.
As noted above, the aforementioned additional elements beyond the recited abstract idea are not sufficient to amount to significantly more than the recited abstract idea because, as an order combination, the additional elements are no more than mere instructions to implement the idea using generic computer components (i.e. apply it).
Additionally, as an order combination, the additional elements append the recited abstract idea to well-understood, routine, and conventional activities in the field as individually evinced by the applicant’s own disclosure, as required by the Berkheimer Memo, in at least:
[0015] Reference is made to embodiments presented in this disclosure. However, the scope of the present disclosure is not limited to specific described embodiments. Instead, any combination of the following features and elements, whether related to different embodiments or not, is contemplated to implement and practice contemplated embodiments. Furthermore, although embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment is not limiting of the scope of the present disclosure. Thus, the aspects, features, embodiments and advantages disclosed herein are merely illustrative and are not considered elements or limitations of the appended claims except where explicitly recited in a claim(s). Likewise, reference to "the invention" shall not be construed as a generalization of any inventive subject matter disclosed herein and shall not be considered to be an element or limitation of the appended claims except where explicitly recited in a claim(s).
[0016] Aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro- code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system."
[0017] Various aspects of the present disclosure are described by narrative text, flowcharts, block diagrams of computer systems and/or block diagrams of the machine logic included in computer program product (CPP) embodiments. With respect to any flowcharts, depending upon the technology involved, the operations can be performed in a different order than what is shown in a given flowchart. For example, again depending upon the technology involved, two operations shown in successive flowchart blocks may be performed in reverse order, as a single integrated step, concurrently, or in a manner at least partially overlapping in time.
[0018] A computer program product embodiment ("CPP embodiment" or "CPP") is a term used in the present disclosure to describe any set of one, or more, storage media (also called "mediums") collectively included in a set of one, or more, storage devices that collectively include machine readable code corresponding to instructions and/or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device that can retain and store instructions for use by a computer processor. Without limitation, the computer readable storage medium may be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these mediums include: diskette, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded device (such as punch cards or pits / lands formed in a major surface of a disc) or any suitable combination of the foregoing. A computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through a fiber optic cable, electrical signals communicated through a wire, and/or other transmission media. As will be understood by those of skill in the art, data is typically moved at some occasional points in time during normal operations of a storage device, such as during access, de- fragmentation or garbage collection, but this does not render the storage device as transitory because the data is not transitory while it is stored.
[0019] Figure 1 depicts an example computing environment 100 for the execution of at least some of the computer code involved in performing the inventive methods.
[0020]Computing environment 100 contains an example of an environment for the execution of at least some of the computer code involved in performing the inventive methods, such as Project Risk Prediction &Explanation Code 180. In addition to block 180, computing environment 100 includes, for example, computer 101,wide area network (WAN) 102,end user device (EUD) 103,remote server 104,public cloud 105, and private cloud 106. In this embodiment, computer 101 includes processor set 110 (including processing circuitry 120 and cache 121),communication fabric 111,volatile memory 112,persistent storage 113 (including operating system 122 and Project Risk Prediction &Explanation Code 180, as identified above), peripheral device set 114 (including user interface (UI) device set 123, storage 124, and Internet of Things (IoT) sensor set 125), and network module 115. Remote server 104 includes remote database 130. Public cloud 105 includes gateway 140,cloud orchestration module 141, host physical machine set 142, virtual machine set 143, and container set 144.
[0021]COMPUTER 101 may take the form of a desktop computer, laptop computer, tablet computer, smart phone, smart watch, or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or to be developed in the future that is capable of running a program, accessing a network or querying a database, such as remote database 130. As is well understood in the art of computer technology, and depending upon the technology, performance of a computer-implemented method may be distributed among multiple computers and/or between multiple locations. On the other hand, in this presentation of computing environment 100, detailed discussion is focused on a single computer, specifically computer 101, to keep the presentation as simple as possible. Computer 101 may be located in a cloud, even though it is not shown in a cloud in Figure 1. On the other hand, computer 101 is not required to be in a cloud except to any extent as may be affirmatively indicated.
[0022]PROCESSOR SET 110 includes one, or more, computer processors of any type now known or to be developed in the future. Processing circuitry 120 may be distributed over multiple packages, for example, multiple, coordinated integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and/or multiple processor cores. Cache 121 is memory that is located in the processor chip package(s) and is typically used for data or code that should be available for rapid access by the threads or cores running on processor set 110. Cache memories are typically organized into multiple levels depending upon relative proximity to the processing circuitry. Alternatively, some, or all, of the cache for the processor set may be located "off chip." In some computing environments, processor set 110 may be designed for working with qubits and performing quantum computing.
[0037] As depicted, the workflow 200 begins by using historical project delivery data 205 and provided machine learning (ML) algorithms 210 (e.g., basic Random Forest models or other default ML models) to train a primary (or first) ML model215. As used herein, the designation of this model215 as "primary" or "first" does not imply it is required or critical feature for the disclosed system. Instead, these terms are used to identify the model as an initial framework for conducting data analysis and generating risk-related predictions. The historical project delivery data 205 may include information related to financial performance, operational performance, resource utilization, past risk results, and other relevant project attributes. The primary (or first) ML model 215 is trained to generate risk-related outputs, such as a risk score (e.g., 97.5), a probability of achieving the target gross profit (GP) (e.g., 2.5 %), a classification of risk levels (e.g., low, medium, high), a predicted impact to plan (e.g., missing the financial goal by more than $500K), or other suitable risk-related metrics (depending on the application).
[0040] Upon identifying the ranges with conflicting (or inconsistent) contributions (e.g., 0.48-0.52 for "DWPRACTICE_LEAKACT"), the system proceeds to train a secondary ML model (e.g., a decision tree model), and identify whether there is a secondary feature that can resolve the current inconsistencies. In some embodiments, the secondary ML model is trained using only the data samples where the first feature (e.g., "DWPRACTICE_LEAKACT") falls within the conflicting range (e.g., 0.48-0.52). During training, the first feature itself is excluded from the dataset to isolate the first feature's contributions to the predictions. For example, if the full training dataset contains 10,000 samples, and only 500 samples are used to train the secondary model. The first feature itself (e.g., "DW_PRACTICE_LEAK ACT") is excluded as an input when training the secondary model. This ensures that the secondary model focuses on analyzing the contributions of other features without bias from the conflicting first feature. For example, the secondary model may evaluate whether a feature like "GLBL_BUY_GRP_KEY_LEAKACT" (representing the experience level with the current buying group) provides stable contributions within its specific value ranges (e.g., 0-0.3, or 0.3- 1) that explain or stabilize the inconsistencies in the first feature's conflicting range. As used herein, the designation of this model as "secondary" or "second" does not imply that the model is less important or subordinate to the primary ML model. Instead, the designation only indicates the sequence in which the model is trained and used for risk analysis. The primary (or first) model focuses on initial risk predictions, while the secondary model addresses specific inconsistencies identified during the primary analysis.
[0063] At block 505 of Figure 5A, a computing system trains a ML model (e.g., a random forest model) using historical project delivery data (e.g., 205 of Figure 2). The model is trained to predict risk-related outputs (e.g., risk scores, probabilities of achieving target GP, or classifications of risk levels).
[0097] While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
Furthermore, as an ordered combination, these elements amount to generic computer components receiving or transmitting data over a network, performing repetitive calculations, electronic record keeping, and storing and retrieving information in memory, which, as held by the courts, are well-understood, routine, and conventional. See MPEP 2106.05(d).
Moreover, the remaining elements of dependent claims do not transform the recited abstract idea into a patent eligible invention because these remaining elements merely recite further abstract limitations that provide nothing more than simply a narrowing of the abstract idea recited in the independent claims.
Looking at these limitations as an ordered combination adds nothing additional that is sufficient to amount to significantly more than the recited abstract idea because they simply provide instructions to use a generic arrangement of generic computer components to “apply” the recited abstract idea, perform insignificant extra-solution activity, and generally link the abstract idea to a technical environment. Thus, the elements of the claims, considered both individually and as an ordered combination, are not sufficient to ensure that the claim as a whole amounts to significantly more than the abstract idea itself. Since there are no limitations in these claims that transform the exception into a patent eligible application such that these claims amount to significantly more than the exception itself, claims 1-16, 19, 21-23 are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections – 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
Determining the scope and contents of the prior art.
Ascertaining the differences between the prior art and the claims at issue.
Resolving the level of ordinary skill in the pertinent art.
Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-16, 19, 21-23 is/are rejected under 35 U.S.C. 103 as being unpatentable by US Patent Publication to US20220318901A1 to McKenna et al., (hereinafter referred to as “McKenna”) in view of US Patent Publication to US20210374562A1 to Chang et al., (hereinafter referred to as “Chang”)
As per Claim 1, McKenna teaches: A method, comprising:
training a first tree-based machine learning (ML) model using historical … data as input data to generate one or more risk-related predictions for … performance; (in at least [0026] The system may implement multiple machine learning (ML) models. For example, a first ML model may receive application data from a borrower user device. In some examples, the application data may correspond with a segment and only application data corresponding with a particular segment may be provided to the first ML model. The system may correlate the application data to a training data set or, in the alternative, may apply the application data to a trained, first ML model. [0089] the ML model may be trained using historical application data by receiving a plurality of application data and determinations of whether fraud was discovered according to the risk profiles described herein. In some examples, the ML model may be trained using a weight applied to one or more input features (e.g., greater weight with a higher correlation of fraud in historical application data, etc.). Signals of fraud that may be common across the historical application data may be used to identify fraud in subsequent application data prior to the fraud occurring. [0114] The fraud detection computer system 120 may apply various ML models and embodiments of the disclosure. The models may correspond with linear or non-linear functions. For example, a ML model may comprise a supervised learning algorithm including a decision tree that accepts the one or more input features associated with the application to provide the score. [0122] the ML model may be trained using a training data set of historical application data. For example, the training data set may comprise a plurality of application data and determinations of whether fraud was discovered according to the risk profiles described herein. The ML model may be trained using historical data to determine one or more weights assigned to each of the input features according to a risk profile. In some examples, input features from the historical data that are common amongst a subset of applications may be identified as indicators of potential fraud according to the risk profile. The ML model may determine subsequent application data that identifies similar features as the training data set in order to determine a score for the application that identifies the similarities between the training data set and the application data.)
determining a plurality of input features from the historical … data; (in at least [0115] The ML model may comprise a Naive Bayes classifier that associates independent assumptions between the input features [0122] receiving the input features associated with the application data, the ML model may be trained using a training data set of historical application data. For example, the training data set may comprise a plurality of application data and determinations of whether fraud was discovered according to the risk profiles described herein. The ML model may be trained using historical data to determine one or more weights assigned to each of the input features according to a risk profile. In some examples, input features from the historical data that are common amongst a subset of applications may be identified as indicators of potential fraud according to the risk profile. [0117] The ML model may comprise a neural network classifier that measures the relationship between the categorical dependent variable (e.g., the likelihood of fraud) and independent variables (e.g., the application data) by estimating probabilities using multiple layers of processing elements that ascertain non-linear relationships and interactions between the independent variables and the dependent variable.)
identifying, based on one or more decision paths within the first tree-based ML model, a first value range for a first input feature, among the plurality of input features, wherein a contribution of the first input feature to one or more of the risk-related predictions, within the first value range, is determined, and the contributions of the first input feature exhibit inconsistency that exceeds a defined threshold; (in at least [0070] The profiling module 138 may also be configured to determine a correlative score for each of the plurality of applications. The correlative score (e.g., one or zero, or a value in a range of correlative scores, etc.) that links dealer user devices with lender user devices identified in the historical data. When the correlative score exceeds a threshold (e.g., 0.5, etc.), a correlation between a particular dealer user and lender user may be identified. One or more input features and the correlative score may be provided to a trained ML model and the output from the trained ML model may be scaled to a range of scores to determine the relative risk associated with the particular dealer user and lender user that requested the score. [0075] The discrepancy module 140 may also be configured to identify one or more risk indicators. For example, the discrepancy module 140 may review the application data and compare a subset of the application data with one or more risk profiles. When the similarities between the application data and the risk profile exceed a risk threshold, the discrepancy module 140 may be configured to identify an increased likelihood of fraud or risk with the application. [0092] The fraud detection computer system 120 may comprise a code module 144. The code module 144 may be configured to determine one or more reason codes for the application. For example, one or more features may influence the application score above a particular threshold to identify a potential risk or fraud. The features may correspond with a reason code that indicates an amount the feature can affect the application score and/or a reason that the feature affects the application score the way that it does. In some examples, the potential reason may be user defined (e.g., an administrator of the service may define reasons for particular features when seen in isolation or in combination with other features). In some examples, the one or more features may be grouped into categories (sometimes referred to as factor groups). Examples of factor groups include income, employment, identity, or the like. In each factor group, one or more features may be identified in order to be used to determine reason codes. [0114] a ML model may comprise a supervised learning algorithm including a decision tree that accepts the one or more input features associated with the application to provide the score. [0120] The ML model may further comprise an outlier detection method, which identifies significant deviations from the multivariate density distributions of a plurality of independent variables, even if such deviations have not previously been correlated with fraud in historical application data. [0123] At 230, a first output may be determined. For example, the fraud detection computer system 120 may determine the first output as the score from the trained ML model. In some examples, the score may be correlated with a user identifier associated with the borrower user device and stored with the scores data store 152. The score may be retrieved in response to a search query of the data store. In some examples, the score may be provided in an application report or electronic message to a user device. [0160] the one or more features may be selected based upon a factor group extraction subsystem. The factor group extraction subsystem may identify one or more features that are likely to point to one or more material misrepresentations in the loan application. [0162] Feature extraction system 320 may also output one or more features to discrepancy detection system 360. Discrepancy detection system 360 may include scorecard models and/or expert rules designed to flag inconsistent and/or out-of-pattern data values within the outputs of feature extraction system 320. [0190] The feature extraction layer 620 may transmit the one or more features to the scoring layer 630. The scoring layer 630 may compute a score. In some examples, the scoring layer 630 may summarize a risk associated with an application or dealer user. The scoring layer 630 may utilize a trained machine learning (ML) model (e.g., pattern recognition model, neural network, decision tree, clustering, etc.) to compute the score. Input to the scoring layer 630 may include one or more features and/or information received in the API input layer 610. In one illustrative example, the score may range from 1 (low risk) to 999 (high risk). However, it should be recognized that the score may be in a different form. [0222] At 1110, a plurality of application scores may be received from a first ML model. For example, the fraud detection computer system 120 may receive a plurality of application scores for a plurality of applications as output from a first trained ML model.)
training a second tree-based ML model using the historical … data as input data to identify a second input feature, wherein a combination of the first input feature and the second input feature resolves the inconsistent contributions of the first input feature, within the first value range; and (in at least [0027] The second ML model may receive the output from the first ML model as input. For example, upon receiving the output from the first ML model and/or application data from one or more borrower user devices, the system may apply the data to a second ML model to determine a second score. The first scores may be used as a training data set for the second ML model or, in some examples, may be provided to a previously-trained second ML model as input. The output from the second ML model may indicate signals of fraud and/or predict the type of fraud associated with a particular dealer user device and the corresponding output from the first ML model. [0028] individual application data from a first borrower user may identify a dealer user and individual application data from a second borrower user may identify the same dealer user. The plurality of application data may be provided as input to an ML model to determine a score associated with the same dealer user. In some examples, the same dealer user may correspond with different addresses or with different dealer identifiers. In some examples, the system may normalize or transform dealer user information so that the application data may be combined, including when the intention is to associate the application data with the same entity. In some examples, the combined data may be associated with a particular dealer user and the second score from the second ML model. [0096] when an application is determined to be high risk based upon a first level score or a second level score calculated for the application, an action may include additional review of the application, submitting the application to a second ML model, providing the application data to an administrator for manual review [0125] At 240, a second input may be received. For example, the fraud detection computer system 120 may receive one or more scores stored with the scores data store 152 and/or historical application data stored with the profiles data store 150. At least some of this information may correspond with output from the first ML model. The application data and/or scores may be received as input for a second ML model. In some examples, the fraud detection computer system 120 may also receive additional data, including third-party and/or consortium data corresponding with one or more dealer user devices. In some examples, the output, historical application data, third-party data, and/or consortium data may correspond with a particular dealer user. In some examples, the application score associated with the dealer user may identify the likelihood of fraud corresponding with applications submitted by the dealer user device to the lender user device. [0126] At 250, the data may be normalized and/or transformed. For example, the fraud detection computer system 120 may receive the dealer user name, address, or other information associated with the dealer user. The fraud detection computer system 120 may normalize this data by removing periods, spaces, or capitalization of characters to form a string of text associated with the dealer user. The fraud detection computer system 120 may transform this data by removing generic words including “a” or “the.” In some examples, the normalization and/or transformation process may add information as well, including adding the word “and” in place of an “&” (ampersand). The normalization and/or transformation of the dealer user information may help standardize different sources of dealer user information. This may include different applications provided by different borrower users. (i.e. combination of the first input feature and the second input feature resolves the inconsistent contributions of the first input feature, within the first value range) [0129] The data sources may be compared using a tiered matching algorithm, including fuzzy matching. For example, the fraud detection computer system 120 may receive the normalized and transformed data and apply fuzzing matching algorithm to the data. When similarities are detected above a similarity threshold (e.g., 90 out of 100 potential match score), the fraud detection computer system 120 may cluster this data to identify individual dealer users. The individual dealer user may be assigned a new dealer user identifier to correspond with the combined data records. [0131] At 270, one or more input features may be applied to a second trained ML model. For example, the fraud detection computer system 120 may determine a second score by selecting the application score from a plurality of application scores and determining the second score by applying at the input features to the trained second ML model. [0132] The fraud detection computer system 120 may apply various ML models and embodiments of the disclosure. For example, the trained second ML model may comprise a supervised learning algorithm including a decision tree that accepts the one or more input features associated with the application to provide the second level score. [0138] The second ML model may further comprise an outlier detection method, which identifies significant deviations from the multivariate density distributions of a plurality of independent variables, even if such deviations have not previously been correlated with fraud in historical application data. [0139] The second ML model may further comprise an ensemble modeling method, which combines scores from a plurality of the above ML methods or other methods to comprise an integrated score. [0140] At 280, a second output may be determined. For example, the fraud detection computer system 120 may determine the second output as the score from the trained second ML model. In some examples, the second level score may be correlated with a user identifier associated with the dealer user device and stored with the scores data store 152. The application score may be retrieved in response to a search query of the data store. In some examples, the second level score may be provided in an application report or electronic message to a user device. [0171] combination subsystem 444 for combining outputs from the different models used for the one or more features to generate a single score (referred to as an application score). For example, combination subsystem 444 may combine an output from a first model with an output from a second model. The output of combination subsystem 444 may then be output from the application scoring system (to a device associated with a lender) and/or to factor group extraction subsystem 445. [0172] Factor group extraction subsystem 445 may order features received from feature modification system 330 and/or discrepancy detection system 360. The ordering may be an order of how much the features affected the application score. In some examples, the ordering may be determined using a sensitivity analysis. For example, the factor group extraction subsystem 445 may remove one or more features to determine how much removal of the one or more features affect the application score. The features that change the application score more than other features may be ordered higher. [0186] The scoring service 530 may determine that the mismatch user identifiers may represent a potential straw borrower or identity risk. As another example, an employer identifier may be input that does not match a list of existing employers from a secretary of state data source. As another example, an email address may be input that is associated with prior fraud from a third-party data source. [0222] At 1110, a plurality of application scores may be received from a first ML model. For example, the fraud detection computer system 120 may receive a plurality of application scores for a plurality of applications as output from a first trained ML model. The plurality of applications may include first information associated with the plurality of borrower devices. The plurality of applications may be exchanged between the plurality of borrower devices and a dealer user device 112. [0223] At 1120, one or more input features may be generated. For example, the fraud detection computer system 120 may generate one or more input features in association with the plurality of applications. [0224] At 1130, a second score may be determined from a second ML model. For example, the fraud detection computer system 120 may determine a second score associate with the plurality of application scores. The second score may be selected from a plurality of second scores (e.g., ranging between one and 999, etc.). In some examples, determining the second score may comprise applying the one or more input features associated with the plurality of applications to the second ML model. [0225] the second score determined based upon a pattern recognition ML model. For example, the pattern recognition ML model may be a scorecard model that may be driven by historical application scores observed for the dealer user device. The process of updating the second score may occur after each application is associated with a first score so that the second score is immediately available for subsequent applications. [0226] At 1140, one or more reason codes may be determined. For example, the fraud detection computer system 120 may determine one or more reason codes for the plurality of applications based at least in part on the second score. [0232] At 1210, a plurality of application scores may be received. For example, the fraud detection computer system 120 may receive a plurality of first scores for a plurality of applications. The plurality of applications may include first information associated with a plurality of borrower user devices. The plurality of first scores may be output from the first trained ML model. [0234] At 1230, a correlative score may be determined. For example, the fraud detection computer system 120 may determine a correlative score for each of the plurality of applications. The correlative score for each of the plurality of applications may identify a link between a dealer user device and a lender user device. In some examples, applications corresponding with a dealer user and a lender user may be clustered or combined to identify application data that corresponds with the linked entities. [0237] At 1260, the output may be scaled. For example, the fraud detection computer system 120 may scale the output from the trained ML model to a range of scores to determine the relative risk associated with the particular dealer user and lender user that requested the score. In some examples, the output may identify a likelihood of fraud in an application submitted between a first dealer device and a first lender device. The likelihood of fraud may be codified as a second score. The likelihood of fraud may be identified by comparing application scores associated with historical application data to a score threshold.)
generating a knowledge base table comprising at least one of the first input feature, the second input feature, the first value range for the first input feature, and the combination of the first input feature and the second input feature. (in at least [0103] a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7. [0129] The data sources may be compared using a tiered matching algorithm, including fuzzy matching. For example, the fraud detection computer system 120 may receive the normalized and transformed data and apply fuzzing matching algorithm to the data. When similarities are detected above a similarity threshold (e.g., 90 out of 100 potential match score), the fraud detection computer system 120 may cluster this data to identify individual dealer users. The individual dealer user may be assigned a new dealer user identifier to correspond with the combined data records. [0140] At 280, a second output may be determined. For example, the fraud detection computer system 120 may determine the second output as the score from the trained second ML model. In some examples, the second level score may be correlated with a user identifier associated with the dealer user device and stored with the scores data store 152. The application score may be retrieved in response to a search query of the data store. In some examples, the second level score may be provided in an application report or electronic message to a user device. [0215] The fraud detection computer system 120 may apply the output of the aggregation process 950 as input to the ML model. For example, multiple application scores may correspond with an input feature to determine a likelihood of fraud associated with the dealer user. In some examples, the number of application scores above a score threshold (e.g., high risk) may be compared to a risk threshold (e.g., more than half). The output of the ML model may determine a score corresponding with the dealer user and/or other information described herein. [0228] At 1160, the application score, reason code(s), and action(s) may be provided. For example, the fraud detection computer system 120 may provide the second score, the one or more reason codes, and the one or more actions to the dealer user device or a lender user device via a communication network. [0229] the second score may be provided as input to compute a first score for a future application associated with a borrower user device. In other examples, the second score may be provided in a dealer user report, as illustrated with FIG. 10. The report may be used to determine a likelihood that an application associated with the dealer user includes a likelihood of fraud.[0238] At 1270, the second score may be provided. For example, the fraud detection computer system 120 may provide the score to the lender device. The score may correspond with the lender user device and the dealer user device associated with the application data that provided the input to the ML model to generate the score(s).)
Although implied, McKenna does not expressly disclose the following limitations, which however, are taught by Chang,
…project delivery…project… (in at least [0024] The entities include users that use online network 118 to establish and maintain professional connections, list work and community experience, endorse and/or recommend one another, search and apply for jobs, and/or perform other actions. The entities also, or instead, include companies, employers, and/or recruiters that use online network 118 to list jobs, search for potential candidates, provide business-related updates to users, advertise, and/or take other action. [0036] online network 118 includes a service that uses a machine learning model to generate a set of relevance scores representing the compatibility of a user with a set of jobs (or the compatibility of a set of users as candidates for a job). The service receives a scoring request in response to the user's job search (or a recruiter's search for candidates matching a job), the user accessing a job recommendation component in online network 118, and/or the user otherwise interacting with job-related functionality in online network 118. To improve the accuracy of the scores, the service inputs, into the machine learning model, features that reflect the latest activity by the user (or recruiter), such as features representing the user's searches, clicks, likes, dislikes, and/or other actions performed in the same user session as the user's interaction with the job-related functionality. In response to the inputted features, the machine learning model calculates the relevance scores between the user and jobs (or a set of candidates and a job) in a real-time or near-real-time basis (e.g., with a latency that is within a limit specified in a service level agreement (SLA) for the service). The service returns the relevance scores in a response to the scoring request, and online network 118 outputs, to the user, a ranking of jobs (or candidates) by descending relevance score as search results, recommendations, and/or other representations of the jobs (or candidates). [0037] Those skilled in the art will appreciate that online network 118 may perform on-demand scoring and/or ranking related to other types or combinations of entities. For example, online network 118 may receive one or more scoring requests for relevance scores between a user and content items created and/or shared by other users of the online network 118. These content items include, but are not limited to, posts, articles, comments, updates, and/or videos. The scoring request(s) may be generated in response to the user accessing a content feed in a homepage, landing page, and/or another part of online network 118. In response to the scoring request(s), a first set of machine learning models 110 is executed to generate recommendations of specific types of content (e.g., network updates, jobs, articles, courses, advertisements, posts, products, connections, products, etc.) for the user based on attributes of the user, recent activity by the user and/or similar users, and/or attributes of the content. The first set of machine learning models 110 output scores representing predictions of the user's level of interest in the content, and a subset of content items with high scores from the first set of machine learning models 110 is selected. Features associated with the user and the selected content items are inputted into one or more additional machine learning models 110, and the additional machine learning model(s) generate an additional set of scores representing predicted likelihoods of the user performing certain actions on (e.g., clicking, liking, commenting on, sharing, etc.) the selected content items. The content items are ranked by descending score, and one or more rules, filters, and/or exceptions are used to update the ranking (e.g., based on business requirements, member preferences, impression discounting, diversification of content, and/or other goals, strategies, or priorities). Finally, the updated ranking is displayed in a content feed for the user. Because the content items are scored and ranked in real-time or near-real-time, the content feed is able to reflect the latest content posted to online network 118 and/or user interactions with the content in online network 118.)
At the time the invention was filed, it would have been obvious for one of ordinary skill in the art to have modified the teachings of McKenna as taught by Chang, with a reasonable expectation of success if arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make this modification to the teachings of McKenna with the motivation of, …machine learning and/or analytics may be facilitated by mechanisms for improving resource consumption, latency, and/or scalability associated with executing machine learning models…To improve the resource overhead of the machine learning model, the feature removal framework calculates an importance score for each feature (or group of features) inputted into the baseline version and removes one or more features with importance scores that are lower than a threshold from the baseline version….by removing features from machine learning models in a way that meets resource overhead targets and/or minimizes the performance impact on the machine learning models, the disclosed embodiments improve computer systems, workflows, tools, and/or technologies related to monitoring, training, executing, and updating machine learning models…Rankings and/or associated insights based on the scores are then used to improve the quality of the candidates and/or recommendations of opportunities to the candidates, increase user activity with online network 118, and/or guide the decisions of the candidates and/or moderators involved in screening for or placing the opportunities (e.g., hiring managers, recruiters, human resources professionals, etc.)…simplified version 214 improved both the runtime performance and predictive performance of the machine learning model, when compared with baseline version 208…., as recited in Chang.
As per Claim 2, McKenna teaches: The method of claim 1, wherein identifying the second input feature comprises:
extracting one or more data samples from the historical … data, wherein the first input feature within the one or more data samples comprises values falling within the identified first value range; (in at least [0122] Prior to receiving the input features associated with the application data, the ML model may be trained using a training data set of historical application data. For example, the training data set may comprise a plurality of application data and determinations of whether fraud was discovered according to the risk profiles described herein. The ML model may be trained using historical data to determine one or more weights assigned to each of the input features according to a risk profile. In some examples, input features from the historical data that are common amongst a subset of applications may be identified as indicators of potential fraud according to the risk profile. The ML model may determine subsequent application data that identifies similar features as the training data set in order to determine a score for the application that identifies the similarities between the training data set and the application data. [0165] application scoring system 340 may further include a factor group extraction subsystem. The factor group extraction subsystem may order the one or more features used as input to application scoring system 340. For instance, the ordering may be based upon an amount that each feature affected a score. The factor group extraction subsystem may also group the one or more features into one or more groups. [0209] A plurality of applications may be received by the fraud detection computer system 120 from a plurality of dealer user devices, including application data associated with a first user 910A and application data associated with a second user 910B (in combination, referred to as “users 910”). In this illustration, the dealer user names may be similar in each application. For example, the first user 910A may correspond with “Cars of Acme” dealer name and the second user 910B may correspond with “Acme Cars” dealer name.)
training the second tree-based ML model using the one or more data samples as input data; and (in at least [0129] The data sources may be compared using a tiered matching algorithm, including fuzzy matching. For example, the fraud detection computer system 120 may receive the normalized and transformed data and apply fuzzing matching algorithm to the data. When similarities are detected above a similarity threshold (e.g., 90 out of 100 potential match score), the fraud detection computer system 120 may cluster this data to identify individual dealer users. The individual dealer user may be assigned a new dealer user identifier to correspond with the combined data records. [0131] At 270, one or more input features may be applied to a second trained ML model. For example, the fraud detection computer system 120 may determine a second score by selecting the application score from a plurality of application scores and determining the second score by applying at the input features to the trained second ML model. [0132] The fraud detection computer system 120 may apply various ML models and embodiments of the disclosure. For example, the trained second ML model may comprise a supervised learning algorithm including a decision tree that accepts the one or more input features associated with the application to provide the second level score.)
identifying, based on one or more decision paths within the second tree-based ML model with the first input feature …, a second value range for the second input feature, wherein a contribution of the second input feature to one or more of the risk-related predictions, within the second value range, is determined, and the contributions of the second input feature exhibit stability within a defined threshold. (in at least [0132] The fraud detection computer system 120 may apply various ML models and embodiments of the disclosure. For example, the trained second ML model may comprise a supervised learning algorithm including a decision tree that accepts the one or more input features associated with the application to provide the second level score. [0138] The second ML model may further comprise an outlier detection method, which identifies significant deviations from the multivariate density distributions of a plurality of independent variables, even if such deviations have not previously been correlated with fraud in historical application data. [0209] A plurality of applications may be received by the fraud detection computer system 120 from a plurality of dealer user devices, including application data associated with a first user 910A and application data associated with a second user 910B (in combination, referred to as “users 910”). In this illustration, the dealer user names may be similar in each application. For example, the first user 910A may correspond with “Cars of Acme” dealer name and the second user 910B may correspond with “Acme Cars” dealer name. [0211] The fraud detection computer system 120 may implement a transformation process 930 using the output of the normalization process 920 as input. For example, the fraud detection computer system 120 may transform this data by removing generic words including “a” or “the.” In another example, the transformation process may add the word “and” in place of an “&” (ampersand). [0224] At 1130, a second score may be determined from a second ML model. For example, the fraud detection computer system 120 may determine a second score associate with the plurality of application scores. The second score may be selected from a plurality of second scores (e.g., ranging between one and 999, etc.). In some examples, determining the second score may comprise applying the one or more input features associated with the plurality of applications to the second ML model. [0225] the second score determined based upon a pattern recognition ML model. For example, the pattern recognition ML model may be a scorecard model that may be driven by historical application scores observed for the dealer user device. The process of updating the second score may occur after each application is associated with a first score so that the second score is immediately available for subsequent applications.)
Although implied, McKenna does not expressly disclose the following limitations, which however, are taught by Chang,
…project delivery… (in at least [0024] The entities include users that use online network 118 to establish and maintain professional connections, list work and community experience, endorse and/or recommend one another, search and apply for jobs, and/or perform other actions. The entities also, or instead, include companies, employers, and/or recruiters that use online network 118 to list jobs, search for potential candidates, provide business-related updates to users, advertise, and/or take other action. [0036] online network 118 includes a service that uses a machine learning model to generate a set of relevance scores representing the compatibility of a user with a set of jobs (or the compatibility of a set of users as candidates for a job). The service receives a scoring request in response to the user's job search (or a recruiter's search for candidates matching a job), the user accessing a job recommendation component in online network 118, and/or the user otherwise interacting with job-related functionality in online network 118. To improve the accuracy of the scores, the service inputs, into the machine learning model, features that reflect the latest activity by the user (or recruiter), such as features representing the user's searches, clicks, likes, dislikes, and/or other actions performed in the same user session as the user's interaction with the job-related functionality. In response to the inputted features, the machine learning model calculates the relevance scores between the user and jobs (or a set of candidates and a job) in a real-time or near-real-time basis (e.g., with a latency that is within a limit specified in a service level agreement (SLA) for the service). The service returns the relevance scores in a response to the scoring request, and online network 118 outputs, to the user, a ranking of jobs (or candidates) by descending relevance score as search results, recommendations, and/or other representations of the jobs (or candidates). [0037] Those skilled in the art will appreciate that online network 118 may perform on-demand scoring and/or ranking related to other types or combinations of entities. For example, online network 118 may receive one or more scoring requests for relevance scores between a user and content items created and/or shared by other users of the online network 118. These content items include, but are not limited to, posts, articles, comments, updates, and/or videos. The scoring request(s) may be generated in response to the user accessing a content feed in a homepage, landing page, and/or another part of online network 118. In response to the scoring request(s), a first set of machine learning models 110 is executed to generate recommendations of specific types of content (e.g., network updates, jobs, articles, courses, advertisements, posts, products, connections, products, etc.) for the user based on attributes of the user, recent activity by the user and/or similar users, and/or attributes of the content. The first set of machine learning models 110 output scores representing predictions of the user's level of interest in the content, and a subset of content items with high scores from the first set of machine learning models 110 is selected. Features associated with the user and the selected content items are inputted into one or more additional machine learning models 110, and the additional machine learning model(s) generate an additional set of scores representing predicted likelihoods of the user performing certain actions on (e.g., clicking, liking, commenting on, sharing, etc.) the selected content items. The content items are ranked by descending score, and one or more rules, filters, and/or exceptions are used to update the ranking (e.g., based on business requirements, member preferences, impression discounting, diversification of content, and/or other goals, strategies, or priorities). Finally, the updated ranking is displayed in a content feed for the user. Because the content items are scored and ranked in real-time or near-real-time, the content feed is able to reflect the latest content posted to online network 118 and/or user interactions with the content in online network 118.)
… excluded…(in at least [0053] simplification apparatus 202 sets feature removal threshold 232 to an estimated number of features to be removed from baseline version 208 to lower baseline resource overhead 216 to target resource overhead 218. For example, simplification apparatus 202 retrieves, from model repository 236 and/or another data store, historical resource overheads of different versions of the machine learning model (e.g., different baseline versions that have been updated over time).)
The reason and rationale to combine McKenna and Chang is the same as recited above.
As per Claim 3, McKenna teaches: The method of claim 2,
wherein the knowledge base table associates the first value range for the first input feature and the second value range for the second input feature with the combination. (in at least [0103] As a sample illustration, a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7. [0216] FIG. 10 illustrates a report for indicating a score according to an embodiment of the disclosure. In illustration 1000, the report may include a score for a dealer user associated with a lender user. The report may comprise aggregated application data and/or application scores by dealer identifier. The report may also comprise dealer information associated with a loan application, including dealer ID, dealer name, location associated with the dealer, a phone number for the dealer, a volume of applications for the domain in a particular amount of time. [0217] The dealer report further includes a second level score (e.g., “997”), which may be calculated as described herein. The application report may further include a risk level (e.g., “high”). The risk level may be determined by comparing the second level score to one or more thresholds, each threshold associated with a different level of risk (e.g., high, medium, and low). [0218] The application report further includes reason codes for the dealer. A reason code may indicate information associated with a feature that may contribute to the application score. The reason codes may be determined based upon features determined for the dealer that contribute most to the application score. The reason codes may be filtered and provided according to features that cause the application score to increase the greatest amount when compared to other reason codes. For example, a fraud rate may be identified with applications originating from the particular dealer user. When the fraud rate is higher than a threshold value (e.g., a national average, or an average for similar dealer users, etc.), the fraud rate may cause the application score to increase at a greater rate. The reason code associated with the fraud rate may be identified on the application report as well.)
As per Claim 4, McKenna teaches: The method of claim 2,
wherein the knowledge base table comprises a reason code associated with the combination of the first input feature and the second input feature, and a text-based explanation describing how the combination, within the first value range for the first input feature and the second value range for the second input feature, influences the risk-related predictions. (in at least [0103] As a sample illustration, a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7. [0216] FIG. 10 illustrates a report for indicating a score according to an embodiment of the disclosure. In illustration 1000, the report may include a score for a dealer user associated with a lender user. The report may comprise aggregated application data and/or application scores by dealer identifier. The report may also comprise dealer information associated with a loan application, including dealer ID, dealer name, location associated with the dealer, a phone number for the dealer, a volume of applications for the domain in a particular amount of time. [0217] The dealer report further includes a second level score (e.g., “997”), which may be calculated as described herein. The application report may further include a risk level (e.g., “high”). The risk level may be determined by comparing the second level score to one or more thresholds, each threshold associated with a different level of risk (e.g., high, medium, and low). [0218] The application report further includes reason codes for the dealer. A reason code may indicate information associated with a feature that may contribute to the application score. The reason codes may be determined based upon features determined for the dealer that contribute most to the application score. The reason codes may be filtered and provided according to features that cause the application score to increase the greatest amount when compared to other reason codes. For example, a fraud rate may be identified with applications originating from the particular dealer user. When the fraud rate is higher than a threshold value (e.g., a national average, or an average for similar dealer users, etc.), the fraud rate may cause the application score to increase at a greater rate. The reason code associated with the fraud rate may be identified on the application report as well.)
As per Claim 5, McKenna teaches: The method of claim 1,
wherein the contribution of the first input feature to one or more of the risk-related predictions comprises at least one of a possibility that the first input feature affects the risk-related prediction, a magnitude of an effect of the first input feature on the risk-related prediction, and a direction of the effect of the first input feature on the risk-related prediction, and wherein the direction is either positive or negative. (in at least [0103] As a sample illustration, a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7. [0212] The fraud detection computer system 120 may implement a matching process 940 using the output of the transformation process 930 as input. For example, the fraud detection computer system 120 may match (e.g., tiered matching process, or fuzzing matching, etc.) one or more users by one or more fields of data. For example, the fields may comprise a dealer name, street address, ZIP Code, state, and phone number. The matching process may receive a first dealer name and compare the first dealer name with all other users 910. The users 910 corresponding with matched data fields may be grouped to form a plurality of clusters. In some examples, multiple data fields may be matched to form clusters of data that are more closely related. [0213] The fraud detection computer system 120 may implement a combination process 950 using the output of the matching process 940 as input. For example, the fraud detection computer system 120 may measure a field distance between fields of the plurality of clusters. When the similarities are detected above a similarity threshold (e.g., 90 out of 100 potential match score), the users may be aggregated to identify a single dealer user (e.g., a dealer user associated with “Acme”). The single dealer user may be assigned a new dealer user identifier to correspond with the combined data records. [0215] The fraud detection computer system 120 may apply the output of the aggregation process 950 as input to the ML model. For example, multiple application scores may correspond with an input feature to determine a likelihood of fraud associated with the dealer user. In some examples, the number of application scores above a score threshold (e.g., high risk) may be compared to a risk threshold (e.g., more than half). The output of the ML model may determine a score corresponding with the dealer user and/or other information described herein. [0216] FIG. 10 illustrates a report for indicating a score according to an embodiment of the disclosure. In illustration 1000, the report may include a score for a dealer user associated with a lender user. The report may comprise aggregated application data and/or application scores by dealer identifier. The report may also comprise dealer information associated with a loan application, including dealer ID, dealer name, location associated with the dealer, a phone number for the dealer, a volume of applications for the domain in a particular amount of time. [0217] The dealer report further includes a second level score (e.g., “997”), which may be calculated as described herein. The application report may further include a risk level (e.g., “high”). The risk level may be determined by comparing the second level score to one or more thresholds, each threshold associated with a different level of risk (e.g., high, medium, and low). [0218] The application report further includes reason codes for the dealer. A reason code may indicate information associated with a feature that may contribute to the application score. The reason codes may be determined based upon features determined for the dealer that contribute most to the application score. The reason codes may be filtered and provided according to features that cause the application score to increase the greatest amount when compared to other reason codes. For example, a fraud rate may be identified with applications originating from the particular dealer user. When the fraud rate is higher than a threshold value (e.g., a national average, or an average for similar dealer users, etc.), the fraud rate may cause the application score to increase at a greater rate. The reason code associated with the fraud rate may be identified on the application report as well.)
As per Claim 6, McKenna teaches: The method of claim 4, further comprising:
receiving real-time … data as input by the first tree-based ML model to generate a real-time risk-related prediction; (in at least [0209] A plurality of applications may be received by the fraud detection computer system 120 from a plurality of dealer user devices, including application data associated with a first user 910A and application data associated with a second user 910B (in combination, referred to as “users 910”). In this illustration, the dealer user names may be similar in each application. For example, the first user 910A may correspond with “Cars of Acme” dealer name and the second user 910B may correspond with “Acme Cars” dealer name. [0215] The fraud detection computer system 120 may apply the output of the aggregation process 950 as input to the ML model. For example, multiple application scores may correspond with an input feature to determine a likelihood of fraud associated with the dealer user. In some examples, the number of application scores above a score threshold (e.g., high risk) may be compared to a risk threshold (e.g., more than half). The output of the ML model may determine a score corresponding with the dealer user and/or other information described herein.)
determining that a value of the first input feature, extracted from the real-time … data, falls within the identified first value range for the first input feature; (in at least [0190] The feature extraction layer 620 may transmit the one or more features to the scoring layer 630. The scoring layer 630 may compute a score. In some examples, the scoring layer 630 may summarize a risk associated with an application or dealer user. The scoring layer 630 may utilize a trained machine learning (ML) model (e.g., pattern recognition model, neural network, decision tree, clustering, etc.) to compute the score. Input to the scoring layer 630 may include one or more features and/or information received in the API input layer 610. In one illustrative example, the score may range from 1 (low risk) to 999 (high risk). However, it should be recognized that the score may be in a different form.)
determining that a value of the second input feature, extracted from the real-time …. data, falls within the identified second value range for the second input feature; (in at least [0100] the fraud detection computer system 120 may also be configured to analyze and determine second level scores (e.g., application scores associated with a dealer user and/or second ML model). Any of the functions described herein may be implemented in determining the second level score. For example, the application engine 136 may be configured to receive application data. The profiling module 138 may be configured to determine a segment associated with the application data and generate one or more input features for providing to a first ML model. The fraud scoring engine 142 may be configured to determine a score for a plurality of applications, corresponding with first level scores. The fraud detection computer system 120 may combine (e.g., aggregate, etc.) these application scores based on common information, including a common dealer user. The combined application scores for a common dealer user may correspond with a second level score. The first level score and/or the second level score, and corresponding user information for each of the one or more scores, may be stored with the scores data store 152. [0103] a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7. [0104] A second level score and/or codes 162 may also be computed. The second level score and/or codes 162 may be based at least in part on scores computed for a dealer user. The second level score, with or without the first level scores, may be transmitted to the lender user, for example, for the lender user to determine the level of diligence required when reviewing the application and/or determination of providing funding to the borrower user. [0209] A plurality of applications may be received by the fraud detection computer system 120 from a plurality of dealer user devices, including application data associated with a first user 910A and application data associated with a second user 910B (in combination, referred to as “users 910”). In this illustration, the dealer user names may be similar in each application. For example, the first user 910A may correspond with “Cars of Acme” dealer name and the second user 910B may correspond with “Acme Cars” dealer name. [0211] The fraud detection computer system 120 may implement a transformation process 930 using the output of the normalization process 920 as input. For example, the fraud detection computer system 120 may transform this data by removing generic words including “a” or “the.” In another example, the transformation process may add the word “and” in place of an “&” (ampersand). [0224] At 1130, a second score may be determined from a second ML model. For example, the fraud detection computer system 120 may determine a second score associate with the plurality of application scores. The second score may be selected from a plurality of second scores (e.g., ranging between one and 999, etc.). In some examples, determining the second score may comprise applying the one or more input features associated with the plurality of applications to the second ML model. [0225] the second score determined based upon a pattern recognition ML model. For example, the pattern recognition ML model may be a scorecard model that may be driven by historical application scores observed for the dealer user device. The process of updating the second score may occur after each application is associated with a first score so that the second score is immediately available for subsequent applications.)
retrieving, from the knowledge base table, the reason code associated with the combination; and (in at least [0093] The code module 144 may also be configured to generate a reason code based on the input features that appear to be most prominent on the application score. For example, an application score risk may be a first type of feature, a fraud rate risk may be a second type of feature, and a volume of risk may be a third type of feature. In some examples, the input features may correspond with a plurality of applications and/or aggregated application data associated with a dealer user device. In some examples, an input feature may correspond with a risk signal that may determine a likelihood of fraud associated with a portion of the application data for the dealer user device. The input feature, in some examples, may be predetermined prior to providing the application data to a trained ML model. [0094] The code module 144 may also be configured to provide a generated reason code to a user device. For example, the fraud detection computer system 120 may determine the reason code according to the determined application score and/or information received by a scoring service. [0103] a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7.)
outputting the reason code and the text-based explanation describing how the combination influences the real-time risk-related prediction. (in at least [0124] the output may also comprise one or more reason codes and/or one or more suggested actions corresponding with the score or the reason codes. For example, the fraud detection computer system 120 may determine the input features that closely corresponded to the risk profiles implemented by the trained ML model. Each of the input features may correspond with one or more reason codes and/or one or more suggested actions. In some examples, the fraud detection computer system 120 may select a first input feature and provide it as a search term to the code/action data store 154. The data store may return the one or more reason codes that correspond with the first input feature as well as one or more actions that correspond with the first input feature. The returned data from the code/action data store 154 may be added, with the score, to the application report or electronic message transmitted to the user device. [0130] At 260, the data may be combined for the particular user. For example, the fraud detection computer system 120 may correlate a dealer user identifier for any data entry corresponding with the particular dealer user. The combination of the dealer user data may identify any application data corresponding with a particular dealer user, including current or historical data. Duplicate data may be removed in the combination process. This may help identify any application data submitted in association with the particular dealer user to any lender user for approval of an application on behalf of any borrower user. [0141] the second output may also correspond with one or more reason codes and/or one or more suggested actions corresponding with the second level score. For example, the fraud detection computer system 120 may determine the input features that closely corresponded to the risk profiles implemented by the trained second ML model. Each of the input features may correspond with one or more reason codes and/or one or more suggested actions. In some examples, the fraud detection computer system 120 may select a second input feature and provide it as a search term to the code/action data store 154. The data store may return the one or more reason codes that correspond with the second input feature as well as one or more actions that correspond with the second input feature. The returned data from the code/action data store 154 may be added, with the application score, to the application report or electronic message transmitted to the user device.)
Although implied, McKenna does not expressly disclose the following limitations, which however, are taught by Chang,
…project… (in at least [0024] The entities include users that use online network 118 to establish and maintain professional connections, list work and community experience, endorse and/or recommend one another, search and apply for jobs, and/or perform other actions. The entities also, or instead, include companies, employers, and/or recruiters that use online network 118 to list jobs, search for potential candidates, provide business-related updates to users, advertise, and/or take other action. [0036] online network 118 includes a service that uses a machine learning model to generate a set of relevance scores representing the compatibility of a user with a set of jobs (or the compatibility of a set of users as candidates for a job). The service receives a scoring request in response to the user's job search (or a recruiter's search for candidates matching a job), the user accessing a job recommendation component in online network 118, and/or the user otherwise interacting with job-related functionality in online network 118. To improve the accuracy of the scores, the service inputs, into the machine learning model, features that reflect the latest activity by the user (or recruiter), such as features representing the user's searches, clicks, likes, dislikes, and/or other actions performed in the same user session as the user's interaction with the job-related functionality. In response to the inputted features, the machine learning model calculates the relevance scores between the user and jobs (or a set of candidates and a job) in a real-time or near-real-time basis (e.g., with a latency that is within a limit specified in a service level agreement (SLA) for the service). The service returns the relevance scores in a response to the scoring request, and online network 118 outputs, to the user, a ranking of jobs (or candidates) by descending relevance score as search results, recommendations, and/or other representations of the jobs (or candidates). [0037] Those skilled in the art will appreciate that online network 118 may perform on-demand scoring and/or ranking related to other types or combinations of entities. For example, online network 118 may receive one or more scoring requests for relevance scores between a user and content items created and/or shared by other users of the online network 118. These content items include, but are not limited to, posts, articles, comments, updates, and/or videos. The scoring request(s) may be generated in response to the user accessing a content feed in a homepage, landing page, and/or another part of online network 118. In response to the scoring request(s), a first set of machine learning models 110 is executed to generate recommendations of specific types of content (e.g., network updates, jobs, articles, courses, advertisements, posts, products, connections, products, etc.) for the user based on attributes of the user, recent activity by the user and/or similar users, and/or attributes of the content. The first set of machine learning models 110 output scores representing predictions of the user's level of interest in the content, and a subset of content items with high scores from the first set of machine learning models 110 is selected. Features associated with the user and the selected content items are inputted into one or more additional machine learning models 110, and the additional machine learning model(s) generate an additional set of scores representing predicted likelihoods of the user performing certain actions on (e.g., clicking, liking, commenting on, sharing, etc.) the selected content items. The content items are ranked by descending score, and one or more rules, filters, and/or exceptions are used to update the ranking (e.g., based on business requirements, member preferences, impression discounting, diversification of content, and/or other goals, strategies, or priorities). Finally, the updated ranking is displayed in a content feed for the user. Because the content items are scored and ranked in real-time or near-real-time, the content feed is able to reflect the latest content posted to online network 118 and/or user interactions with the content in online network 118.)
The reason and rationale to combine McKenna and Chang is the same as recited above.
As per Claim 7, McKenna teaches: The method of claim 1, further comprising:
identifying, based on the one or more decision paths within the first tree-based ML model, a second value range for the first input feature, wherein a contribution of the first input feature to one or more of the risk-related predictions, within the second value range, is determined, and the contributions of the first input feature exhibit stability falling within the defined threshold; and (in at least [0102] first level score and/or codes 160 may be compared with one or more score ranges. A first range for application scores may correspond with providing the application score as output to a user device and a second range for application scores may correspond with not providing the application score as output to a user device. [0103] a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7. [0120] The ML model may further comprise an outlier detection method, which identifies significant deviations from the multivariate density distributions of a plurality of independent variables, even if such deviations have not previously been correlated with fraud in historical application data. [0193] The report further includes a score (illustrated as score “998”), which may be calculated as described above. The report may further include a risk level (illustrated as “high”). The risk level may be determined by comparing the score to one or more thresholds. For example, if the score is above the threshold, a risk level of “high” may be determined. If the score is below the threshold, a risk level of “low” may be determined. Each threshold may be associated with a different level of risk (e.g., low, medium, and high). When there are more than one threshold, a threshold may be defined as between two thresholds.)
updating the knowledge base table to associate the first input feature with the second value range, a reason code, and a text-based explanation describing how the first input feature, within the second value range, influences the risk-related predictions. (in at least [0193] The report further includes a score (illustrated as score “998”), which may be calculated as described above. The report may further include a risk level (illustrated as “high”). The risk level may be determined by comparing the score to one or more thresholds. For example, if the score is above the threshold, a risk level of “high” may be determined. If the score is below the threshold, a risk level of “low” may be determined. Each threshold may be associated with a different level of risk (e.g., low, medium, and high). When there are more than one threshold, a threshold may be defined as between two thresholds. [0194] The report further includes reason codes. A reason code may comprise information corresponding to a feature that contributes above a particular threshold to the score. An example of a reason code is that the car dealer is located a significant distance from the borrower address.)
As per Claim 8, McKenna teaches: The method of claim 7, further comprising:
receiving real-time … data as input data by the first tree-based ML model to generate a real-time risk-related prediction; (in at least [0087] The fraud detection computer system 120 may comprise a fraud scoring engine 142. The fraud scoring engine 142 may be configured to determine one or more machine learning (ML) models to apply to application data. In some examples, the discrepancies or similarities between the risk profile and the application data may be provided as input to the ML model as well. Different models may also be used to detect different entities committing the fraud (e.g., borrower fraud v. dealer fraud), or to predict a fraud type (e.g., income fraud, collateral fraud, identity fraud, straw borrower fraud, employment fraud, etc.). These different models may be constructed using an input feature library where one or more input features may be based on a variety of micro fraud patterns observed in the application data. The fraud scoring engine 142 may be configured to compute a first level score corresponding with a borrower user, or a second level score corresponding with a dealer user. The first and second level scores may be computed concurrently or separately, as further described herein. [0110] At 210, a first input may be received. For example, the fraud detection computer system 120 may receive an application object for an application that includes information associated with a first borrower user device 110 as well as a segment corresponding with the application object. The fraud detection computer system 120 may receive a request from the third lender user device 116 associated with application data. [0209] A plurality of applications may be received by the fraud detection computer system 120 from a plurality of dealer user devices, including application data associated with a first user 910A and application data associated with a second user 910B (in combination, referred to as “users 910”). In this illustration, the dealer user names may be similar in each application. For example, the first user 910A may correspond with “Cars of Acme” dealer name and the second user 910B may correspond with “Acme Cars” dealer name.)
determining that a value of the first input feature, extracted from the real-time … data, falls within the second value range for the first input feature; (in at least [0097] a high application score may correspond with a score of 700 or greater. Accordingly, any application with a score above the threshold of 700, for example, may be determined to be high risk, such that an application corresponding with this score may be, for example, 70% more likely to contain fraudulent information than an application corresponding with a lower score. In another example, one out of twenty applications might be fraudulent, in comparison with one out of one hundred applications that may be fraudulent with a lower score. For another example, when the application is determined to be medium risk, an action of additional review of one or more portions of the application with underwriter review may be suggested. A medium threshold associated with medium risk may correspond with the score range of 300-700. Accordingly, any application with a score within the medium threshold may be determined to be medium risk. For another example, when the application is determined to be low risk, an action of streamlining the application (without additional review) may be suggested. A low threshold associated with low risk may correspond with score range of 1-399. Accordingly, any application with a score between 1-399 may be determined to be low risk. [0102] In some examples, first level score and/or codes 160 may be compared with one or more score ranges. A first range for application scores may correspond with providing the application score as output to a user device and a second range for application scores may correspond with not providing the application score as output to a user device. [0103] a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7. [0104] A second level score and/or codes 162 may also be computed. The second level score and/or codes 162 may be based at least in part on scores computed for a dealer user. The second level score, with or without the first level scores, may be transmitted to the lender user, for example, for the lender user to determine the level of diligence required when reviewing the application and/or determination of providing funding to the borrower user.)
retrieving, from the knowledge base table, the reason code associated with the first input feature; and (in at least [0092] The fraud detection computer system 120 may comprise a code module 144. The code module 144 may be configured to determine one or more reason codes for the application. For example, one or more features may influence the application score above a particular threshold to identify a potential risk or fraud. The features may correspond with a reason code that indicates an amount the feature can affect the application score and/or a reason that the feature affects the application score the way that it does. In some examples, the potential reason may be user defined (e.g., an administrator of the service may define reasons for particular features when seen in isolation or in combination with other features). In some examples, the one or more features may be grouped into categories (sometimes referred to as factor groups). Examples of factor groups include income, employment, identity, or the like. In each factor group, one or more features may be identified in order to be used to determine reason codes. [0100] One or more modules and engines of the fraud detection computer system 120 may also be configured to analyze and determine second level scores (e.g., application scores associated with a dealer user and/or second ML model). Any of the functions described herein may be implemented in determining the second level score. For example, the application engine 136 may be configured to receive application data. The profiling module 138 may be configured to determine a segment associated with the application data and generate one or more input features for providing to a first ML model. The fraud scoring engine 142 may be configured to determine a score for a plurality of applications, corresponding with first level scores. The fraud detection computer system 120 may combine (e.g., aggregate, etc.) these application scores based on common information, including a common dealer user. The combined application scores for a common dealer user may correspond with a second level score. The first level score and/or the second level score, and corresponding user information for each of the one or more scores, may be stored with the scores data store 152.)
outputting the reason code and the text-based explanation describing how the first input feature influences the real-time risk-related prediction. (in at least [0029] provide output to a user interface, including one or more scores, reason codes, or actions. For example, the system may provide a first score associated with a first borrower user and/or a second score associated with a dealer user. Each of the corresponding scores may be provided with additional information, including the borrower user, user identifiers, reason codes, or suggested actions, for example, to mitigate risk or identify fraud associated with the application data. [0095] The fraud detection computer system 120 may comprise an action engine 146. The action engine 146 may be configured to determine one or more actions to perform in association with the application score, input features, segment, and other data described herein. The one or more actions may be determined based upon the one or more reason codes or any application data that may influence the application score above the particular threshold (e.g., when a discrepancy is determined between the application data and a third party data source, when a similarity is determined between a risk profile and the application data, etc.). The fraud detection computer system 120, via the ML model, may output the application score along with the one or more reason codes with an indication associated with each of the one or more actions. [0103] a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7. [0216] FIG. 10 illustrates a report for indicating a score according to an embodiment of the disclosure. In illustration 1000, the report may include a score for a dealer user associated with a lender user. The report may comprise aggregated application data and/or application scores by dealer identifier. The report may also comprise dealer information associated with a loan application, including dealer ID, dealer name, location associated with the dealer, a phone number for the dealer, a volume of applications for the domain in a particular amount of time.)
Although implied, McKenna does not expressly disclose the following limitations, which however, are taught by Chang,
…project… (in at least [0024] The entities include users that use online network 118 to establish and maintain professional connections, list work and community experience, endorse and/or recommend one another, search and apply for jobs, and/or perform other actions. The entities also, or instead, include companies, employers, and/or recruiters that use online network 118 to list jobs, search for potential candidates, provide business-related updates to users, advertise, and/or take other action. [0036] online network 118 includes a service that uses a machine learning model to generate a set of relevance scores representing the compatibility of a user with a set of jobs (or the compatibility of a set of users as candidates for a job). The service receives a scoring request in response to the user's job search (or a recruiter's search for candidates matching a job), the user accessing a job recommendation component in online network 118, and/or the user otherwise interacting with job-related functionality in online network 118. To improve the accuracy of the scores, the service inputs, into the machine learning model, features that reflect the latest activity by the user (or recruiter), such as features representing the user's searches, clicks, likes, dislikes, and/or other actions performed in the same user session as the user's interaction with the job-related functionality. In response to the inputted features, the machine learning model calculates the relevance scores between the user and jobs (or a set of candidates and a job) in a real-time or near-real-time basis (e.g., with a latency that is within a limit specified in a service level agreement (SLA) for the service). The service returns the relevance scores in a response to the scoring request, and online network 118 outputs, to the user, a ranking of jobs (or candidates) by descending relevance score as search results, recommendations, and/or other representations of the jobs (or candidates). [0037] Those skilled in the art will appreciate that online network 118 may perform on-demand scoring and/or ranking related to other types or combinations of entities. For example, online network 118 may receive one or more scoring requests for relevance scores between a user and content items created and/or shared by other users of the online network 118. These content items include, but are not limited to, posts, articles, comments, updates, and/or videos. The scoring request(s) may be generated in response to the user accessing a content feed in a homepage, landing page, and/or another part of online network 118. In response to the scoring request(s), a first set of machine learning models 110 is executed to generate recommendations of specific types of content (e.g., network updates, jobs, articles, courses, advertisements, posts, products, connections, products, etc.) for the user based on attributes of the user, recent activity by the user and/or similar users, and/or attributes of the content. The first set of machine learning models 110 output scores representing predictions of the user's level of interest in the content, and a subset of content items with high scores from the first set of machine learning models 110 is selected. Features associated with the user and the selected content items are inputted into one or more additional machine learning models 110, and the additional machine learning model(s) generate an additional set of scores representing predicted likelihoods of the user performing certain actions on (e.g., clicking, liking, commenting on, sharing, etc.) the selected content items. The content items are ranked by descending score, and one or more rules, filters, and/or exceptions are used to update the ranking (e.g., based on business requirements, member preferences, impression discounting, diversification of content, and/or other goals, strategies, or priorities). Finally, the updated ranking is displayed in a content feed for the user. Because the content items are scored and ranked in real-time or near-real-time, the content feed is able to reflect the latest content posted to online network 118 and/or user interactions with the content in online network 118.)
The reason and rationale to combine McKenna and Chang is the same as recited above.
As per Claim 9, McKenna teaches: The method of claim 1, wherein identifying the second input feature comprises:
extracting one or more data samples from the historical project delivery data, wherein the first input feature within the one or more data samples comprises values falling within the identified first value range; (in at least [0122] Prior to receiving the input features associated with the application data, the ML model may be trained using a training data set of historical application data. For example, the training data set may comprise a plurality of application data and determinations of whether fraud was discovered according to the risk profiles described herein. The ML model may be trained using historical data to determine one or more weights assigned to each of the input features according to a risk profile. In some examples, input features from the historical data that are common amongst a subset of applications may be identified as indicators of potential fraud according to the risk profile. The ML model may determine subsequent application data that identifies similar features as the training data set in order to determine a score for the application that identifies the similarities between the training data set and the application data. [0165] application scoring system 340 may further include a factor group extraction subsystem. The factor group extraction subsystem may order the one or more features used as input to application scoring system 340. For instance, the ordering may be based upon an amount that each feature affected a score. The factor group extraction subsystem may also group the one or more features into one or more groups.)
training the second tree-based ML model using the data samples as input data; (in at least [0129] The data sources may be compared using a tiered matching algorithm, including fuzzy matching. For example, the fraud detection computer system 120 may receive the normalized and transformed data and apply fuzzing matching algorithm to the data. When similarities are detected above a similarity threshold (e.g., 90 out of 100 potential match score), the fraud detection computer system 120 may cluster this data to identify individual dealer users. The individual dealer user may be assigned a new dealer user identifier to correspond with the combined data records. [0131] At 270, one or more input features may be applied to a second trained ML model. For example, the fraud detection computer system 120 may determine a second score by selecting the application score from a plurality of application scores and determining the second score by applying at the input features to the trained second ML model. [0132] The fraud detection computer system 120 may apply various ML models and embodiments of the disclosure. For example, the trained second ML model may comprise a supervised learning algorithm including a decision tree that accepts the one or more input features associated with the application to provide the second level score.)
identifying, based on one or more decision paths within the second tree-based ML model with the first input feature excluded, a second value range for the second input feature, wherein a contribution of the second input feature to one or more of the risk-related predictions, within the second value range, is determined, and the contributions exhibit inconsistency that exceeds a defined threshold; and (in at least [0132] The fraud detection computer system 120 may apply various ML models and embodiments of the disclosure. For example, the trained second ML model may comprise a supervised learning algorithm including a decision tree that accepts the one or more input features associated with the application to provide the second level score. [0138] The second ML model may further comprise an outlier detection method, which identifies significant deviations from the multivariate density distributions of a plurality of independent variables, even if such deviations have not previously been correlated with fraud in historical application data. [0209] A plurality of applications may be received by the fraud detection computer system 120 from a plurality of dealer user devices, including application data associated with a first user 910A and application data associated with a second user 910B (in combination, referred to as “users 910”). In this illustration, the dealer user names may be similar in each application. For example, the first user 910A may correspond with “Cars of Acme” dealer name and the second user 910B may correspond with “Acme Cars” dealer name. [0211] The fraud detection computer system 120 may implement a transformation process 930 using the output of the normalization process 920 as input. For example, the fraud detection computer system 120 may transform this data by removing generic words including “a” or “the.” In another example, the transformation process may add the word “and” in place of an “&” (ampersand). [0224] At 1130, a second score may be determined from a second ML model. For example, the fraud detection computer system 120 may determine a second score associate with the plurality of application scores. The second score may be selected from a plurality of second scores (e.g., ranging between one and 999, etc.). In some examples, determining the second score may comprise applying the one or more input features associated with the plurality of applications to the second ML model. [0225] the second score determined based upon a pattern recognition ML model. For example, the pattern recognition ML model may be a scorecard model that may be driven by historical application scores observed for the dealer user device. The process of updating the second score may occur after each application is associated with a first score so that the second score is immediately available for subsequent applications.)
updating the knowledge base table to associate the combination with the first value range for the first input feature, the second value range for the second input feature, a reason code, and a text-based explanation indicating conflicts are not resolved and describing a generic reason about how the first input feature, within the first value range for the first input feature, influences the risk-related predictions. (in at least [0029] provide output to a user interface, including one or more scores, reason codes, or actions. For example, the system may provide a first score associated with a first borrower user and/or a second score associated with a dealer user. Each of the corresponding scores may be provided with additional information, including the borrower user, user identifiers, reason codes, or suggested actions, for example, to mitigate risk or identify fraud associated with the application data. [0095] The fraud detection computer system 120 may comprise an action engine 146. The action engine 146 may be configured to determine one or more actions to perform in association with the application score, input features, segment, and other data described herein. The one or more actions may be determined based upon the one or more reason codes or any application data that may influence the application score above the particular threshold (e.g., when a discrepancy is determined between the application data and a third party data source, when a similarity is determined between a risk profile and the application data, etc.). The fraud detection computer system 120, via the ML model, may output the application score along with the one or more reason codes with an indication associated with each of the one or more actions. [0186] The scoring service 530 may determine that the mismatch user identifiers may represent a potential straw borrower or identity risk. As another example, an employer identifier may be input that does not match a list of existing employers from a secretary of state data source. As another example, an email address may be input that is associated with prior fraud from a third-party data source. [0216] FIG. 10 illustrates a report for indicating a score according to an embodiment of the disclosure. In illustration 1000, the report may include a score for a dealer user associated with a lender user. The report may comprise aggregated application data and/or application scores by dealer identifier. The report may also comprise dealer information associated with a loan application, including dealer ID, dealer name, location associated with the dealer, a phone number for the dealer, a volume of applications for the domain in a particular amount of time.)
Although implied, McKenna does not expressly disclose the following limitations, which however, are taught by Chang,
…project delivery… (in at least [0024] The entities include users that use online network 118 to establish and maintain professional connections, list work and community experience, endorse and/or recommend one another, search and apply for jobs, and/or perform other actions. The entities also, or instead, include companies, employers, and/or recruiters that use online network 118 to list jobs, search for potential candidates, provide business-related updates to users, advertise, and/or take other action. [0036] online network 118 includes a service that uses a machine learning model to generate a set of relevance scores representing the compatibility of a user with a set of jobs (or the compatibility of a set of users as candidates for a job). The service receives a scoring request in response to the user's job search (or a recruiter's search for candidates matching a job), the user accessing a job recommendation component in online network 118, and/or the user otherwise interacting with job-related functionality in online network 118. To improve the accuracy of the scores, the service inputs, into the machine learning model, features that reflect the latest activity by the user (or recruiter), such as features representing the user's searches, clicks, likes, dislikes, and/or other actions performed in the same user session as the user's interaction with the job-related functionality. In response to the inputted features, the machine learning model calculates the relevance scores between the user and jobs (or a set of candidates and a job) in a real-time or near-real-time basis (e.g., with a latency that is within a limit specified in a service level agreement (SLA) for the service). The service returns the relevance scores in a response to the scoring request, and online network 118 outputs, to the user, a ranking of jobs (or candidates) by descending relevance score as search results, recommendations, and/or other representations of the jobs (or candidates). [0037] Those skilled in the art will appreciate that online network 118 may perform on-demand scoring and/or ranking related to other types or combinations of entities. For example, online network 118 may receive one or more scoring requests for relevance scores between a user and content items created and/or shared by other users of the online network 118. These content items include, but are not limited to, posts, articles, comments, updates, and/or videos. The scoring request(s) may be generated in response to the user accessing a content feed in a homepage, landing page, and/or another part of online network 118. In response to the scoring request(s), a first set of machine learning models 110 is executed to generate recommendations of specific types of content (e.g., network updates, jobs, articles, courses, advertisements, posts, products, connections, products, etc.) for the user based on attributes of the user, recent activity by the user and/or similar users, and/or attributes of the content. The first set of machine learning models 110 output scores representing predictions of the user's level of interest in the content, and a subset of content items with high scores from the first set of machine learning models 110 is selected. Features associated with the user and the selected content items are inputted into one or more additional machine learning models 110, and the additional machine learning model(s) generate an additional set of scores representing predicted likelihoods of the user performing certain actions on (e.g., clicking, liking, commenting on, sharing, etc.) the selected content items. The content items are ranked by descending score, and one or more rules, filters, and/or exceptions are used to update the ranking (e.g., based on business requirements, member preferences, impression discounting, diversification of content, and/or other goals, strategies, or priorities). Finally, the updated ranking is displayed in a content feed for the user. Because the content items are scored and ranked in real-time or near-real-time, the content feed is able to reflect the latest content posted to online network 118 and/or user interactions with the content in online network 118.)
… excluded…(in at least [0053] simplification apparatus 202 sets feature removal threshold 232 to an estimated number of features to be removed from baseline version 208 to lower baseline resource overhead 216 to target resource overhead 218. For example, simplification apparatus 202 retrieves, from model repository 236 and/or another data store, historical resource overheads of different versions of the machine learning model (e.g., different baseline versions that have been updated over time).)
The reason and rationale to combine McKenna and Chang is the same as recited above.
As per Claim 10, McKenna teaches: The method of claim 9, further comprising:
receiving real-time … data as input data by the first tree-based ML model to generate a real-time risk-related prediction; (in at least [0087] The fraud detection computer system 120 may comprise a fraud scoring engine 142. The fraud scoring engine 142 may be configured to determine one or more machine learning (ML) models to apply to application data. In some examples, the discrepancies or similarities between the risk profile and the application data may be provided as input to the ML model as well. Different models may also be used to detect different entities committing the fraud (e.g., borrower fraud v. dealer fraud), or to predict a fraud type (e.g., income fraud, collateral fraud, identity fraud, straw borrower fraud, employment fraud, etc.). These different models may be constructed using an input feature library where one or more input features may be based on a variety of micro fraud patterns observed in the application data. The fraud scoring engine 142 may be configured to compute a first level score corresponding with a borrower user, or a second level score corresponding with a dealer user. The first and second level scores may be computed concurrently or separately, as further described herein. [0110] At 210, a first input may be received. For example, the fraud detection computer system 120 may receive an application object for an application that includes information associated with a first borrower user device 110 as well as a segment corresponding with the application object. The fraud detection computer system 120 may receive a request from the third lender user device 116 associated with application data. [0209] A plurality of applications may be received by the fraud detection computer system 120 from a plurality of dealer user devices, including application data associated with a first user 910A and application data associated with a second user 910B (in combination, referred to as “users 910”). In this illustration, the dealer user names may be similar in each application. For example, the first user 910A may correspond with “Cars of Acme” dealer name and the second user 910B may correspond with “Acme Cars” dealer name.)
determining that a value of the first input feature, extracted from the real-time … data, falls within the identified first value range for the first input feature; (in at least [0097] a high application score may correspond with a score of 700 or greater. Accordingly, any application with a score above the threshold of 700, for example, may be determined to be high risk, such that an application corresponding with this score may be, for example, 70% more likely to contain fraudulent information than an application corresponding with a lower score. In another example, one out of twenty applications might be fraudulent, in comparison with one out of one hundred applications that may be fraudulent with a lower score. For another example, when the application is determined to be medium risk, an action of additional review of one or more portions of the application with underwriter review may be suggested. A medium threshold associated with medium risk may correspond with the score range of 300-700. Accordingly, any application with a score within the medium threshold may be determined to be medium risk. For another example, when the application is determined to be low risk, an action of streamlining the application (without additional review) may be suggested. A low threshold associated with low risk may correspond with score range of 1-399. Accordingly, any application with a score between 1-399 may be determined to be low risk. [0102] In some examples, first level score and/or codes 160 may be compared with one or more score ranges. A first range for application scores may correspond with providing the application score as output to a user device and a second range for application scores may correspond with not providing the application score as output to a user device. [0103] a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7.)
determining that a value of the second input feature, extracted from the real-time … data, falls within the identified second value range for the second input feature; (in at least [0102] In some examples, first level score and/or codes 160 may be compared with one or more score ranges. A first range for application scores may correspond with providing the application score as output to a user device and a second range for application scores may correspond with not providing the application score as output to a user device. [0103] a score of three hundred may correspond with a second range for scores that identify applications with a lower likelihood of fraud or risk. When the score is determined in this second range, the fraud detection computer system 120 may identify that fraud is less likely with this application. As another illustration, a score of nine hundred may correspond with a first range of scores that identify applications with a higher likelihood of risk or fraud. When the score is determined in this first range, the fraud detection computer system 120 may provide the score one or more reason codes for the application corresponding with this higher risk, and one or more actions for the application to help mitigate the risk and reduce the instance of fraud. A sample output of a score corresponding with this first range is provided with FIG. 7. [0125] At 240, a second input may be received. For example, the fraud detection computer system 120 may receive one or more scores stored with the scores data store 152 and/or historical application data stored with the profiles data store 150. At least some of this information may correspond with output from the first ML model. The application data and/or scores may be received as input for a second ML model. In some examples, the fraud detection computer system 120 may also receive additional data, including third-party and/or consortium data corresponding with one or more dealer user devices. In some examples, the output, historical application data, third-party data, and/or consortium data may correspond with a particular dealer user. In some examples, the application score associated with the dealer user may identify the likelihood of fraud corresponding with applications submitted by the dealer user device to the lender user device. [0209] A plurality of applications may be received by the fraud detection computer system 120 from a plurality of dealer user devices, including application data associated with a first user 910A and application data associated with a second user 910B (in combination, referred to as “users 910”). In this illustration, the dealer user names may be similar in each application. For example, the first user 910A may correspond with “Cars of Acme” dealer name and the second user 910B may correspond with “Acme Cars” dealer name. [0210] The fraud detection computer system 120 may implement a normalization process 920 using information corresponding with users 910 as input. For example, the fraud detection computer system 120 may normalize the impact by removing periods, spaces, or capitalization of characters to form a string of text associated with the dealer user.)
retrieving, from the knowledge base table, the reason code associated with the combination; and (in at least [0092] The fraud detection computer system 120 may comprise a code module 144. The code module 144 may be configured to determine one or more reason codes for the application. For example, one or more features may influence the application score above a particular threshold to identify a potential risk or fraud. The features may correspond with a reason code that indicates an amount the feature can affect the application score and/or a reason that the feature affects the application score the way that it does. In some examples, the potential reason may be user defined (e.g., an administrator of the service may define reasons for particular features when seen in isolation or in combination with other features). In some examples, the one or more features may be grouped into categories (sometimes referred to as factor groups). Examples of factor groups include income, employment, identity, or the like. In each factor group, one or more features may be identified in order to be used to determine reason codes. [0100] One or more modules and engines of the fraud detection computer system 120 may also be configured to analyze and determine second level scores (e.g., application scores associated with a dealer user and/or second ML model). Any of the functions described herein may be implemented in determining the second level score. For example, the application engine 136 may be configured to receive application data. The profiling module 138 may be configured to determine a segment associated with the application data and generate one or more input features for providing to a first ML model. The fraud scoring engine 142 may be configured to determine a score for a plurality of applications, corresponding with first level scores. The fraud detection computer system 120 may combine (e.g., aggregate, etc.) these application scores based on common information, including a common dealer user. The combined application scores for a common dealer user may correspond with a second level score. The first level score and/or the second level score, and corresponding user information for each of the one or more scores, may be stored with the scores data store 152.)
outputting the reason code and the text-based explanation indicating conflicts are not resolved and describing a generic reason about how the first input feature, within the first value range for the first input feature, influences the real-time risk-related prediction. (in at least [0029] provide output to a user interface, including one or more scores, reason codes, or actions. For example, the system may provide a first score associated with a first borrower user and/or a second score associated with a dealer user. Each of the corresponding scores may be provided with additional information, including the borrower user, user identifiers, reason codes, or suggested actions, for example, to mitigate risk or identify fraud associated with the application data. [0095] The fraud detection computer system 120 may comprise an action engine 146. The action engine 146 may be configured to determine one or more actions to perform in association with the application score, input features, segment, and other data described herein. The one or more actions may be determined based upon the one or more reason codes or any application data that may influence the application score above the particular threshold (e.g., when a discrepancy is determined between the application data and a third party data source, when a similarity is determined between a risk profile and the application data, etc.). The fraud detection computer system 120, via the ML model, may output the application score along with the one or more reason codes with an indication associated with each of the one or more actions. [0186] The scoring service 530 may determine that the mismatch user identifiers may represent a potential straw borrower or identity risk. As another example, an employer identifier may be input that does not match a list of existing employers from a secretary of state data source. As another example, an email address may be input that is associated with prior fraud from a third-party data source. [0216] FIG. 10 illustrates a report for indicating a score according to an embodiment of the disclosure. In illustration 1000, the report may include a score for a dealer user associated with a lender user. The report may comprise aggregated application data and/or application scores by dealer identifier. The report may also comprise dealer information associated with a loan application, including dealer ID, dealer name, location associated with the dealer, a phone number for the dealer, a volume of applications for the domain in a particular amount of time.)
Although implied, McKenna does not expressly disclose the following limitations, which however, are taught by Chang,
…project… (in at least [0024] The entities include users that use online network 118 to establish and maintain professional connections, list work and community experience, endorse and/or recommend one another, search and apply for jobs, and/or perform other actions. The entities also, or instead, include companies, employers, and/or recruiters that use online network 118 to list jobs, search for potential candidates, provide business-related updates to users, advertise, and/or take other action. [0036] online network 118 includes a service that uses a machine learning model to generate a set of relevance scores representing the compatibility of a user with a set of jobs (or the compatibility of a set of users as candidates for a job). The service receives a scoring request in response to the user's job search (or a recruiter's search for candidates matching a job), the user accessing a job recommendation component in online network 118, and/or the user otherwise interacting with job-related functionality in online network 118. To improve the accuracy of the scores, the service inputs, into the machine learning model, features that reflect the latest activity by the user (or recruiter), such as features representing the user's searches, clicks, likes, dislikes, and/or other actions performed in the same user session as the user's interaction with the job-related functionality. In response to the inputted features, the machine learning model calculates the relevance scores between the user and jobs (or a set of candidates and a job) in a real-time or near-real-time basis (e.g., with a latency that is within a limit specified in a service level agreement (SLA) for the service). The service returns the relevance scores in a response to the scoring request, and online network 118 outputs, to the user, a ranking of jobs (or candidates) by descending relevance score as search results, recommendations, and/or other representations of the jobs (or candidates). [0037] Those skilled in the art will appreciate that online network 118 may perform on-demand scoring and/or ranking related to other types or combinations of entities. For example, online network 118 may receive one or more scoring requests for relevance scores between a user and content items created and/or shared by other users of the online network 118. These content items include, but are not limited to, posts, articles, comments, updates, and/or videos. The scoring request(s) may be generated in response to the user accessing a content feed in a homepage, landing page, and/or another part of online network 118. In response to the scoring request(s), a first set of machine learning models 110 is executed to generate recommendations of specific types of content (e.g., network updates, jobs, articles, courses, advertisements, posts, products, connections, products, etc.) for the user based on attributes of the user, recent activity by the user and/or similar users, and/or attributes of the content. The first set of machine learning models 110 output scores representing predictions of the user's level of interest in the content, and a subset of content items with high scores from the first set of machine learning models 110 is selected. Features associated with the user and the selected content items are inputted into one or more additional machine learning models 110, and the additional machine learning model(s) generate an additional set of scores representing predicted likelihoods of the user performing certain actions on (e.g., clicking, liking, commenting on, sharing, etc.) the selected content items. The content items are ranked by descending score, and one or more rules, filters, and/or exceptions are used to update the ranking (e.g., based on business requirements, member preferences, impression discounting, diversification of content, and/or other goals, strategies, or priorities). Finally, the updated ranking is displayed in a content feed for the user. Because the content items are scored and ranked in real-time or near-real-time, the content feed is able to reflect the latest content posted to online network 118 and/or user interactions with the content in online network 118.)
The reason and rationale to combine McKenna and Chang is the same as recited above.
As per Claim 21, McKenna teaches: (New) The method of claim 1, further comprising:
determining the contribution of the first input feature based on aggregation of feature contributions along the one or more decision paths, wherein the inconsistency corresponds to variation in contribution magnitude or direction across the one or more decision paths. (in at least [0093] the input features may correspond with a plurality of applications and/or aggregated application data associated with a dealer user device. In some examples, an input feature may correspond with a risk signal that may determine a likelihood of fraud associated with a portion of the application data for the dealer user device. The input feature, in some examples, may be predetermined prior to providing the application data to a trained ML model. [0124] the output may also comprise one or more reason codes and/or one or more suggested actions corresponding with the score or the reason codes. For example, the fraud detection computer system 120 may determine the input features that closely corresponded to the risk profiles implemented by the trained ML model. Each of the input features may correspond with one or more reason codes and/or one or more suggested actions. In some examples, the fraud detection computer system 120 may select a first input feature and provide it as a search term to the code/action data store 154. The data store may return the one or more reason codes that correspond with the first input feature as well as one or more actions that correspond with the first input feature. The returned data from the code/action data store 154 may be added, with the score, to the application report or electronic message transmitted to the user device. [0125] At 240, a second input may be received. For example, the fraud detection computer system 120 may receive one or more scores stored with the scores data store 152 and/or historical application data stored with the profiles data store 150. At least some of this information may correspond with output from the first ML model. The application data and/or scores may be received as input for a second ML model. In some examples, the fraud detection computer system 120 may also receive additional data, including third-party and/or consortium data corresponding with one or more dealer user devices. In some examples, the output, historical application data, third-party data, and/or consortium data may correspond with a particular dealer user. In some examples, the application score associated with the dealer user may identify the likelihood of fraud corresponding with applications submitted by the dealer user device to the lender user device [0126] At 250, the data may be normalized and/or transformed. For example, the fraud detection computer system 120 may receive the dealer user name, address, or other information associated with the dealer user. The fraud detection computer system 120 may normalize this data by removing periods, spaces, or capitalization of characters to form a string of text associated with the dealer user. The fraud detection computer system 120 may transform this data by removing generic words including “a” or “the.” In some examples, the normalization and/or transformation process may add information as well, including adding the word “and” in place of an “&” (ampersand). The normalization and/or transformation of the dealer user information may help standardize different sources of dealer user information. This may include different applications provided by different borrower users. [0162] Feature extraction system 320 may also output one or more features to discrepancy detection system 360. Discrepancy detection system 360 may include scorecard models and/or expert rules designed to flag inconsistent and/or out-of-pattern data values within the outputs of feature extraction system 320. One example of this type of data value discrepancy may be a loan amount substantially greater than the list price of the collateral. For another example, if the borrower's stated income exceeds the known average income for the borrower's residential area by a certain percentage. Discrepancy detection system 360 may output a result to application scoring system 340, consortium data lookup system 326, and/or third-party data lookup system 322. [0187] FIG. 6 illustrates a score process according to an embodiment of the disclosure. In illustration 600, a sample process is illustrated to generate a score for an application. The diagram includes an API input layer 610, a feature extraction layer 620, and a scoring layer 630. [0188] The API input layer 610 may receive information extracted from or correlated with an application. Examples of such information include income of the borrower, an identification of an occupation of the borrower, employer of the borrower, employment type of the borrower, collateral information associated with collateral (e.g., if the collateral is a car, the collateral information may include make, model, or sales price), borrower information (e.g., DTI, PTI, credit, loan type, etc.), or dealer information (e.g., identification of a name or location of the dealer). [0189] The API input layer 610 may transmit the information to the feature extraction layer 620. The feature extraction layer 620 may be configured to identify input features from the API input layer 610 and generate relationships and correlations between the data based on similarities, clusters, aggregation of the data, and the like. The feature extraction layer 620 may generate one or more input features from information received in the API input layer 620. Examples of features include income to collateral, income to dealer average, income to employer, employer risk level, close to tier risk, make model risk ratio, etc. [0190] The feature extraction layer 620 may transmit the one or more features to the scoring layer 630. The scoring layer 630 may compute a score. In some examples, the scoring layer 630 may summarize a risk associated with an application or dealer user. The scoring layer 630 may utilize a trained machine learning (ML) model (e.g., pattern recognition model, neural network, decision tree, clustering, etc.) to compute the score. Input to the scoring layer 630 may include one or more features and/or information received in the API input layer 610. In one illustrative example, the score may range from 1 (low risk) to 999 (high risk). [0203] The application score may be determined by applying the one or more input features associated with the application to the trained ML model. In some examples, the application score may be determined based upon a pattern recognition model using the one or more features as input. The pattern recognition model may be trained based upon previous applications. In some examples, the pattern recognition model may be trained such that each feature has a weight computed for the feature. [0225] the second score determined based upon a pattern recognition ML model. For example, the pattern recognition ML model may be a scorecard model that may be driven by historical application scores observed for the dealer user device. The process of updating the second score may occur after each application is associated with a first score so that the second score is immediately available for subsequent applications.)
As per Claim 22, McKenna teaches: (New) The method of claim 1, wherein training the second tree-based ML model comprises:
training the second tree-based ML model on a subset of the historical … data corresponding to the first value range, such that the second input feature is identified based on feature interaction within the subset. (in at least [0027] The second ML model may receive the output from the first ML model as input. For example, upon receiving the output from the first ML model and/or application data from one or more borrower user devices, the system may apply the data to a second ML model to determine a second score. The first scores may be used as a training data set for the second ML model or, in some examples, may be provided to a previously-trained second ML model as input. The output from the second ML model may indicate signals of fraud and/or predict the type of fraud associated with a particular dealer user device and the corresponding output from the first ML model. [0090] A plurality of ML models may be trained to correspond with a plurality of segments, such that at least one ML model may be trained to determine an output associated with a single segment. Various devices or computer systems may assign segmentation information to the application data, including the lender user device 116 or the fraud detection computer system 120. In either instance, these computing systems may be configured to determine the segmentation information corresponding with the application data. [0091] a segment may correspond with prime loan or subprime loan, each of which may correspond with application data. If an application is associated with a prime loan, the application may be submitted to a ML model corresponding with a prime model. Similarly, if an application is associated with a subprime loan, the application may be submitted to a ML model corresponding with a subprime model. [0125] At 240, a second input may be received. For example, the fraud detection computer system 120 may receive one or more scores stored with the scores data store 152 and/or historical application data stored with the profiles data store 150. At least some of this information may correspond with output from the first ML model. The application data and/or scores may be received as input for a second ML model. In some examples, the fraud detection computer system 120 may also receive additional data, including third-party and/or consortium data corresponding with one or more dealer user devices. In some examples, the output, historical application data, third-party data, and/or consortium data may correspond with a particular dealer user. In some examples, the application score associated with the dealer user may identify the likelihood of fraud corresponding with applications submitted by the dealer user device to the lender user device. [0126] At 250, the data may be normalized and/or transformed. For example, the fraud detection computer system 120 may receive the dealer user name, address, or other information associated with the dealer user. The fraud detection computer system 120 may normalize this data by removing periods, spaces, or capitalization of characters to form a string of text associated with the dealer user. The fraud detection computer system 120 may transform this data by removing generic words including “a” or “the.” In some examples, the normalization and/or transformation process may add information as well, including adding the word “and” in place of an “&” (ampersand). The normalization and/or transformation of the dealer user information may help standardize different sources of dealer user information. This may include different applications provided by different borrower users. [0130] At 260, the data may be combined for the particular user. For example, the fraud detection computer system 120 may correlate a dealer user identifier for any data entry corresponding with the particular dealer user. The combination of the dealer user data may identify any application data corresponding with a particular dealer user, including current or historical data. Duplicate data may be removed in the combination process. This may help identify any application data submitted in association with the particular dealer user to any lender user for approval of an application on behalf of any borrower user. [0131] At 270, one or more input features may be applied to a second trained ML model. For example, the fraud detection computer system 120 may determine a second score by selecting the application score from a plurality of application scores and determining the second score by applying at the input features to the trained second ML model. [0201] At 820, a segment may be determined. For example, the computer system may determine a segment associated with the first information of the application object. The segment may identify a grouping associated with the application. In some examples, the segment may help identify and determine a corresponding ML model for the application data. [0202] At 830, one or more features may be generated for the application. For example, the computer system may generate an input feature based at least in part on the first information, borrower user device, dealer user device, lender user device, or the application data. In some examples, a feature may be based upon information associated with one or more user devices other than these devices. [0203] At 840, a score may be determined for the application. For example, the computer system may determine the application score for the application. The application score may be selected from a plurality of application scores. The application score may be determined by applying the one or more input features associated with the application to the trained ML model. In some examples, the application score may be determined based upon a pattern recognition model using the one or more features as input. The pattern recognition model may be trained based upon previous applications. In some examples, the pattern recognition model may be trained such that each feature has a weight computed for the feature. [0225] the second score determined based upon a pattern recognition ML model. For example, the pattern recognition ML model may be a scorecard model that may be driven by historical application scores observed for the dealer user device. The process of updating the second score may occur after each application is associated with a first score so that the second score is immediately available for subsequent applications.)
Although implied, McKenna does not expressly disclose the following limitations, which however, are taught by Chang,
…project delivery… (in at least [0024] The entities include users that use online network 118 to establish and maintain professional connections, list work and community experience, endorse and/or recommend one another, search and apply for jobs, and/or perform other actions. The entities also, or instead, include companies, employers, and/or recruiters that use online network 118 to list jobs, search for potential candidates, provide business-related updates to users, advertise, and/or take other action. [0036] online network 118 includes a service that uses a machine learning model to generate a set of relevance scores representing the compatibility of a user with a set of jobs (or the compatibility of a set of users as candidates for a job). The service receives a scoring request in response to the user's job search (or a recruiter's search for candidates matching a job), the user accessing a job recommendation component in online network 118, and/or the user otherwise interacting with job-related functionality in online network 118. To improve the accuracy of the scores, the service inputs, into the machine learning model, features that reflect the latest activity by the user (or recruiter), such as features representing the user's searches, clicks, likes, dislikes, and/or other actions performed in the same user session as the user's interaction with the job-related functionality. In response to the inputted features, the machine learning model calculates the relevance scores between the user and jobs (or a set of candidates and a job) in a real-time or near-real-time basis (e.g., with a latency that is within a limit specified in a service level agreement (SLA) for the service). The service returns the relevance scores in a response to the scoring request, and online network 118 outputs, to the user, a ranking of jobs (or candidates) by descending relevance score as search results, recommendations, and/or other representations of the jobs (or candidates). [0037] Those skilled in the art will appreciate that online network 118 may perform on-demand scoring and/or ranking related to other types or combinations of entities. For example, online network 118 may receive one or more scoring requests for relevance scores between a user and content items created and/or shared by other users of the online network 118. These content items include, but are not limited to, posts, articles, comments, updates, and/or videos. The scoring request(s) may be generated in response to the user accessing a content feed in a homepage, landing page, and/or another part of online network 118. In response to the scoring request(s), a first set of machine learning models 110 is executed to generate recommendations of specific types of content (e.g., network updates, jobs, articles, courses, advertisements, posts, products, connections, products, etc.) for the user based on attributes of the user, recent activity by the user and/or similar users, and/or attributes of the content. The first set of machine learning models 110 output scores representing predictions of the user's level of interest in the content, and a subset of content items with high scores from the first set of machine learning models 110 is selected. Features associated with the user and the selected content items are inputted into one or more additional machine learning models 110, and the additional machine learning model(s) generate an additional set of scores representing predicted likelihoods of the user performing certain actions on (e.g., clicking, liking, commenting on, sharing, etc.) the selected content items. The content items are ranked by descending score, and one or more rules, filters, and/or exceptions are used to update the ranking (e.g., based on business requirements, member preferences, impression discounting, diversification of content, and/or other goals, strategies, or priorities). Finally, the updated ranking is displayed in a content feed for the user. Because the content items are scored and ranked in real-time or near-real-time, the content feed is able to reflect the latest content posted to online network 118 and/or user interactions with the content in online network 118.)
The reason and rationale to combine McKenna and Chang is the same as recited above.
As per Claim 23, McKenna teaches: (New) The method of claim 1, further comprising:
associating, in the knowledge base table, the combination of the first input feature and the second input feature with a model-derived explanation indicating resolution of the inconsistent contributions within the first value range. (in at least [0085] The dealer user may add fraudulent information to the applications rather than the borrower user (e.g., to increase the likelihood of approval of a loan from a lender user). In some examples, this type of fraud may affect a second level score (e.g., a dealer user application score) more than a first level score (e.g., a borrower user application score). The discrepancy module 140 may be configured to identify a higher rate of fraud across the plurality of applications that originate from the dealer user and update the application score to identify a higher likelihood of fraud for applications originating from the dealer user. [0092] one or more features may influence the application score above a particular threshold to identify a potential risk or fraud. The features may correspond with a reason code that indicates an amount the feature can affect the application score and/or a reason that the feature affects the application score the way that it does. In some examples, the potential reason may be user defined (e.g., an administrator of the service may define reasons for particular features when seen in isolation or in combination with other features). In some examples, the one or more features may be grouped into categories (sometimes referred to as factor groups). Examples of factor groups include income, employment, identity, or the like. In each factor group, one or more features may be identified in order to be used to determine reason codes. [0162] Feature extraction system 320 may also output one or more features to discrepancy detection system 360. Discrepancy detection system 360 may include scorecard models and/or expert rules designed to flag inconsistent and/or out-of-pattern data values within the outputs of feature extraction system 320. (i.e. inconsistent contributions within the first value range)One example of this type of data value discrepancy may be a loan amount substantially greater than the list price of the collateral. For another example, if the borrower's stated income exceeds the known average income for the borrower's residential area by a certain percentage. Discrepancy detection system 360 may output a result to application scoring system 340, consortium data lookup system 326, and/or third-party data lookup system 322. [0192] FIG. 7 illustrates a report for indicating a score according to an embodiment of the disclosure. In illustration 700, the report may include a score for a loan application from a lender user device. The report may comprise application data, including an application ID, location associated with the loan application, loan amount, loan term, car make, car model, dealer ID, and the like. [0193] The report further includes a score (illustrated as score “998”), which may be calculated as described above. The report may further include a risk level (illustrated as “high”). The risk level may be determined by comparing the score to one or more thresholds. For example, if the score is above the threshold, a risk level of “high” may be determined. If the score is below the threshold, a risk level of “low” may be determined. Each threshold may be associated with a different level of risk (e.g., low, medium, and high). When there are more than one threshold, a threshold may be defined as between two thresholds. [0194] The report further includes reason codes. A reason code may comprise information corresponding to a feature that contributes above a particular threshold to the score. (i.e. associating) [0195] The application report may further include actions. An action may be mapped to a reason code. In some examples, the action may suggest an action for a party of the loan application to perform to increase the likelihood that the loan application is not fraudulent. (i.e. resolving, resolution) [0216] FIG. 10 illustrates a report (i.e. combination) for indicating a score according to an embodiment of the disclosure. In illustration 1000, the report may include a score for a dealer user associated with a lender user. The report may comprise aggregated application data and/or application scores by dealer identifier. The report may also comprise dealer information associated with a loan application, including dealer ID, dealer name, location associated with the dealer, a phone number for the dealer, a volume of applications for the domain in a particular amount of time. [0217] The dealer report further includes a second level score (e.g., “997”), which may be calculated as described herein. The application report may further include a risk level (e.g., “high”). The risk level may be determined by comparing the second level score to one or more thresholds, each threshold associated with a different level of risk (e.g., high, medium, and low). [0227] At 1150, one or more actions may be determined. For example, the fraud detection computer system 120 may determine one or more actions for the plurality of applications based at least in part on the second score. [0228] At 1160, the application score, reason code(s), and action(s) may be provided. (i.e. combination of the first input feature and the second input feature) For example, the fraud detection computer system 120 may provide the second score, the one or more reason codes, and the one or more actions (i.e. model-derived explanation indicating resolution) to the dealer user device or a lender user device via a communication network)
As per Claim 11-16 for a system (see at least McKenna [0247]), respectively, substantially recite the subject matter of Claim 1-6, 21-23 and are rejected based on the same reasoning and rationale.
As per Claim 19 for computer-readable media (see at least McKenna [0248]), respectively, substantially recite the subject matter of Claim 1 and are rejected based on the same reasoning and rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PO HAN MAX LEE whose telephone number is (571)272-3821. The examiner can normally be reached on Mon-Thurs 8:00 am - 7:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rutao Wu can be reached on (571) 272-6045. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PO HAN LEE/Primary Examiner, Art Unit 3623
Prosecution Insights