DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1, 10, 15, 17 are currently amended. Claims 7 – 9 have been canceled. Claims 22 – 23 are new. Claims 1 – 6, 10 – 23 are pending and examined herein.
Claims 1 – 6, 10 – 23 are rejected under 35 U.S.C. 101.
Claims 1 – 6, 10 – 23 are rejected under 35 U.S.C. 103.
Response to Amendment
The amendment filed February 18th, 2026 has been entered. Claims 1, 10, 15, 17 are currently amended. Claims 7 – 9 have been canceled. Claims 22 – 23 are new. Claims 1 – 6, 10 – 23 are pending and examined herein.
Response to Arguments
Applicant's arguments filed February 18th, 2026 regarding the 35 U.S.C. 101 rejection of claims 1 – 6, 10 – 23 have been fully considered but they are not persuasive. Applicant argues, on pages 9 - 11, that the claims are directed to improvements in computer related technology because the claims recite training a neural network by dropping variables during training in a manner intended to make the network more robust to later variable deprecation. However, the claims as amended, still recites training a neural network using variable data, determining a risk assessment decision for a user based on that risk prediction. Under a broadest reasonable interpretation, these limitations are directed to evaluating information and making a judgement based on that information, which is a mental process and therefore an abstract idea. Although claim 1 recites determining probabilities of deprecation for variables, randomly determining which variables are dropped based on those probabilities, and setting dropped input values to zero, these limitations merely describe how data is selected and omitted during training for the claimed risk prediction process. The claim does not recite any specific improvement to the functioning of the computer or neural network itself. Rather, the neural network is recited at a high level of generality as a tool used to carry out the claimed analysis.
The additional elements identified by applicant do not integrate this abstract idea into a practical application. At most, the alleged benefit is improved robustness or accuracy in the resulting risk prediction when certain variables later become unavailable, which is an improvement in the result of the abstract idea rather than an improvement in the underlying technology itself. The additional elements therefore amount to no more than using generic computer components and a generic neural network to perform the abstract idea. Further, claim 1 does not recite significantly more than the abstract idea. The additional elements, considered individually and as an ordered combination, merely refine how the abstract idea is carried out and do not provide an inventive concept sufficient to transform the claim into patent eligible subject matter.
The same reasoning applies to independent claims 10, 17 and all the dependent claims. The additional limitations relating to likelihoods of deprecation, zero probability variables, or preventing certain variables from being dropped likewise pertain to how variable data is selected or omitted in carrying out the claimed risk prediction and risk assessment process, and do not recite a specific technological improvement sufficient to render the claims patent eligible. Therefore, claims 1- 6, 10 – 23 remain ineligible under 35 U.S.C. § 101.
Applicant's arguments filed February 18th, 2026 regarding the rejections under 35 U.S.C. 103 have been fully considered and are persuasive. The cited references do not fairly teach or suggest the claim as amended. However, new references, Hinton (U.S. Pub. 9406017 B2) and Chang et al. (NPL: ”Dropout Feature Ranking for Deep Learning Models”) are introduced in the below 35 U.S.C. 103 rejection to teach the new features.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 – 6, 10 – 23 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
MPEP § 2109(III) sets out steps for evaluating whether a claim is drawn to patent-eligible subject matter. The analysis of claims 1 – 6, 10 – 23, in accordance with these steps, follows.
Step 1 Analysis:
Step 1 is to determine whether the claim is directed to a statutory category (process, machine, manufacture, or composition of matter.
Claims 1 – 6, 21 – 23 are directed to a method, meaning that it is directed to the statutory category of process. Claims 10 - 16 are directed to a method for training, which is also the statutory category of process. Claims 17 - 20 are directed to a non-transitory computer-readable medium storing instruction, which can be an article of manufacture.
Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis:
Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101.
Regarding claim 1, the following claim elements are abstract ideas:
determining probabilities of deprecation for the variables provided as input to the input nodes (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components. Also, determining probabilities of deprecation for the variables merely recites mathematical calculation, which is mathematical concept.)
randomly determining a set of variables to be dropped as input to the input nodes for the training step based on the determined probabilities of deprecation; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components. Also, randomly determining a set of variables to be dropped could recite mathematical calculation, which is mathematical concept.)
setting input values of the set of variables to be dropped to zero for the training step; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
determining a risk prediction associated with the specified operation based on the specified dataset; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
determining the specified risk assessment decision for the specified user based on the risk prediction. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
training a neural network to determine risk assessment decisions for operations associated with users based on datasets of variables, wherein the neural network includes one or more input nodes feeding into an intermediate layer with one or more intermediate nodes, and wherein the training includes dropping a portion of the variables provided as input to the input nodes of the neural network during a portion of the training, wherein dropping the portion of the variables provided as input to the input nodes includes, for a training step in the training; (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
receiving, by a computer system implementing the trained neural network, a specified request to determine a specified risk assessment decision for a specified operation associated with a specified user, wherein the specified request includes a specified dataset of variables associated with the specified user; (This is mere data gathering, an insignificant extra solution activity, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
providing the specified dataset to the trained neural network; (This is mere transmitting data, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
by the neural network, by the computer system (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 2, the rejection of claim 1 is incorporated herein. Further, claim 2 recites the following abstract ideas:
wherein the risk prediction is adjusted based on a dropped variable factor (The adjusting prediction with factor is merely mathematical calculation, which is mathematical concept.)
wherein the dropped variable factor is based on a number of variables in the portion of variables dropped during the portion of the training. (This is merely mathematical relationship expressed in words, which is mathematical concept.)
Claim 2 does not recite additional elements.
Regarding claims 3 and 4, the rejection of claim 2 is incorporated herein.
Claim 3 further recites the following abstract idea:
the method further comprising adjusting the risk prediction based on the dropped variable factor. (The adjusting prediction with factor is merely mathematical calculation, which is mathematical concept.)
Claim 3 further recites following additional element
wherein the specified dataset has no deprecated variables (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Claim 4 further recites the following abstract idea:
the method further comprising adjusting the risk prediction based on both the dropped variable factor and a deprecated variable factor (The adjusting prediction with factor is merely mathematical calculation, which is mathematical concept.)
Claim 4 further recites following additional elements
wherein the specified dataset has a specified number of deprecated variables (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
, wherein the deprecated variable factor is based on the specified number of deprecated variables (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claims 5 - 6, the rejection of claim 1 is incorporated herein.
Claim 5 further recites the following abstract ideas:
to generate a predictive score indicative of whether an unclassified item corresponds to at least one classification category based on the values for the set of variables and the known labels; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
for determining a risk prediction output for an unknown dataset of variables. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
Claim 5 further recites the following additional elements:
training the neural network, with a training dataset that indicates values for a set of variables corresponding to one or more classification categories and known labels for one or more subsets of the training data set, (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
and generating a set of trained parameters (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Claim 6 further recites the following abstract idea:
wherein dropping the portion of the variables provided as input to the input nodes includes setting input values of the variables to zero. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 6 does not recite additional elements.
Regarding claim 10, the following claim elements are abstract ideas:
training the neural network to generate a predictive score indicative of whether an unclassified item corresponds to at least one classification category based on the values for the set of variables and the known labels; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components. In addition, generating score based on values could fall under a mathematical concept.)
generating probabilities for deprecation of variables in the set of variables based on likelihoods of specific variables being deprecated from the training dataset, wherein at least one variable has a probability of zero; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components. Also, generating probabilities for deprecation of variables based on likelihoods and at least one having a probability of zero merely recites mathematical relationship, which is mathematical concept.)
determining a subset of variables to be dropped based on the generated probabilities, wherein the at least one variable having the probability of zero is inhibited from being part of the subset of variables; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components. Also, determining a subset of variables to be dropped based on the generated probabilities recites mathematical relationship, which is mathematical concept.)
wherein at least one of the parameters is a dropped variable factor for adjusting the risk prediction output, (The adjusting prediction with factor is merely mathematical calculation, which is mathematical concept.)
the dropped variable factor being determined based on the predetermined number of variables in the dropped subset of variables. (This is merely mathematical relationship expressed in words, which is mathematical concept.)
Regarding claim 10, the following claim elements are additional elements:
accessing, by a neural network implemented on a computer system, a training dataset that indicates values for a set of variables corresponding to one or more classification categories and known labels for one or more subsets of the training dataset (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
wherein the set of variables are associated with assessments of risk; (Using variables associated with assessments of risk while dealing with risk analysis is generally linking to a particular technological environment or field of use. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(h). Therefore, this does not amount to significantly more than the judicial exception.)
,and wherein the neural network includes one or more input nodes feeding into an intermediate layer with one or more intermediate nodes; (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
dropping, for a specified period of time during the training, the subset of variables provided as input to the input nodes of the neural network, wherein the subset of variables includes a predetermined number of variables; (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
and generating, for the neural network, a set of trained parameters for determining a risk prediction output for an unknown dataset of variables, (This falls under mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 11, the rejection of claim 10 is incorporated herein. Further, claim 11 recites the following abstract idea:
wherein the risk prediction output is multiplied by the dropped variable factor to adjust the risk prediction output. (Multiplying output by the factor to adjust output recites mathematical calculation, which is mathematical concept.)
Claim 11 does not recite additional elements.
Regarding claim 12, the rejection of claim 11 is incorporated herein. Further, claim 12 recites the following abstract idea:
wherein the dropped variable factor is based on a fraction determined as a number of variables in the subset of variables divided by a total number of variables in the set of variables. (Determining dropped variable factor by dividing a number of variables in subset with total number of variables in subset recites a numerical equation, which is mathematical concept.)
Claim 12 does not recite additional elements.
Claim 13 recites substantially similar subject matter to claim 6 respectively and is rejected with the same rationale, mutatis mutandis.
Regarding claim 14, the rejection of claim 10 is incorporated herein. Further, claim 14 further recites following abstract ideas:
randomly determining variables from the subset of variables to be dropped (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
Claim 14 further recites following additional element:
during the specified period of time. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 15, the rejection of claim 10 is incorporated herein. Further, claim 15 further recites following abstract ideas:
wherein the subset of variables to be dropped is determined using a randomization based on the generated probabilities (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components. Also, using a randomization based on the generated probabilities recite mathematical calculation, which is mathematical concept.)
Claim 15 does not recite additional elements.
Regarding claim 16, the rejection of claim 10 is incorporated herein. Further, claim 16 further recites following abstract ideas:
wherein the dataset includes a deprecated set of variables associated with the user; (Associating deprecated variable with a user is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
the operation based on the deprecated set of variables; (Associating an operation with the deprecated variables is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
The rest of claim 16 recites substantially similar subject matter to claim 1 and 4 respectively and are rejected with the same rationale, mutatis mutandis.
Regarding claim 17, the following claim element is abstract idea:
deprecating one or more variables from the dataset to generate a deprecated dataset; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components. Deprecating certain variables could also be done with mathematical calculations, which is mathematical concept.)
The rest of claim 17 recites substantially similar subject matter to claim 1 respectively and is rejected with the same rationale, mutatis mutandis.
Regarding claims 18 and 19, the rejection of claim 17 is incorporated herein.
Claim 18 further recites the following abstract idea:
deprecating the variables in the deprecated dataset includes assigning predetermined input values to the variables. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
Claim 18 does not recite additional element.
Claim 19 recites substantially similar subject matter to claim 2, 4 respectively and is rejected with the same rationale, mutatis mutandis.
Regarding claim 20, the rejection of claim 19 is incorporated herein. Further, claim 20 recites the following abstract ideas:
adjusting the risk prediction includes multiplying the risk prediction by the number of variables in the portion of variables dropped as input to the input nodes and a total number of variables in the dataset before deprecation and dividing the risk prediction by a total number of variables provided as input to the input nodes during training and a number of variables after deprecation. (The whole claim element is merely reciting a mathematical formula, which is mathematical concept.)
Claim 20 does not recite additional element.
Regarding claim 21, the rejection of claim 1 is incorporated herein. Further, claim 21 recites the following additional element:
wherein dropping the portion of the variables provided as input to the input nodes causes intermediate nodes in the intermediate layer having edges connected to the input nodes with dropped variables to compensate for a lack of the edges to during training. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 22, the rejection of claim 1 is incorporated herein. Further, claim 22 recites the following abstract idea:
wherein at least one variable provided as input has a probability of deprecation of zero. (Having at least one input having a probability of deprecation of zero is merely mathematical relationship, which is mathematical concept.)
Claim 22 does not recite additional element
Regarding claim 23, the rejection of claim 22 is incorporated herein. Further, claim 22 recites the following additional element:
wherein the at least one variable having the probability of deprecation of zero is a variable essential for determining the risk assessment decision. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 – 3, 5 – 6, 10 – 15, 17 – 18, 21 – 23 are rejected under 35 U.S.C. 103 as being unpatentable over Turner et al. (U.S. Pub. 2018/0068219) in view of Hinton (U.S. Pub. 9406017 B2), Chang et al. (NPL: ”Dropout Feature Ranking for Deep Learning Models”), further in view of Goel et al. (U.S. Pub. 2016/0307098 A1).
Regarding Claim 1, Turner teaches
A method, comprising: training a neural network to determine risk assessment decisions for operations associated with users based on datasets of variables, ([0017] of Turner states “Optimizing the neural network in this manner can allow the neural network to be used both for accurately determining risk indicators using predictor variables and determining adverse action codes for the predictor variables.”)
wherein the neural network includes one or more input nodes feeding into an intermediate layer with one or more intermediate nodes, and ([0056] of Turner states “For example, the single-layer neural network 400 includes inputs X1 through Xn. The input nodes X1 through Xn represent predictor variables, which can be obtained as inputs 103 1 through 103 n (e.g., from predictor variable database 103 of FIG. 1). The node Y in FIG. 4 represents a risk indicator that can be determined using the predictor variables. The example of a single-layer neural network 400 depicted in FIG. 4 includes a single layer of hidden nodes H1 through Hm which represent intermediate values. But neural networks with any number of hidden layers can be optimized using the operations described herein.” [0062] of Turner states “For example, FIG. 5 is a diagram depicting an example of multi-layer neural network 500 that can be generated and optimized by the risk assessment application 102 of FIGS. 1 and 2. In the example depicted in FIG. 5, the multi-layer neural network 500 is a feed-forward neural network. The neural network 500 includes n input nodes that represent predictor variables, mk hidden nodes in the kth hidden layer, and p hidden layers.”)
receiving, by a computer system implementing the trained neural network, a specified request to determine a specified risk assessment decision for a specified operation associated with a specified user, wherein the specified request includes a specified dataset of variables associated with the specified user; ([0023] of Turner states "The user device 108 may include any computing device that can communicate with the computing environment 100. For example, the user device 108 may send data to the computing environment or a device in the computing environment (e.g., the risk assessment application 102 or the predictor variable database 103) to be stored or processed.” And [0024] of Turner states “Communication within the computing environment may occur on a network. The risk assessment application, the user device, and the predictor variable database may communicate with each other”)
providing the specified dataset to the trained neural network; ([0023] of Turner states “The user device may send data to the computing environment (e.g. The predictor variable database) to be stored or processed”. Fig. 2 of Turner shows these data being sent to network 110 and network sends them to Risk assessment server 104 containing risk assessment application 102. [0025] of Turner states “Executing the instructions causes the risk assessment application 102 to generate a neural network and optimize the neural network for assessing risk.” And [0026] of Turner states “The risk assessment application 102 can use the predictor variable module 202 for obtaining or receiving data.”)
determining, by the neural network, a risk prediction associated with the specified operation based on the specified dataset; ([0017] of Turner states “optimized neural network can be used for both determining a credit score associated with an entity (e.g., an individual or business) based on predictor variables associated with the entity. A predictor variable can be any variable predictive of risk that is associated with an entity.”)
determining, by the computer system, the specified risk assessment decision for the specified user based on the risk prediction. ([0047] of Turner states “the risk assessment application 102 can use an optimized neural network to provide recommendations to a consumer based on adverse action codes. The recommendations may indicate one or more actions that the consumer can take to improve the change the risk indicator (e.g., improve a credit score)”.)
Turner does not explicitly teach that
wherein the training includes dropping a portion of the variables provided as input to the input nodes of the neural network during a portion of the training, wherein dropping the portion of the variables provided as input to the input nodes includes, for a training step in the training:
determining probabilities of deprecation for the variables provided as input to the input nodes;
randomly determining a set of variables to be dropped as input to the input nodes for the training step based on the determined probabilities of deprecation;
and setting input values of the set of variables to be dropped to zero for the training step;
However, Hinton teaches that
and wherein the training includes dropping a portion of the variables provided as input to the input nodes of the neural network during a portion of the training, wherein dropping the portion of the variables provided as input to the input nodes includes, for a training step in the training: (Column 3 Lines 39 – 47, 55 - 67 of Hinton states “In an embodiment, the switch (108) is linked to all feature detectors of the hidden layers. In another embodiment, the switch (108) is linked to all feature detectors of the input layers. In yet another embodiment, the switch (108) may be linked to all feature detectors in both the hidden and input layers. In yet further embodiments, the switch (108) may be linked to the feature detectors of a subset of the input and hidden layers. In another aspect, the switch may be connected to all hidden layers that are fully connected layers… In a more specific embodiment, feature detectors in hidden layers may be selectively disabled with probability 0.5 (that is, on average, each feature detector will be enabled for half of the training cases and disabled for the other half of the training cases) while feature detectors of input layers are disabled with probability 0.2 (that is, on average, these feature detectors will be enabled for 80% of training cases and disabled for 20% of training cases). Therefore, in this example, for each training case, each hidden layer feature detector is randomly omitted from the network with a probability of 0.5 and each input layer feature detector is randomly omitted from the network with a probability 0.2, so each hidden or input feature detector cannot rely on other hidden or input feature detectors being present.” Column 2 Lines 57 – 62 of Hinton states “It has been found that overfitting may be reduced by selectively disabling a randomly (or pseudorandomly) selected subset of feature detectors in a neural network during each training case of the training stage, and adapting the weights of each feature detector accordingly during application of the neural network in the test stage.” Applying Hinton’s input layer dropout to Turner’s risk assessment neural network would’ve been obvious combination.)
determining probabilities of [disablement]… provided as input to the input nodes; (Column 3 Lines 48 – 54 of Hinton states “Referring now to FIG. 2, during the training stage, a plurality of training cases are input, one at a time, to the neural network in order to train the neural network. For each such training case, the switch selectively disables a subset of the feature detectors to which it is linked (200). In particular embodiments, the switch is configured to disable each such feature detector in accordance with a preconfigured probability.” Column 4 Lines 64 – 67 of Hinton states “In further examples, for datasets in which the required input-output mapping has several suitably different regimes, performance may be further improved by adapting the preconfigured probabilities to be a learned function of the input.”)
randomly determining a set of variables to be dropped as input to the input nodes for the training step based on the determined probabilities of [disablement]; (Column 3 Lines 28 – 38 of Hinton states “A switch (108) is linked to at least a subset of the feature detectors. The switch is operable to selectively disable each feature detector in the neural network to which it is linked, with a learned or preconfigured probability. A random number generator (110) is linked to the switch and provides the switch with a random value that enables the switch to selectively disable each linked feature detector. The possible values generated by the random number generator (110) each correspond to a decision of whether to disable any particular feature detector in accordance with the preconfigured probability.”)
Chang further teaches that
probabilities … for the variables provided as input to the input nodes (Pg. 1 Introduction of Chang states “In this work we use the Dropout concept on the input feature layer and optimize the corresponding feature-wise dropout rate. Since each feature is removed stochastically, our method creates a similar effect to feature bagging (Ho, 1995) and manages to rank correlated features better than other non-bagging methods such as LASSO. We compare our method to Random Forest (RF), LASSO, ElasticNet, Marginal ranking and several techniques to derive importance in DNN such as Deep Feature Selection and various heuristics.” Pg. 2 3 Methods of Chang states “To achieve minimum loss, the Dropout FR model should learn small dropout rate for features that are important for correct target prediction by the analyzed model M, while increasing the dropout rate for the rest of unimportant features. Specifically, given D features, we set a variational mask distribution qθ(z) = Dj=1 q(zj|θj) = D j=1 Bern(zj|θj) as a fully factorized distribution. This gives us a feature-wise dropout rate θj where magnitude indicates the importance of feature j” Hinton and Chang together teaches determining feature specific probabilities for variables provided as input to the input nodes, which corresponds to the claimed “probabilities of deprecation” as features are in form of unusable state. )
and setting input values of the set of variables to be dropped to zero (Pg. 1 1 Introduction of Chang states “Dropout is an effective technique commonly used to regularize neural networks by randomly removing a subset of hidden node values and setting them to 0. In this work we use the Dropout concept on the input feature layer and optimize the corresponding feature-wise dropout rate.”)
Goel teaches that
and setting input values of the set of variables to be dropped to zero for the training step; ([0066] of Goel states “For example, given a dropout probability of pd for a node, dropout may be applied during the forward pass of training by randomly setting the input or output of the node to zero with probability pd. This may be performed using the sampler 304 to sample from, for example, a uniform distribution ε(0,1), and then setting the output to zero if the sample is less than pd.”)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Turner, Hinton, Chang, and Goel. Turner teaches training a neural network on predictor variables and known outcome categories to generate a risk indicator for risk assessment. Hinton teaches randomly disabling input layer feature detector during training according to preconfigured probabilities to reduce overfitting and improve generalization. Chang teaches assigning different dropout rates to different input features to control dropout on a feature basis at the input layer. Goel teaches implementing dropout by setting node input or output values to zero and also teaches that dropout probability could be set to zero. One of ordinary skill in the art would have been motivated to incorporate the teachings of Hinton, Chang, and Goel into Turner so that Turner’s predictor variable neural network could be trained using probability based, feature specific dropout implemented through zeroing of dropped inputs, which would have predictably improved robustness and generalization through the use of known dropout techniques in a risk assessment application.
Regarding claim 2, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
the risk prediction is adjusted based on a dropped variable factor, wherein the dropped variable factor is based on a number of variables in the portion of variables dropped during the portion of the training. (Column 4 Lines 9 – 18 “Normalization comprises reducing the outgoing weights of each feature detector or input by multiplying them by the probability that the feature detector or input was not disabled. In an example, if the feature detectors of each hidden layer were selectively disabled with probability 0.5 in the training stage, the outgoing weights are halved for the test case since approximately twice as many feature detectors will be enabled. A similar approach is applied to the input layers.” It would’ve been obvious to apply this normalization to Turner’s risk prediction output.)
Regarding claim 3, the rejection of claim 2 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
wherein the specified dataset has no deprecated variables, the method further comprising adjusting the risk prediction based on the dropped variable factor. (Column 4 Lines 7 – 13 of Hinton states “Once the training set has been learned by the neural network, the switch may enable all feature detectors and normalize their outgoing weights (204). Normalization comprises reducing the outgoing weights of each feature detector or input by multiplying them by the probability that the feature detector or input was not disabled.” Hinton’s normalization is applied at every inference step. The scaling factor is embedded in trained weights and applies universally regardless of whether any inputs are actually missing from the production dataset.)
Regarding claim 5, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
training the neural network, with a training dataset that indicates values for a set of variables corresponding to one or more classification categories and known labels for one or more subsets of the training dataset, to generate a predictive score indicative of whether an unclassified item corresponds to at least one classification category based on the values for the set of variables and the known labels; and generating a set of trained parameters for determining a risk prediction output for an unknown dataset of variables. ([0015] of Turner states ”In some aspects, a risk assessment application can generate or optimize a neural network for risk assessment. For example, the risk assessment application can receive various predictor variables and determine a relationship between each predictor variable and an outcome such as, but not limited to, a positive outcome indicating that a condition is satisfied or a negative outcome indicating that the condition is not satisfied. The risk assessment application can generate the neural network using the relationship between each predictor variable and the outcome. The neural network can then be used to determine a relationship between each of the predictor variables and a risk indicator.” Turner’s training uses labeled outcomes, as positive or negative, to train the neural network to classify new items, which maps to known label supervised training for predictive score generation in this limitation.)
Regarding claim 6, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
dropping the portion of the variables provided as input to the input nodes includes setting input values of the variables to zero. ([0005] of Goel states “randomly zeroing, or “dropping out” a fixed percentage of the inputs or outputs of a given node or layer in the neural network (e.g., dropout training) for each of one or more training sets (including a set of inputs and corresponding expected outputs) to tune network parameters” [0054] of Goel states “the dropout training performed by the dropping mechanism 302 may include, for each new training case for one or more models, randomly zeroing each dimension of the input to the model, node, or input later with probability pd, where pd is the dropout rate.” [0066] of Goel states “For example, given a dropout probability of pd for a node, dropout may be applied during the forward pass of training by randomly setting the input or output of the node to zero with probability pd” Pg. 1 1 Introduction of Chang states “Dropout is an effective technique commonly used to regularize neural networks by randomly removing a subset of hidden node values and setting them to 0. In this work we use the Dropout concept on the input feature layer and optimize the corresponding feature-wise dropout rate.”)
Regarding claim 10, the combination of Turner, Hinton, Chang, and Goel teaches
accessing, by a neural network implemented on a computer system, a training dataset that indicates values for a set of variables corresponding to one or more classification categories and known labels for one or more subsets of the training dataset, wherein the set of variables are associated with assessments of risk, and wherein the neural network includes one or more input nodes feeding into an intermediate layer with one or more intermediate nodes; ([0015] of Turner states “For example, the risk assessment application can receive various predictor variables and determine a relationship between each predictor variable and an outcome such as, but not limited to, a positive outcome indicating that a condition is satisfied or a negative outcome indicating that the condition is not satisfied. The risk assessment application can generate the neural network using the relationship between each predictor variable and the outcome. The neural network can then be used to determine a relationship between each of the predictor variables and a risk indicator.” [0049] of Turner states “Examples of predictor variables can include data associated with an entity that describes prior actions or transactions involving the entity (e.g., information that can be obtained from credit files or records, financial records, consumer records, or other data about the activities or characteristics of the entity), behavioral traits of the entity, demographic traits of the entity, or any other traits of that may be used to predict risks associated with the entity. In some aspects, predictor variables can be obtained from credit files, financial records, consumer records, etc.” Turner teaches a training dataset of predictor variables mapped to known outcome labels, used to train the neural network for risk assessment. )
training the neural network to generate a predictive score indicative of whether an unclassified item corresponds to at least one classification category based on the values for the set of variables and the known labels; ([0060] of Turner states ”In some aspects, the risk assessment application (e.g., the risk assessment application 102 of FIGS. 1 and 2) can use the single-layer neural network 400 to determine a value for the risk indicator Y. As an example, in credit decision applications, the risk indicator Y may be a modeled probability of a binary random variable associated with the risk indicator and can be continuous with respect to the predictor variables X1 through Xn. In some aspects, the risk assessment application can use the feed-forward neural network 400 having a single hidden layer that is monotonic with respect to each predictor variable used in the neural network for risk assessment. The single-layer neural network 400 can be used by the risk assessment application to determine a value for a continuous random variable P(Y=1) that represents a risk indicator or other output probability.” [0054] of Turner states “The neural network can include input nodes corresponding to a set of predictor variables having a monotonic relationship with an associated odds index (e.g., the set of predictor variables identified in block 304). For example, the risk assessment application can generate the neural network such that the neural network models the monotonic relationship between the set of predictor variables and one or more odds indices.” Turner teaches training the neural network using labeled outcome data to generate a predictive score indicating whether an entity corresponds to risk category.)
generating probabilities for deprecation of variables in the set of variables based on likelihoods of specific variables being deprecated from the training dataset, wherein at least one variable has a probability of zero; (Column 3 Lines 48 – 54 of Hinton states “Referring now to FIG. 2, during the training stage, a plurality of training cases are input, one at a time, to the neural network in order to train the neural network. For each such training case, the switch selectively disables a subset of the feature detectors to which it is linked (200). In particular embodiments, the switch is configured to disable each such feature detector in accordance with a preconfigured probability.” Pg. 2 3 Methods of Chang states “To achieve minimum loss, the Dropout FR model should learn small dropout rate for features that are important for correct target prediction by the analyzed model M, while increasing the dropout rate for the rest of unimportant features. Specifically, given D features, we set a variational mask distribution qθ(z) = Dj=1 q(zj|θj) = D j=1 Bern(zj|θj) as a fully factorized distribution. This gives us a feature-wise dropout rate θj where magnitude indicates the importance of feature j” [0027] of Goel states “According to particularly useful embodiments, annealing the dropout rate from a high initial value to a zero or a low non-zero final value over the course of annealed dropout training may substantially improve word error rate (WER) when training neural networks based on acoustic models (e.g., for automatic speech recognition (ASR)), and may significantly reduce WER over-training, which may occur when not employing dropout, and with conventional dropout training systems and methods.” Assigning zero to the most essential variable is obvious with respect to Hinton, Chang, and Goel. )
determining a subset of variables to be dropped based on the generated probabilities, wherein the at least one variable having the probability of zero is inhibited from being part of the subset of variables; (Column 3 Lines 28 – 38 of Hinton states “A switch (108) is linked to at least a subset of the feature detectors. The switch is operable to selectively disable each feature detector in the neural network to which it is linked, with a learned or preconfigured probability. A random number generator (110) is linked to the switch and provides the switch with a random value that enables the switch to selectively disable each linked feature detector. The possible values generated by the random number generator (110) each correspond to a decision of whether to disable any particular feature detector in accordance with the preconfigured probability.” Column 3 Lines 48 – 54 of Hinton states “Referring now to FIG. 2, during the training stage, a plurality of training cases are input, one at a time, to the neural network in order to train the neural network. For each such training case, the switch selectively disables a subset of the feature detectors to which it is linked (200). In particular embodiments, the switch is configured to disable each such feature detector in accordance with a preconfigured probability.” Feature with probability of 0 is never selected by Hinton. )
dropping, for a specified period of time during the training, the subset of variables provided as input to the input nodes of the neural network, wherein the subset of variables includes a predetermined number of variables; (Column 2 Lines 66 – Column 3 Lines 3 of Hinton states “Although it is preferred that the selective disabling of feature detectors be changed for each training case, it is contemplated herein that disabling of particular feature detectors may be held constant for a plurality of training cases” The predetermined variables from the subset of variables to be dropped could be held constant for a plurality of cases, which is the specified period of time.)
and generating, for the neural network, a set of trained parameters for determining a risk prediction output for an unknown dataset of variables, wherein at least one of the parameters is a dropped variable factor for adjusting the risk prediction output, the dropped variable factor being determined based on the predetermined number of variables in the dropped subset of variables. (Column 4 Lines 14 – 18 of Hinton states “Normalization comprises reducing the outgoing weights of each feature detector or input by multiplying them by the probability that the feature detector or input was not disabled. In an example, if the feature detectors of each hidden layer were selectively disabled with probability 0.5 in the training stage, the outgoing weights are halved for the test case since approximately twice as many feature detectors will be enabled. A similar approach is applied to the input layers.” The normalization factor is embedded in the trained network’s output weights, which is a trained parameter. Also, the factor is directly determined by the predetermined number of variables in the dropped subset. )
Regarding Claim 11, the rejection of claim 10 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
wherein the risk prediction output is multiplied by the dropped variable factor to adjust the risk prediction output. (Column 4 Lines 14 – 18 of Hinton states “Normalization comprises reducing the outgoing weights of each feature detector or input by multiplying them by the probability that the feature detector or input was not disabled. In an example, if the feature detectors of each hidden layer were selectively disabled with probability 0.5 in the training stage, the outgoing weights are halved for the test case since approximately twice as many feature detectors will be enabled. A similar approach is applied to the input layers.”)
Regarding Claim 12, the rejection of claim 11 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
the dropped variable factor is based on a fraction determined as a number of variables in the subset of variables divided by a total number of variables in the set of variables (Column 4 Lines 7 – 18 of Hinton states “Once the training set has been learned by the neural network, the switch may enable all feature detectors and normalize their outgoing weights (204). Normalization comprises reducing the outgoing weights of each feature detector or input by multiplying them by the probability that the feature detector or input was not disabled. In an example, if the feature detectors of each hidden layer were selectively disabled with probability 0.5 in the training stage, the outgoing weights are halved for the test case since approximately twice as many feature detectors will be enabled. A similar approach is applied to the input layers.” The example from Hinton shows that the factor is computed from the fraction of dropped units, which is mathematically equivalent to the formular here that dropped variable factor = number dropped / total. )
Regarding claim 13, the rejection of claim 10 is incorporated herein. Claim 13 recites substantially similar subject matter as claim 6 respectively, and is rejected with the same rationale, mutatis mutandis.
Regarding claim 14, the rejection of claim 10 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
further comprising randomly determining variables from the subset of variables to be dropped during the specified period of time. (Column 2 Lines 66 – Column 3 Lines 3, Lines 27 – 38 of Hinton states “Although it is preferred that the selective disabling of feature detectors be changed for each training case, it is contemplated herein that disabling of particular feature detectors may be held constant for a plurality of training cases… A switch (108) is linked to at least a subset of the feature detectors. The switch is operable to selectively disable each feature detector in the neural network to which it is linked, with a learned or preconfigured probability. A random number generator (110) is linked to the switch and provides the switch with a random value that enables the switch to selectively disable each linked feature detector. The possible values generated by the random number generator (110) each correspond to a decision of whether to disable any particular feature detector in accordance with the preconfigured probability.” These randomly determined variables from the subset of variables to be dropped could be held constant for a plurality of cases, which is the specified period of time.)
Regarding claim 15, the rejection of claim 10 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
wherein the subset of variables to be dropped is determined using a randomization based on the generated probabilities. (Column 3 Lines 27 – 38 of Hinton states “A switch (108) is linked to at least a subset of the feature detectors. The switch is operable to selectively disable each feature detector in the neural network to which it is linked, with a learned or preconfigured probability. A random number generator (110) is linked to the switch and provides the switch with a random value that enables the switch to selectively disable each linked feature detector. The possible values generated by the random number generator (110) each correspond to a decision of whether to disable any particular feature detector in accordance with the preconfigured probability.” Randomization in Hinton is in accordance with the preconfigured probability. )
Regarding claim 17, the combination of Turner, Hinton, Chang, and Goel teaches
deprecating one or more variables from the dataset to generate a deprecated dataset; (In the applicant’s disclosure, [0041] states “deprecated (e.g., removed). As stated in Turner [0043] “Adjusting the predictor variables can include eliminating the predictor variable from the neural network”. Therefore, variables are eliminated[deprecated] and results in deprecated dataset.)
setting a probability of deprecation for at least one variable provided as input to zero based on the at least one variable being a primary variable for the risk assessment decision (Pg. 2 3 Methods of Chang states “To achieve minimum loss, the Dropout FR model should learn small dropout rate for features that are important for correct target prediction by the analyzed model M, while increasing the dropout rate for the rest of unimportant features. Specifically, given D features, we set a variational mask distribution qθ(z) = Dj=1 q(zj|θj) = D j=1 Bern(zj|θj) as a fully factorized distribution. This gives us a feature-wise dropout rate θj where magnitude indicates the importance of feature j” Once primary variables are identified within Turner’s risk assessment, it would have been obvious to assign a probability of zero, which is the natural lower bound protection of the importance based per feature drop framework by Hinton, Chang, and Goel.)
The rest of claim 17 recite substantially similar subject matter as claim 1 respectively, and is rejected with the same rationale, mutatis mutandis.
Regarding claim 18, the rejection of claim 17 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
deprecating the variables in the deprecated dataset includes assigning predetermined input values to the variables. ([0005] of Goel states “randomly zeroing, or “dropping out” a fixed percentage of the inputs or outputs of a given node or layer in the neural network (e.g., dropout training) for each of one or more training sets (including a set of inputs and corresponding expected outputs) to tune network parameters” [0054] of Goel states “the dropout training performed by the dropping mechanism 302 may include, for each new training case for one or more models, randomly zeroing each dimension of the input to the model, node, or input later with probability pd, where pd is the dropout rate.” [0066] of Goel states “For example, given a dropout probability of pd for a node, dropout may be applied during the forward pass of training by randomly setting the input or output of the node to zero with probability pd” Pg. 1 1 Introduction of Chang states “Dropout is an effective technique commonly used to regularize neural networks by randomly removing a subset of hidden node values and setting them to 0. In this work we use the Dropout concept on the input feature layer and optimize the corresponding feature-wise dropout rate.” A POSITA would understand that deprecating a variable (making variable unavailable) is implemented by assigning it predetermined input value like zero.)
Regarding claim 21, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
wherein dropping the portion of the variables provided as input to the input nodes causes intermediate nodes in the intermediate layer having edges connected to the input nodes with dropped variables to compensate for a lack of the edges to during training. ([0040] of Goel states “A neural network 100 may include a plurality of neurons/nodes 108, and the nodes 108 may communicate using one or more of a plurality of connections 103. The neural network 100 may include a plurality of layers, including, for example, one or more input layers 102, one or more hidden layers 104, and one or more output layers 106.” [0066] of Goel states “For example, given a dropout probability of pd for a node, dropout may be applied during the forward pass of training by randomly setting the input or output of the node to zero with probability pd.” [0076] of Goel states “This may adjust the bias so that it is correct for the subspace implied by the dropout mask. This correction may be subspace specific, and may enable an ensemble of models defined by the dropout procedure (e.g., one model per dropout mask) to properly share their bias parameters, and the overall model to handle any “missing data” scenarios implied by dropout properly” Pg. 2 3 Methods of Chang states “The mechanism is to inject a multiplicative Bernoulli noise for each hidden unit within a neural network. Specifically, during forward pass, for each hidden unit k in layer j a dropout mask zjk ∼ Bern(z|θjk) is sampled. The original hidden node value hjk is then multiplied by this mask hjk = hjkzjk, which stochastically sets the hidden node value to hjk or 0.” It is well known concept to compensate after masking the layer during dropout process.)
Regarding claim 22, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
wherein at least one variable provided as input has a probability of deprecation of zero. (Column 3 Lines 50 – 54 of Hinton states “For each such training case, the switch selectively disables a subset of the feature detectors to which it is linked (200). In particular embodiments, the switch is configured to disable each such feature detector in accordance with a preconfigured probability.” Pg. 2 3 Methods of Chang states “To achieve minimum loss, the Dropout FR model should learn small dropout rate for features that are important for correct target prediction by the analyzed model M, while increasing the dropout rate for the rest of unimportant features. Specifically, given D features, we set a variational mask distribution qθ(z) = D j=1 q(zj|θj) = D j=1 Bern(zj|θj) as a fully factorized distribution. This gives us a feature-wise dropout rate θj where magnitude indicates the importance of feature j…. Fig. 1: Dropout feature ranking diagram. Before training (Left), the dropout rate for each feature is initialized to 0.5. After training (Right), each feature gets a different dropout rate. We then rank all features based on the magnitude of the dropout rate- the lower the magnitude, the higher the rank.” [0023] of Goel states “In an illustrative embodiment, annealing the dropout rate from a high initial value (e.g., 0.5) to zero over the course of training in, for example, situations where there is plentiful data, can substantially improve the quality of the resulting model over conventional systems and methods, and annealed dropout is also highly effective even in limited data scenarios.” Incorporating Hinton, Chang, and Goel’s teaching, at least assigning zero to a variable identified as essential is obvious implementation of the combined framework. )
Regarding claim 23, the rejection of claim 22 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, and Goel teaches
wherein the at least one variable having the probability of deprecation of zero is a variable essential for determining the risk assessment decision. (Pg. 2 3 Methods of Chang states “To achieve minimum loss, the Dropout FR model should learn small dropout rate for features that are important for correct target prediction by the analyzed model M, while increasing the dropout rate for the rest of unimportant features. Specifically, given D features, we set a variational mask distribution qθ(z) = D j=1 q(zj|θj) = D j=1 Bern(zj|θj) as a fully factorized distribution. This gives us a feature-wise dropout rate θj where magnitude indicates the importance of feature j…. Fig. 1: Dropout feature ranking diagram. Before training (Left), the dropout rate for each feature is initialized to 0.5. After training (Right), each feature gets a different dropout rate. We then rank all features based on the magnitude of the dropout rate- the lower the magnitude, the higher the rank.” [0023] of Goel states “In an illustrative embodiment, annealing the dropout rate from a high initial value (e.g., 0.5) to zero over the course of training in, for example, situations where there is plentiful data, can substantially improve the quality of the resulting model over conventional systems and methods, and annealed dropout is also highly effective even in limited data scenarios.” Chang links dropout probability magnitude to feature importance for correct prediction. Assigning zero to a variable identified as essential for the risk assessment decision is obvious with Goel’s lower bound implementation of zero dropout rate. )
Claims 4, 16, 19 – 20 are rejected under 35 U.S.C. 103 as being unpatentable over Turner et al. (U.S. Pub. 2018/0068219) in view of Hinton (U.S. Pub. 9406017 B2), Chang et al. (NPL: ”Dropout Feature Ranking for Deep Learning Models”), Goel et al. (U.S. Pub. 2016/0307098 A1), further in view of John et al. (U.S. Pub. 2022/0058513).
Regarding claim 4, the rejection of claim 2 is incorporated herein. The combination of Turner, Hinton, Chang, and Goel does not explicitly teach
the specified dataset has a specified number of deprecated variables
the method further comprising adjusting the risk prediction based on both the dropped variable factor and a deprecated variable factor, wherein the deprecated variable factor is based on the specified number of deprecated variables.
However, John teaches
the specified dataset has a specified number of deprecated variables ([0042] of John states “the staleness analyzer 280 may calculate a staleness factor 288 based on the stale target feature(s) and the determined importance value 252 of the input category(ies) corresponding to the stale target feature(s)”. Specific amount of stale target categories will be determined through importance value of each categories.)
the method further comprising adjusting the risk prediction based on both the dropped variable factor and a deprecated variable factor, ([0031] of John states “In an embodiment, the model assessment device 150 may provide to assess the veracity of an existing data model such as the data model 216 based on a relevance (e.g., bias, feature gap, manipulation, staleness, or a combination thereof) of data, such as the input feature 218, used by the data model 216 to provide an outcome” and [0019] of John states that the embodiment could be used for “risk assessment scenarios”. Therefore, factors calculated from manipulation[drop] and staleness[deprecation] could be used to provide an outcome [risk prediction].)
wherein the deprecated variable factor is based on the specified number of deprecated variables. ([0043] of John states “As shown in Equation 6, the staleness factor 288 may be calculated as a sum of products of a normalized staleness value of each input category and a corresponding importance value.” Equation 6. Shows that factor will be based on number categories that matches importance value.)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Turner, Hinton, Chang, and Goel with John because both aims for accurate outcome of ML model dealing with predictive analysis. One with ordinary skill in the art would be motivated to incorporate the teachings of John into the combination of Turner, Hinton, Chang, and Goel because manipulating training data could lead to improving the accuracy of the data model (John [0017]). These manipulated and staled factors should be taken account with the outcome for improved accuracy of data model and assess a combined impact of these data (John [0018]). Said person would incorporate teachings of Turner, Hinton, Chang, Goel, and John to adjust prediction with both factors to compensate affected data from drop mechanism.
Regarding claim 16, the rejection of claim 10 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, Goel, and John teaches
wherein the additional dataset includes a deprecated set of variables associated with the specified user; ([0041] of John states “In a first step, the staleness analyzer 280 may determine the input feature 218 as being a target feature based on the importance value 252 of the input category 222 in the production data 212 corresponding to the input feature 218” As staleness factor is calculated on the production data, dataset already includes deprecated set of variables to be measured.)
based on the deprecated set of variables associated with the specified user; ([0041] of John stated “As illustrated in FIG. 8, the production data 212 may include a set of one or more input tables 810 including input categories, namely, “User_ID,” “Address_city,” “First Name,” “Last Name,” “Gender,” “Age,” “Marital Status,” “Occupation,” and “Income”” These values that could potentially be deprecated based on importance value contain data associated with specific user, such as user_id, name or address.)
The rest of claim 16 recite substantially similar subject matter as combination of claims 1 and 4 respectively, and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 19, the rejection of claim 17 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, Goel, and John teaches
adjusting the risk prediction based on a number of deprecated variables in the deprecated dataset ([0043] of John states “As shown in Equation 6, the staleness factor 288 may be calculated as a
sum of products of a normalized staleness value of each input category and a corresponding importance
value.” Equation 6. Shows that factor will be based on number categories that matches importance
value. [0031] of John states “In an embodiment, the model assessment device 150 may provide to assess the veracity of an existing data model such as the data model 216 based on a relevance (e.g., bias, feature gap, manipulation, staleness, or a combination thereof) of data, such as the input feature 218, used by the data model 216 to provide an outcome” and [0019] of John states that the embodiment could be used for “risk assessment scenarios”. Therefore, factors calculated from staleness[deprecation] could be used to provide an outcome [risk prediction].)
and a number of variables in the portion of variables dropped as input to the input nodes. ([0005] of Goel states “Recently, it has been shown that neural network performance may be improved by training the neural network by randomly zeroing, or “dropping out” a fixed percentage of the inputs or outputs of a given node or layer in the neural network (e.g., dropout training) for each of one or more training sets (including a set of inputs and corresponding expected outputs) to tune network parameters (number of layers, number of nodes per layer, number of training iterations, learning rate, etc.).” [0054] of Goel states “In one embodiment, the dropout training performed by the dropping mechanism 302 may include, for each new training case for one or more models, randomly zeroing each dimension of the input to the model, node, or input later with probability pd, where pd is the dropout rate. This is similar to introducing independent, identical, distributed (i.i.d.) Bernoulli multiplicative noise into the model, which masks each input with probability pd.”)
Regarding Claim 20, the rejection of claim 19 is incorporated herein. Furthermore, the combination of Turner, Hinton, Chang, Goel, and John teaches
adjusting the risk prediction includes multiplying the risk prediction by the number of variables in the portion of variables dropped as input to the input nodes and a total number of variables in the dataset before deprecation and dividing the risk prediction by a total number of variables provided as input to the input nodes during training and a number of variables after deprecation. (Column 4 Lines 14 – 18 of Hinton states “Normalization comprises reducing the outgoing weights of each feature detector or input by multiplying them by the probability that the feature detector or input was not disabled. In an example, if the feature detectors of each hidden layer were selectively disabled with probability 0.5 in the training stage, the outgoing weights are halved for the test case since approximately twice as many feature detectors will be enabled. A similar approach is applied to the input layers.” Normalization factor in Hinton is (total-dropped)/total during training. John Equation 5 of manipulation factor accounts total number of manipulated features/ total number of features. Dropping could be interpreted as manipulation as dropping could be done by manipulating value with zero. Deprecation could be interpreted with zeroed values as applicant’s disclosure indicated deprecation (e.g. removed) [0041]. It would have been obvious to POSITA to combine the equations expressed in Hinton with John to compute the equation formed in the claimed limitation.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BYUNGKWON HAN whose telephone number is (571)272-5294. The examiner can normally be reached M-F: 9:00AM-6PM PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYUNGKWON HAN/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights