DETAILED ACTION
This action is written in response to the application filed October 17, 2023. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Subject Matter Eligibility
In determining whether the claims are subject matter eligible, the examiner has considered and applied guidance from MPEP § 2106. The examiner finds that the independent claims are directed to the practical application of iteratively monitoring and updating machine learning classification models.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The following are the references relied upon in the rejections below:
Agrusa (US 2009/0210814 A1)
James (James G, Witten D, Hastie T, Tibshirani R. An introduction to statistical learning. New York: springer; 2013 Jun, corrected 8th printing 2017. PP. 1-264.)
Kandel (Kandel, Sean, et al. "Research directions in data wrangling: Visualizations and transformations for usable and credible data." Information visualization 10.4 (2011): 271-288.)
Lu (Lu, Jie, et al. "Learning under concept drift: A review." IEEE transactions on knowledge and data engineering 31.12 (2018): 2346-2363.)
Yang (Yang, Yafang, et al. "Detection of Behavior Aging from Keystroke Dynamics." 2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS). IEEE, 2021.)
Claims 1-8 and 13-20 are rejected under 35 U.S.C. 103 as being unpatentable over James, Yang and Lu.
Regarding claims 1, 2 and 17, James discloses a system (and a related method and computer-readable media) for iteratively updating dynamic model validation algorithms for machine learning–based classification models, the system comprising:
one or more processors; and
one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause operations comprising:
A processor and a memory are inherent throughout James.
receiving a first dataset …. ;
P. 37, “Suppose that we seek to estimate f on the basis of training observations {(x1, y1), . . . , (xn, yn)}, where now y1, . . . , yn are qualitative.”
based on providing the first dataset and first output to a first validation model, generating a first validation metric for a classification model, wherein the first output is generated based on providing the first dataset to the classification model …
The examiner interprets “validation metric” according to its broadest reasonable interpretation in view of the applicant’s specification, particularly the illustrative description at [0032]: “In some embodiments, the validation metric may include a percentage match, where the percentage match indicates a fraction of outputs that are deemed accurate or correct.” The applicant does not define this term.
P. 37, “Equation 2.8 is referred to as the training error rate because it is computed based on the data that was used to train our classifier. As in the regression setting, we are most interested in the error rates that result from applying our classifier to test observations that were not used in training. The test error rate associated with a set of test observations of the form test error (x0, y0) is given by
PNG
media_image1.png
26
152
media_image1.png
Greyscale
where ˆy0 is the predicted class label that results from applying the classifier to the test observation with predictor x0. A good classifier is one for which the test error (2.9) is smallest.”
P. 176, “In Chapter 2 we discuss the distinction between the test error rate and the training error rate. The test error is the average error that results from using a statistical learning method to predict the response on a new observation—that is, a measurement that was not used in training the method. Given a data set, the use of a particular statistical learning method is warranted if it results in a low test error.”
generating, according to transformation criteria, a first plurality of datasets, wherein the first plurality of datasets comprises representations, in a second data format different from the first data format, of the first dataset and a second plurality of datasets … ;
P. 115, “3.6.5 Non-linear Transformations of the Predictors The lm() function can also accommodate non-linear transformations of the predictors. For instance, given a predictor X, we can create a predictor X2 using I(X^2).”
P. 122, “Try a few different transformations of the variables, such as log(X), √X, X2.”
based on providing each dataset of the first plurality of datasets to the classification model, generating a plurality of outputs, …
See pp. 115-116, showing code for generating a linear regression model including transformed predictors as described above.
based on providing the first plurality of datasets and the plurality of outputs to a second validation model, generating a second validation metric;
P. 37, “Equation 2.8 is referred to as the training error rate because it is computed based on the data that was used to train our classifier. As in the regression setting, we are most interested in the error rates that result from applying our classifier to test observations that were not used in training. The test error rate associated with a set of test observations of the form test error (x0, y0) is given by
PNG
media_image1.png
26
152
media_image1.png
Greyscale
where ˆy0 is the predicted class label that results from applying the classifier to the test observation with predictor x0. A good classifier is one for which the test error (2.9) is smallest.”
based on comparing the first validation metric with the second validation metric, …
P. 210, “6.1.3 Choosing the Optimal Model Best subset selection, forward selection, and backward selection result in the creation of a set of models, each of which contains a subset of the p predictors. In order to implement these methods, we need a way to determine which of these models is best. As we discussed in Section 6.1.1, the model containing all of the predictors will always have the smallest RSS and the largest R2, since these quantities are related to the training error. Instead, we wish to choose a model with a low test error.”
Yang discloses the following further limitations which James does not disclose:
receiving a first dataset, wherein the first dataset comprises information in a first data format corresponding to a first user;
PP. 586-87 “We choose GREYC-Web database [28] for validation the proposed method. In the dataset, 118 users were involved in its creation and typed the same password “SÉSAME”. 26 users provided more than 130 samples. These users have different sessions with different time intervals. Some users have a long time span (e.g., 500 days). The concept drift of users’ typing rhythm is very obvious in long time span. In the dataset, most participants are students in computer science, chemistry and electronics, which they are familiar with various keyboards. Further, each user chooses typing login and password without any pressure.”
… wherein the first output comprises a first evaluation of user activity of the first user;
P. 584 “The extracted features from keystroke dynamics pattern in most of the researches are timing information of key down/hold/up events”.
P. 584 “The approach of building the keystroke dynamics model can be divided into statistical approach and machine learning [13]. Statistics approach is easy to execute and acquires low overhead. These methods authenticate the user identity by matching the distance between user reference and test query (e.g., Euclidean metric [15]). …. Various systems can authenticate the legitimate user and prevent imposter intrusion with good performance.”
the first dataset and a second plurality of datasets corresponding to a plurality of users;
Id.
wherein each output of the plurality of outputs comprises a corresponding evaluation of corresponding user activity of a corresponding user of the plurality of users;
Id.
At the time of filing, it would have been obvious to a person of ordinary skill to apply the machine learning model techniques disclosed by James to the problem of user authentication as addressed by Yang. James is a basic teaching reference for ML which describes how to build classification models like those described throughout Yang.
Lu discloses the following further limitations which James/Yang do not disclose:
… generating an evaluation metric of the first validation model with respect to the second validation model;
P. 2350, “The second largest category of drift detection algorithms is data distribution-based drift detection. Algorithms of this category use a distance function/metric to quantify the dissimilarity between the distribution of historical data and the new data. If the dissimilarity is proven to be statistically significantly different, the system will trigger a learning model upgradation process.”
See also p. 2349, fig. 5 (reproduced below).
PNG
media_image2.png
556
618
media_image2.png
Greyscale
based on the evaluation metric, generating a plurality of updated model parameters for the first validation model;
Id. “learning model upgradation process”.
generating an updated first validation model based on the plurality of updated model parameters; and
PNG
media_image3.png
148
676
media_image3.png
Greyscale
P. 2349, fig. 6 (reproduced above). This figure illustrates a continuous / ongoing receipt of new data. At each time step, existing models are validated with respect to new data.
See also fig. 5 (reproduced above).
training the classification model to output updated user evaluations based on a third validation metric generated by the updated first validation model.
P. 2350, “Id. “learning model upgradation process”.
At the time of filing, it would have been obvious to a person of ordinary skill to combine the James/Yang system with the techniques disclosed by Lu for detecting and accounting for concept drift. This ensures that models retain their predictive performance, and do not become stale due to changes in the distribution of incoming data.
Regarding claims 3 and 18, Yang discloses the further limitation wherein generating the first validation metric comprises:
determining first user activity data from the first dataset, wherein the first user activity data includes information characterizing activities by the first user; …
P. 584 “The extracted features from keystroke dynamics pattern in most of the researches are timing information of key down/hold/up events”.
P. 585, “A. Pre-treatment and Feature Extraction Phase To accurately describe typing rhythm of an individual, we choose timing information of typing rhythm:
RR: time duration between two release events of two successive keys.PP: time duration between two press events of two successive keys.RP: time interval between release event of current key and press event of next key.PR: time interval between press event and release event of one key.
Hence, the feature vector of keystroke dynamic is composed of [RR, PP, RP, PR] containing the necessary information.”
providing the first user activity data to the classification model, wherein the classification model is an artificial neural network–based decision-making model;
P. 584, second col., “neural network”.
based on providing the first user activity data to the classification model, generating the first output, wherein the first output indicates a first predicted user permission; and
P. 583, introduction, “Keystroke dynamics authenticate or recognize a user by the habitual patterns or typing rhythm of an individual. Moreover, fixed keystroke dynamics that combine password and typing rhythm is a strong authentication method. In addition, it is invisible and static or continuously authenticates user identity without any additional hardware [1]. Therefore, keystroke dynamics have high acceptability for users.” (Emphasis added.)
P. 586, algorithm 1 (reproduced below).
PNG
media_image4.png
400
540
media_image4.png
Greyscale
PP. 586-87 “We choose GREYC-Web database [28] for validation the proposed method. In the dataset, 118 users were involved in its creation and typed the same password “SÉSAME”. 26 users provided more than 130 samples. These users have different sessions with different time intervals. Some users have a long time span (e.g., 500 days). The concept drift of users’ typing rhythm is very obvious in long time span. In the dataset, most participants are students in computer science, chemistry and electronics, which they are familiar with various keyboards. Further, each user chooses typing login and password without any pressure.”
generating the first validation metric based on the first predicted user permission.
See algorithm 1 above. “Calculate number ne of misjudgment samples”.
Regarding claims 4 and 19, Yang discloses the further limitation wherein generating the first validation metric comprises:
obtaining a first validation indicator for the first user, wherein the first validation indicator indicates a first determined user permission for the first user; and
See algorithm 1 (reproduced supra), “yi”
providing the first validation indicator, the first output, and the first user activity data to the first validation model to generate the first validation metric.
See algorithm 1 (reproduced supra), “Calculate number ne of misjudgment samples”.
Regarding claims 5 and 20, Yang discloses the further limitation wherein generating the first validation metric comprises:
determining a match between the first determined user permission and the first predicted user permission;
P. 584, second col., “These methods authenticate the user identity by matching the distance between user reference and test query (e.g., Euclidean metric [15]).”
obtaining prior validation data for the classification model, wherein the prior validation data includes indications of matches between determined user permissions and predicted user permissions for a set of datasets, wherein the set of datasets includes a pre-determined number of datasets previously provided to the classification model;
P. 588, “sliding window” technique.
based on the match and the prior validation data, updating a moving-average accuracy metric for the classification model; and
Id.
determining the first validation metric based on the moving-average accuracy metric for the classification model.
See generally p. 588.
Regarding claim 6, Yang discloses the further limitation wherein generating the first plurality of datasets comprises:
determining a set of values for the first dataset, wherein the set of values includes values associated with a set of variables within the first dataset;
P. 586, algorithm 1 (reproduced supra), “Input: Initial test set with time label, namely ITS={(xi, yi, recalli, ti)}, (i=1,2,…,N)”
modifying the set of values to generate a modified set of values associated with a modified set of variables; and
P. 584, first col., “Static authentication/verification systems firstly obtain either samples of genuine or samples of both genuine and imposters. Then these samples are preprocessed by some technologies (e.g., aberration and normalization [12]) and extract features.”
generating a first representation of the first dataset in the second data format based on the modified set of values.
P. 584, first col., “The extracted features from keystroke dynamics pattern in most of the researches are timing information of key down/hold/up events [15]. Hence, the feature vector is composed of the temporal timing information {Press-Press, Press-Release, Release-Release, Release-Press} [12,15].”
Regarding claim 7, Yang discloses the further limitation wherein modifying the set of values to generate the modified set of values comprises:
determining a first value corresponding to a first variable in the first dataset;
P. 584, first col., “The extracted features from keystroke dynamics pattern in most of the researches are timing information of key down/hold/up events [15]. Hence, the feature vector is composed of the temporal timing information {Press-Press, Press-Release, Release-Release, Release-Press} [12,15].”
generating a second value and a third value, wherein the first value comprises the second value and the third value, wherein the second value is associated with a second variable, and wherein the third value is associated with a third variable;
Id.
generating the modified set of values to include the second value and the third value; and
P. 584, first col., “Static authentication/verification systems firstly obtain either samples of genuine or samples of both genuine and imposters. Then these samples are preprocessed by some technologies (e.g., aberration and normalization [12]) and extract features.”
generating the modified set of variables to include the second variable and the third variable.
Id., aberration and normalization are typically performed on all attributes / variables.
Regarding claim 8, James discloses the further limitation wherein modifying the set of values to generate the modified set of values comprises:
generating a plurality of validation statuses for the set of values, wherein each validation status of the set of values indicates a validity of a corresponding value of the set of values for a corresponding variable of the set of variables;
P. 37, “Equation 2.8 is referred to as the training error rate because it is computed based on the data that was used to train our classifier. As in the regression setting, we are most interested in the error rates that result from applying our classifier to test observations that were not used in training. The test error rate associated with a set of test observations of the form test error (x0, y0) is given by
PNG
media_image1.png
26
152
media_image1.png
Greyscale
where ˆy0 is the predicted class label that results from applying the classifier to the test observation with predictor x0. A good classifier is one for which the test error (2.9) is smallest.”
Yang discloses the following further limitation:
generating a set of validated values corresponding to a set of validated variables, wherein each validated value of the set of validated values indicates a corresponding valid value of the set of values; and
P. 584, first col., “Static authentication/verification systems firstly obtain either samples of genuine or samples of both genuine and imposters. Then these samples are preprocessed by some technologies (e.g., aberration and normalization [12]) and extract features.”
generating the modified set of values to include the set of validated values.
Id.
Regarding claim 10, Yang discloses the further limitation wherein providing the first dataset to the classification model comprises:
determining a real-time classification model, wherein the real-time classification model is configured to accept datasets of the first data format without modification;
P. 586, algorithm 1 (reproduced supra), “Input: Initial test set with time label, namely ITS={(xi, yi, recalli, ti)}, (i=1,2,…,N)”
determining that the first dataset includes missing values or erroneous values;
P. 584, first col., “Static authentication/verification systems firstly obtain either samples of genuine or samples of both genuine and imposters. Then these samples are preprocessed by some technologies (e.g., aberration and normalization [12]) and extract features.”
providing the first dataset to the real-time classification model; and
P. 587, first col., “We build a user model based on iforest with different parameters. In addition, various experiments are repeated 20 times to get rid of abnormal values. We present the aggregated recall and precision of the model by trim means in different time intervals according to chronological order.”
generating the first output based on providing the first dataset to the real-time classification model.
P. 587, first col., “Adaptive method based on threshold [29]: It updates user reference based on threshold in real time,”
Regarding claim 13, Lu discloses the further limitation comprising:
receiving a third plurality of datasets corresponding to a plurality of validated users;
PNG
media_image3.png
148
676
media_image3.png
Greyscale
P. 2349, fig. 6 (reproduced above). This figure illustrates a continuous / ongoing receipt of new data. At each time step, existing models are validated with respect to new data.
James discloses the following further limitations:
receiving a validation dataset, wherein the validation dataset includes a plurality of validation indicators, wherein each validation indicator of the plurality of validation indicators indicates a corresponding determined user permission for a corresponding dataset of the third plurality of datasets;
P. 37, “Equation 2.8 is referred to as the training error rate because it is computed based on the data that was used to train our classifier. As in the regression setting, we are most interested in the error rates that result from applying our classifier to test observations that were not used in training. The test error rate associated with a set of test observations of the form test error (x0, y0) is given by
PNG
media_image1.png
26
152
media_image1.png
Greyscale
where ˆy0 is the predicted class label that results from applying the classifier to the test observation with predictor x0. A good classifier is one for which the test error (2.9) is smallest.”
P. 176, “In Chapter 2 we discuss the distinction between the test error rate and the training error rate. The test error is the average error that results from using a statistical learning method to predict the response on a new observation—that is, a measurement that was not used in training the method. Given a data set, the use of a particular statistical learning method is warranted if it results in a low test error.”
providing the third plurality of datasets and the validation dataset to the classification model; and
Id.
based on providing the third plurality of datasets and the validation dataset to the classification model, training the classification model to predict user permissions for users as output.
Id.
Regarding claim 14, Yang discloses the further limitation wherein providing the first dataset and the first output to the first validation model comprises:
providing the first dataset to a machine learning model, wherein the machine learning model is a model configured to generate user evaluation metrics based on user data;
PP. 586-87 “We choose GREYC-Web database [28] for validation the proposed method. In the dataset, 118 users were involved in its creation and typed the same password “SÉSAME”. 26 users provided more than 130 samples. These users have different sessions with different time intervals. Some users have a long time span (e.g., 500 days). The concept drift of users’ typing rhythm is very obvious in long time span. In the dataset, most participants are students in computer science, chemistry and electronics, which they are familiar with various keyboards. Further, each user chooses typing login and password without any pressure.”
generating a user evaluation status for the first user as output based on providing the first dataset to the machine learning model; and
P. 584 “The extracted features from keystroke dynamics pattern in most of the researches are timing information of key down/hold/up events”.
P. 584 “The approach of building the keystroke dynamics model can be divided into statistical approach and machine learning [13]. Statistics approach is easy to execute and acquires low overhead. These methods authenticate the user identity by matching the distance between user reference and test query (e.g., Euclidean metric [15]). …. Various systems can authenticate the legitimate user and prevent imposter intrusion with good performance.”
providing the first dataset and the user evaluation status to the first validation model.
P. 587, first col., “user model”.
See also Lu, p. 2350, describing a drift detection model.
Regarding claim 15, Yang discloses the further limitation comprising:
retrieving a second dataset corresponding to a second user;
PP. 586-87 “We choose GREYC-Web database [28] for validation the proposed method. In the dataset, 118 users were involved in its creation and typed the same password “SÉSAME”. 26 users provided more than 130 samples. These users have different sessions with different time intervals. Some users have a long time span (e.g., 500 days). The concept drift of users’ typing rhythm is very obvious in long time span. In the dataset, most participants are students in computer science, chemistry and electronics, which they are familiar with various keyboards. Further, each user chooses typing login and password without any pressure.”
providing the second dataset to the classification model to generate second output; …
P. 584, second col., “neural network”.
providing the third validation metric and the second dataset to the classification model; and
P. 584, second col., “neural network”.
based on providing the third validation metric and the second dataset to the classification model, training the classification model to generate evaluation data for users.
Id.
James discloses the following further limitation:
providing the second dataset and the second output to the updated first validation model to generate a third validation metric;
See citations to p. 37 and p. 176 above, discussing test error rate.
Regarding claim 16, its further limitations are an obvious extension of the method of claim 14: they merely specify repeating the functionality of the parent claim. The obviousness analysis of claim 14 applies equally here.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over James, Yang, Lu and Agrusa.
Regarding claim 9, Agrusa discloses the further limitation which James/Yang/Lu do not disclose wherein generating the first plurality of datasets comprises:
obtaining a specification for the second data format, wherein the specification indicates format requirements for the second data format and security requirements for the second data format; and
[0004] “Recently, OPC Foundation, which oversees the development of the object-linking and embedding (OLE) for process control has adopted a Unified Architecture Specification that sets forth standard data formats and security and access protocols so that its members may build and implement systems that work together under a common framework.”
generating a first representation of the first dataset in the second data format to include data satisfying the format requirements and the security requirements.
Id. “so that its members may build and implement systems”.
At the time of filing, it would have been obvious to a person of ordinary skill to apply the data formatting techniques of Agrusa with the James/Yang/Lu system because this would provide for data privacy and security in an environment that relies upon external (received) data.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over James, Yang, Lu and Kandel.
Regarding claim 11, Lu discloses the further limitation wherein generating the plurality of outputs comprises:
determining a batch classification model, wherein the batch classification model is configured to accept datasets of the second data format;
P. 2349, fig. 6 (reproduced below). This figure illustrates a continuous / ongoing receipt of new data in batches. At each time step, existing models are validated with respect to new data.
PNG
media_image3.png
148
676
media_image3.png
Greyscale
Kandel discloses the following further limitations which James/Yang/Lu do not disclose:
detecting that a second dataset of the second plurality of datasets includes data inconsistent with the second data format; and
P. 272, second col., “John loads the merged data file in a visualization tool (step 3). The tool immediately gives the error message ‘Empty cells in column 3’; it cannot cope with missing data.”.
generating, for display in a user interface of a user device, an error message, wherein the error message indicates that the second dataset is of an invalid data format for the batch classification model.
Id.
At the time of filing, it would have been obvious to a person of ordinary skill to apply the data formatting technique disclosed by Kandel with the James/Yang/Lu system because this would ensure that data received by the system is usable to the machine learning models being trained.
Additional Relevant Prior Art
The following references were identified by the Examiner as being relevant to the disclosed invention, but are not relied upon in any particular prior art rejection:
Jain discloses a model validation technique with applications to user authentication problems. (US 12,106,205 B1)
Claim Objections and Allowable Subject Matter
Dependent claim 12 is allowable over the prior art, but are objected to as depending upon a rejected parent claim.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Vincent Gonzales whose telephone number is (571) 270-3837. The examiner can normally be reached on Monday-Friday 7 a.m. to 4 p.m. MT. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang, can be reached at (571) 270-7092.
Information regarding the status of an application may be obtained from the USPTO Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov.
/Vincent Gonzales/Primary Examiner, Art Unit 2124
Prosecution Insights