DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The following action is in response to the original filing of 09/20/2023.
Claims 1-20 are pending and have been considered below.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-5, 7, 11-15 and 17-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by KHAVRONIN, US 2022/0188700 A1.
Regarding claim 1, KHAVRONIN discloses a system, comprising:
a memory storing instructions; and
a processor configured to execute the instructions to cause the processor (¶251, Fig. 21) to be configured to:
train, during a training process, a plurality of trainable machine learning models using a labeled dataset (¶218: training nodes 1732-N train respective plurality of models 1734-N using training and testing data 1706) containing data verified as suspicious or non-suspicious (¶219: training and testing data 1706 includes labeled dataset verifying for correct topic classification, ¶39-40) to generate a plurality of trained machine learning models based on a plurality of candidate machine learning algorithms (¶218: generate plurality of trained models 1734-N, ¶224); and
generate an optimal machine learning model from the plurality of trained machine learning models (¶231),
wherein the optimal machine learning model has an optimal hyperparameter combination and the optimal machine learning model has an optimal model parameter combination learned via the training process using the optimal hyperparameter combination (¶229: performance scores are compared and best-known hyperparameters are fed back to retrain, ¶231: process repeats until an optimized set of hyperparameters and model parameters to generate optimized model, ¶23-26); and
wherein the optimal model machine learning model has a highest performance for suspiciousness determination according to a performance metric when compared to other trained machine learning models of the plurality of trained machine learning models (¶227-231: optimized model has performance value above threshold value of other trained models).
Regarding claim 2, KHAVRONIN discloses the system of claim 1, wherein the processor is further configured to:
activate, via a control signal, a training pipeline service of the system to conduct the training process (¶195: ML workflow includes model building, training and deployment, i.e. a training pipeline service, Fig. 16A, Fig. 16B show workflow activation);
receive, via the training pipeline service, a plurality of feature matrices for a plurality of samples from a feature store (¶206: feature vectors used from table, ¶16); and
train the plurality of trainable machine learning models using the plurality of feature matrices (¶206).
Regarding claim 3, KHAVRONIN system of claim 1, wherein the processor is further configured to search for the optimal machine learning model among the plurality of trained machine learning models using a search strategy consisting of a blended search strategy, a randomized direct search strategy, a Bayesian search strategy, other search strategy, or a combination thereof (¶25).
Regarding claim 4, KHAVRONIN discloses the system of claim 1, wherein the processor is further configured to determine the optimal hyperparameter combination by conducting tuning of a plurality of combinations of hyperparameters associated with the plurality of trainable machine learning models (¶25).
Regarding claim 5, KHAVRONIN discloses the system of claim 1, wherein the processor is further configured to:
determine an amount of time to train the optimal machine learning model (¶26, ¶241);
determine a time when a training pipeline service of the system for training the optimal machine learning model was triggered (¶241); and
generating metadata including the amount of time for training the optimal machine learning model, the time when the training pipeline service was activated, or a combination thereof (¶241: repeat optimization process for a specified threshold period of time).
Regarding claim 7, KHAVRONIN discloses the system of claim 1, wherein the processor is further configured to:
activate a training pipeline service to train the plurality of trainable machine learning models in response to a trigger (¶195: ML workflow includes model building, training and deployment, i.e. a training pipeline service, Fig. 16A, Fig. 16B show workflow activation, ¶243: training process starts); and
transmit a completion signal to a process, a user, or a combination thereof, that triggered activation of the training pipeline service of the system after the plurality of trainable machine learning models are trained (¶246: training process receives stop command).
Regarding claim 11, KHAVRONIN discloses the system of claim 1, wherein the processor is further configured to:
provide, by utilizing the optimal machine learning model and in response to a request to determine whether an identifier associated with a resource attempting to be accessed is suspicious, an indication of whether the identifier is suspicious (¶202: output prediction indicator values 16b36, ¶195: model validation, ¶197: validation/test dataset).
and train plurality of machine learning models using information, wherein the information comprises a verification of the indication (¶197, ¶195: retraining, ¶35: based on results, retrain and retest the model).
Regarding claim 12, KHAVRONIN discloses the system of claim 11, wherein the processor is further configured to:
generate, based on the information, a new optimal machine learning model from the plurality of trained machine learning models (¶197, ¶195, ¶35).
Regarding claim 13, claim 13 recites limitations similar to claim 1 and is similarly rejected.
Regarding claim 14, KHAVRONIN discloses the method of claim 13, further comprising:
adjusting the performance metric to generate an adjusted performance metric (¶229); and
generating a new optimal machine learning model from the plurality of trained machine learning models based on the adjusted performance metric (¶230-231).
Regarding claim 15, KHAVRONIN discloses the method of claim 13, further comprising:
monitoring a current state associated with training the trainable machine learning models during the training process (¶243-247, Fig. 19); and
persisting the current state (¶243-247).
Regarding claim 17, KHAVRONIN discloses the method of claim 13, further comprising triggering the training process after the labeled dataset is persisted in a database (¶204: select topics from sample set of documents).
Regarding claim 18, KHAVRONIN discloses the method of claim 13, further comprising:
modifying a training objective for optimal machine learning model (¶229);
generating a new optimal machine learning model based on the training objective (¶230-231).
Regarding claim 19, KHAVRONIN discloses the method of claim 13, further comprising estimating the optimal model parameters for the optimal machine learning model by utilizing the optimal hyperparameter combination (¶229-231).
Regarding claim 20, claim 20 recites limitations similar to claim 1 and is similarly rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 6, 8-10 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over KHAVRONIN in view of GOODSITT, US 2021/0097343 A1.
Regarding claim 6, KHAVRONIN discloses the system of claim 5, wherein generated metadata includes an amount of time for training the optimal machine model.
KHAVRONIN fails to disclose persisting the metadata and the optimal machine learning model into a model registry.
GOODSITT discloses methods for hyperparameter tuning to rapidly generate models (GOODSITT ¶2). In particular, GOODSITT discloses storing an optimized model and associated descriptive information metadata in a model store (GOODSITT ¶169-170). Therefore it would have been obvious to one having ordinary skill in the art and the teachings of KHAVRONIN and GOODSITT before them before the effective filing of the claimed invention to store metadata and an optimal model in a model store, as taught by GOODSITT, with the generated associated information and optimal machine model of KHAVRONIN. One would have been motivated to make this combination in order to consume less resources through storing results for later retrieval, thereby managing a hypertuning process more efficiently, as suggested by GOODSITT (GOODSITT ¶4).
Regarding claim 8, KHAVRONIN discloses the system of claim 1, wherein the processor is further configured to:
activate a training pipeline service of the system (KHAVRONIN ¶195: ML workflow includes model building, training and deployment, i.e. a training pipeline service, Fig. 16A, Fig. 16B show workflow activation);
obtain, using the training pipeline service, the labeled dataset from a database (KHAVRONIN ¶204: select topics from sample set of documents); and
compute at least one training sample, at least one validation sample, at least one test sample, or a combination thereof, from the at least one labeled dataset to facilitate training of at least one of the plurality of trainable machine learning models (KHAVRONIN ¶204, ¶197).
KHAVRONIN fails to disclose persisting the at least one training sample, the at least one validation sample, the at least one test sample, or a combination thereof, in a sample store.
GOODSITT discloses methods for hyperparameter tuning to rapidly generate models (GOODSITT ¶2). In particular, GOODSITT discloses storing a training dataset to a database (GOODSITT ¶49, ¶52) Therefore it would have been obvious to one having ordinary skill in the art and the teachings of KHAVRONIN and GOODSITT before them before the effective filing of the claimed invention to store training dataset samples, as taught by GOODSITT, with the computed training sample, validation sample and test sample dataset of KHAVRONIN. One would have been motivated to make this combination in order to consume less resources through storing results for later retrieval, thereby managing a hypertuning process more efficiently, as suggested by GOODSITT (GOODSITT ¶4).
Regarding claim 9, KHAVRONIN and GOODSITT disclose the system of claim 8, wherein the processor is further configured to:
obtain the at least one training sample, the at least one validation sample, the at least one test sample, or a combination thereof, from the sample store (GOODSITT ¶52);
compute feature matrices for the at least one training sample, the at least one validation sample, the at least one test sample, or a combination thereof (KHAVRONIN ¶206); and
persist the feature matrices in a feature store (GOODSITT ¶52).
Regarding claim 10, KHAVRONIN and GOODISTT disclose the system of claim 9, wherein the processor is further configured to:
obtain the feature matrices from the feature store (GOODSITT ¶52); and
train the plurality of trainable machine learning models to generate the plurality of trained machine learning models by utilizing the feature matrices (KHAVRONIN ¶206).
Regarding claim 16, KHAVRONIN discloses the method of claim 15.
KHAVRONIN fails to disclose determining whether the training process has been interrupted, an exception has occurred, or a combination thereof; and restarting the training process based on the current state if the training process has been determined to be interrupted, the exception has occurred, or a combination thereof.
GOODSITT discloses methods for hyperparameter tuning to rapidly generate models (GOODSITT ¶2). In particular, GOODSITT discloses storing monitoring a state of a training process of trainable machine models, determining if the process has been interrupted and restarting based on a current state if the process was interrupted (GOODSITT ¶189). Therefore it would have been obvious to one having ordinary skill in the art and the teachings of KHAVRONIN and GOODSITT before them before the effective filing of the claimed invention to provide monitoring and restarting of a training process state based on interruption, as taught by GOODSITT, with the current state monitoring of the training process of KHAVRONIN. One would have been motivated to make this combination in order to provide a user with additional information regarding the and options during training for dealing with potential errors, as suggested by GOODSITT (GOODSITT ¶189).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
XU
US 20230072913 A1
Classification based on imbalanced dataset
KUTT
US 20220046057 A1
Deep learning for malicious URL classification (URLC) with the innocent until proven guilty (IUPG) learning framework
LINDAUER
US 20220012636 A1
Method and device for creating a system for the automated creation of machine learning systems
SHIUE
US 20210406346 A1
Determining optimal machine learning models
MATYSKA
US 20210248611 A1
Method, use thereof, computer program product and system for fraud detection
CHANG
US 20210142222 A1
Automated data and label creation for supervised machine learning regression testing
HEYMAN
US 20210006592 A1
Phishing detection based on interaction with end user
MANOSELVAM
US 20200374313 A1
Resolving redirects for enhanced security
HAJIMIRSADEGHI
US 20200076841 A1
Context-aware feature embedding and anomaly detection of sequential log data using deep recurrent neural networks
Saxena, Aditi, et al. "Detection of web attacks using machine learning based URL classification techniques." 2022 2nd International conference on intelligent technologies (CONIT). IEEE, 2022.
Rajagopal, Smitha, and Poornima Panduranga Kundapur. "Towards effective network intrusion detection: from concept to creation on Azure cloud." IEEE Access 9 (2021): 19723-19742.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW L TANK whose telephone number is (571)270-1692. The examiner can normally be reached Monday-Thursday 9a-6p.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW L TANK/Primary Examiner, Art Unit 2141
Prosecution Insights